[fluentd] add condition based output field

fluentd is an amazing piece of software but can sometimes give one a hard time. Case in point, how can one add a field to an output only if a certain string exists in another record.

Turns out with a little regex, it’s decently simple.

The below code will add a field called “_newfield” with value “OURSTRING” when the “log” record contains the exact same string at the beginning of the line. The regex can be further adjusted as needed.

<filter **>
  @type parser
  key_name log
  reserve_data true
  emit_invalid_record_to_error false
  <parse>
    @type regexp
    expression /^(?<_newfield>^OURSTRING)/
  </parse>
</filter>

 

“emit_invalid_record_to_error false” will avoid sending error logs if the regex does not match while “reserve_data true” will preserve the initial message and just add a field to it before passing it out further.