[fluentd] add condition based output field

fluentd is an amazing piece of software but can sometimes give one a hard time. Case in point, how can one add a field to an output only if a certain string exists in another record.

Turns out with a little regex, it’s decently simple.

The below code will add a field called “_newfield” with value “OURSTRING” when the “log” record contains the exact same string at the beginning of the line. The regex can be further adjusted as needed.

<filter **>
  @type parser
  key_name log
  reserve_data true
  emit_invalid_record_to_error false
  <parse>
    @type regexp
    expression /^(?<_newfield>^OURSTRING)/
  </parse>
</filter>

 

“emit_invalid_record_to_error false” will avoid sending error logs if the regex does not match while “reserve_data true” will preserve the initial message and just add a field to it before passing it out further.

[Ansible] How to json_query a key that contains a dot

Today I found myself needing to extract the value of a json key that contained a dot (‘.’). As usual, json_query to the rescue, except that it didn’t work, returning an empty value. With a little bit of digging and trial and error testing, the solution presented itself. The key is to enclose the json key in quotes and then escape them.

 

Here is an example json file:

{
	"menu": {
		"id": "file",
		"value": "File",
		"popup": {
			"dot.key": [{
					"value": "New",
					"onclick": "CreateNewDoc()"
				},
				{
					"value": "Open",
					"onclick": "OpenDoc()"
				},
				{
					"value": "Close",
					"onclick": "CloseDoc()"
				}
			]
		}
	}
}

 

And an example ansible playbook:

- name: json manipulation 
  gather_facts: no
  hosts: localhost
  vars:
    json_var: "{{ lookup('file', 'file.json') | from_json }}"
  tasks:
  - name: get "menu.item" from json
    set_fact:
      json_out: "{{ json_var | json_query('menu.popup.\"dot.key\"[].value') }}"

 

We can clearly see the result:

PLAY [json manipulation] ******************************************************************************************************************************************************************************************
META: ran handlers

TASK [get "menu.item" from json] **********************************************************************************************************************************************************************************
task path: /tmp/1/1.yaml:7
ok: [localhost] => {
    "ansible_facts": {
        "json_out": [
            "New", 
            "Open", 
            "Close"
        ]
    }, 
    "changed": false
}
META: ran handlers
META: ran handlers

PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0   

 

PS: In this case we have 3x keys under “dot.key”. We obtain all of them in our result because we are selecting the whole dictionary ( ‘\”dot.key\”[]’ ) — notice the square brackets. We insert a number here and select the position of the key we want to get the value for. If for instance we would only need the 2nd key, we would use ‘\”dot.key\”[1]’ — “0” being the first value

ESX 6.5 and 6.7 on HPE G6/G7 server PSOD fix

Background

HP’s done it again. They’ve managed to break their custom ESX ISO on the G6 and G7 servers. I suspect it’s the same for G8.

If like me you found yourself perplexed by the PSOD (pink screen of death) after upgrading from ESX 6.0 to 6.5, keep on reading for the fix (skip to the bottom for the download link to an already fixed ISO).

The issue

It seems the “hpe-smx-provider” driver version 6.5.0 from the ESX 6.5 ISO is causing the PSOD.

The fix and the drawbacks

(Simply) Replace the driver with it’s older counterpart (version 6.0.0) and reinstall (see below for the procedure or check the “download link” section at the very end for an already fixed ISO).  Unfortunately if you decide to create a custom ISO with the previous version of “hpe-smx-provider”, you will no longer be able to upgrade to any future ESX version and will need to do a full installation every time (thank you HP!). That being said, here is the procedure to customize the ISO:

Customize your own HPE ESX ISO

  1. Install PowerCLI from here.  At the time of this writing, the latest version was 6.5.0R1.
  2. Download both the ESX 6.5 and 6.0 offline bundles and save them to a convenient place (ex: C:\HP) — download from this link. To make it easier, rename them to something like HPE_ESX6.5.zip and HPE_ESX6.0.zip.
  3. Open up PowerCLI and do the following:Add the 6.5 bundle
    cd C:\HP
    Add-EsxSoftwareDepot -DepotUrl HPE_ESX6.5.zip

    # check that the profile was loaded and the vendor is HPE

    Get-EsxImageProfile

    # clone the profile (feel free to specify whatever you like for the “vendor” when prompted)

    New-EsxImageProfile -CloneProfile HPE-ESXi-6.5.0* -Name "HPE-ESX-6.5-CUST"

    # double check the profile was indeed cloned

    Get-EsxImageProfile

    # Remove the “hpe-smx-provider” driver from the clone

    Remove-EsxSoftwarePackage HPE-ESX-6.5-CUST hpe-smx-provider

    # Add the 6.0 bundle

    Add-EsxSoftwareDepot -DepotUrl HPE_ESX6.0.zip

    # check the profile

    Get-EsxImageProfile

    # check driver versions from both bundles

    Get-EsxSoftwarePackage | findstr smx

    # Add hpe-smx-provider version 6.00 to the custom profile

    add-esxsoftwarepackage -imageprofile HPE-ESX-6.5-CUST -softwarepackage "hpe-smx-provider 600.03.11.00.9-2768847"

    # Export the profile to an ISO

    Export-EsxImageProfile -ImageProfile HPE-ESX-6.5-CUST -ExportToIso -filepath "HPE-ESX-6.5-CUST.iso"

The results

Download links

The following images have hpe-smx-provider version 600.03.11.00.9-2768847. Nothing else was changed. The 6.5.0d (build 5310538) is running on an HP Proliant DL380 G7 since March 2018 and was also tested on a DL380 G6 for a couple of months. For convenience, I’ve also built a 6.7.0 ISO and at the request of a reader, 6.7.0 Update1 from April of 2019:

VMware-ESXi-6.5.0-5310538-HPE-650.10.1.5.20-Oct2017_CUSTOM.iso
VMware-ESXi-6.7.0-9484548-HPE-Gen9plus-670.10.3.5.6-Sep2018_CUSTOM.iso
VMware-ESXi-6.7.0-Update1-11675023-HPE-Gen9plus-670.U1.10.4.0.19-Apr2019_CUSTOM.iso

Enjoy!