fluent-plugin-insert-id

Fluentd filter plugin to insert unique ID string.

The original implementation was copied from fluent-plugin-google-cloud which was created by Google to handle GKE's official Stackdriver Logging reporting in sidecar containe for each pod.

How it works

# Input message
{"a": "foo", "b": "bar"}
{"a": "foo", "b": "bar"}
{"a": "foo", "b": "bar"}
{"a": "foo", "b": "bar"}
{"a": "foo", "b": "bar"}

# Output message
2019-08-25 21:20:43.539536458 +0000 message.test: {"a":"foo","b":"bar","insert-id":"nu8a3ptahpbetddw"}
2019-08-25 21:20:44.625784851 +0000 message.test: {"a":"foo","b":"bar","insert-id":"nu8a3ptahpbetddx"}
2019-08-25 21:20:45.579060596 +0000 message.test: {"a":"foo","b":"bar","insert-id":"nu8a3ptahpbetddy"}
2019-08-25 21:20:47.019565139 +0000 message.test: {"a":"foo","b":"bar","insert-id":"nu8a3ptahpbetddz"}
2019-08-25 21:20:50.035415329 +0000 message.test: {"a":"foo","b":"bar","insert-id":"nu8a3ptahpbetdea"}
  • It is guaranteed that ID is fixed length string which contains 0-9 and a-z lowercase characters.
  • Initial ID generated is random string like "nu8a3ptahpbetddc".
  • Series ID after the initial ID are 'incremented' string which uses Ruby's String.next().
  • 'incremented' string also has 'carry' feature. Please check below links for more details.
  • This ordered ID makes debugging easier in most cases.

Installation

RubyGems

$ gem install fluent-plugin-insert-id

Bundler

Add following line to your Gemfile:

gem "fluent-plugin-insert-id"

And then execute:

$ bundle

Configuration

You can generate configuration template:

$ fluent-plugin-config-format filter insert-id

You can copy and paste generated documents here.

  • Copyright(c) 2019- Kamome Shido
  • Copyright(c) 2014- Google Inc. All rights reserved.
  • License
    • Apache License, Version 2.0