fluent-plugin-formatter-protobuf

Fluentd formatter plugin to format messages into Protobuf v3 encoded binary.

Installation

RubyGems

$ gem install fluent-plugin-formatter-protobuf

GitHub RubyGems Registry

$ gem install \
  fluent-plugin-formatter-protobuf \
  --version "<version>" \
  --source "https://rubygems.pkg.github.com/fluent-plugins-nursery"

Bundler (RubyGems)

Add following line to your Gemfile:

gem "fluent-plugin-formatter-protobuf", "<version>"

Bundler (GitHub RubyGems Registry),

source "https://rubygems.pkg.github.com/fluent-plugins-nursery" do
  gem "fluent-plugin-formatter-protobuf", "<version>"
end

And then execute:

$ bundle

Quick start

1. Generate the protobuf ruby methods
   1. e.g protoc --proto_path=. --ruby_out=. ./log.proto
2. Modify the <format> section to include class_name, which is your Protobuf message name, and include_paths, the path where the generated Ruby types are stored
   1. Given protobuf class Your::Protobuf::Class::Name class should be given as Your.Protobuf.Class.Name in class_name. The exact name can be found in the generated Ruby files

Note

This plugin only supports Protobuf v3.

Example

<source>
    @type tcp
    tag tcp.events
    port 5170
    delimiter "\n"
    <parse>
        @type json
    </parse>
</source>

<match tcp.events>
    @type kafka2

    brokers "#{BROKER_ENDPOINTS}"
    compression_codec lz4

    <buffer topic>
        @type memory
        flush_interval 5s
    </buffer>

    <format>
        @type protobuf
        class_name "Log"
        include_paths ["/opt/fluent-plugin-formatter-protobuf/log_pb.rb"]
    </format>
</match>

Configuration

parameter	type	description	default
class_name	string (required)	Ruby Protobuf class name. Used to encode into Protobuf binary
decode_json	boolean (optional)	Serializes record from canonical proto3 JSON mapping (https://developers.google.com/protocol-buffers/docs/proto3#json) into binary	false
include_paths	array (required)	Generated Ruby Protobuf class files path	[]
require_method	string (optional)	Determine how to bring the generated Ruby Protobuf class files into scope. If your generated Ruby Protobuf class files are provided by a Ruby Gem, you may want to use 'require'. If you are providing the generated files as files, you may want to use 'require_relative'	require

Tips

1. I have multiple _pb.rb files and they depend on one another. When I use this plugin, I'm getting a LoadError.

This happens because the plugin currently loads paths in include_paths sequentially. You can either sort the files in correct dependency order (which is cumbersome), or add to Ruby's $LOAD_PATH.

For example, you have 2 generated ruby files, a_pb.rb and b_pb.rb, and both a_pb.rb depends on b_pb.rb.

You can either order them in appropriate dependency order, such as

<format>
  @type protobuf
  class_name "SomePackage.A"
  include_paths ["/some/path/b_pb.rb", "/some/path/a_pb.rb"]
</format>

or you can put the generated files in a directory say /some/path/, and add to Ruby's $LOAD_PATH via the RUBYLIB environment variable.

e.g.

<format>
  @type protobuf
  class_name "SomePackage.A"
  include_paths ["/some/path/a_pb.rb"]
</format>

export RUBYLIB="${RUBYLIB}:/some/path/"
fluentd \
  --gemfile "/some/other/path/Gemfile" \
  --config "/some/other/path/fluent.conf"

License

Apache License, Version 2.0