spotify/ffwd-ruby

Enabling protobuf input seems to prevent json input listener

Closed this issue · 3 comments

When following the instructions at docs/basic-local-ffwd.md, a ffwd config that looks like

:input:
  - :type: "json"
    :kind: :line

:input:
  - :type: "json"
    :kind: :frame

:debug: {}

results in ffwd listening on port 19000 for JSON messages:

$ ffwd ~/code/ffwd-local-config.yml
Loaded: /Users/mattbrown/code/ffwd-local-config.yml
I, [2016-03-03T14:04:53.552221 #84153]  INFO -- FFWD::Debug::TCP: Bind on tcp://localhost:19001 (attempt 1)
I, [2016-03-03T14:04:53.553788 #84153]  INFO -- FFWD::Plugin::JSON: Bind on udp://localhost:19000 (attempt 1)
I, [2016-03-03T14:04:53.553875 #84153]  INFO -- FFWD::Plugin::JSON:   config: {:type=>"json", :kind=>:frame, :host=>"localhost", :port=>19000, :rebind_timeout=>10, :receive_buffer_size=>nil}
I, [2016-03-03T14:04:53.553972 #84153]  INFO -- FFWD::Core: Memory limited to 1000.0 MB (:core -> :memory -> :limit)

When I added the protobuf plugin to the config and leave the other elements:

:input:
  - :type: "json"
    :kind: :line

:input:
  - :type: "json"
    :kind: :frame

:input:
  - :type: "protobuf"

:debug: {}

then ffwd seems to listen on port 19091 for protobuf messages as expected but no longer listens on port 19000 for JSON messages.

$ ffwd ~/code/ffwd-local-config.yml
Loaded: /Users/mattbrown/code/ffwd-local-config.yml
I, [2016-03-03T13:56:10.051132 #83717]  INFO -- FFWD::Debug::TCP: Bind on tcp://localhost:19001 (attempt 1)
I, [2016-03-03T13:56:10.053441 #83717]  INFO -- FFWD::Plugin::Protobuf: Bind on udp://localhost:19091 (attempt 1)
I, [2016-03-03T13:56:10.053550 #83717]  INFO -- FFWD::Plugin::Protobuf:   config: {:type=>"protobuf", :host=>"localhost", :port=>19091, :protocol=>"udp", :rebind_timeout=>10, :receive_buffer_size=>nil}
I, [2016-03-03T13:56:10.053693 #83717]  INFO -- FFWD::Core: Memory limited to 1000.0 MB (:core -> :memory -> :limit)

A command like netstat -an | grep 19000 confirms that nothing is listening on port 19000 any longer with this second config.

Am I following the config wrong? I find it useful to run ffwd locally when adding new metrics to a service that reports to ffwd, and it is inconvenient to have to switch the config around based on whether the service is reporting via protobuf or json.

$ gem list | grep ffwd ffwd (0.4.7) ffwd-protobuf (0.4.7)

The documentation appear to have been wrong. It's only possible to define one instance of a given key at a time, so it should have been:

:input:
  - :type: "json"
    :kind: :line
  - :type: "json"
    :kind: :frame
  - :type: "protobuf"

I've fixed the example in ad3b2c7, please check that it works and re-open if that's not the case.

@udoprog ah that makes sense. Thanks!

With the update to the doc, I think it can still be read that adding the protobuf listener means adding a second :input: element