FIX Protocol Logstash Filter

A LogStash filter plugin for FIX Message parsing

Given a FIX log file that looks like this:

2015-08-26 23:08:38,096 FIX.4.2:DUMMY_INC->ANOTHER_INC: 8=FIX.4.2�9=184�35=F�34=2�49=ANOTHER_INC�50=DefaultSenderSubID�52=20150826-23:08:38.094�56=DUMMY_INC�1=DefaultAccount�11=clordid_of_cancel�41=151012569�54=1�55=ITER�60=20250407-13:14:15�167=FUT�200=201512�10=147�
2015-08-31 17:48:20,890 FIXT.1.1:DUMMY_INC->ANOTHER_INC: 8=FIXT.1.1�9=140�35=W�34=2�49=DUMMY_INC�52=20150831-17:48:20.890�56=ANOTHER_INC�22=99�48=.AQUA-W�262=golden_path_test�268=1�269=3�270=640754�272=20150831�273=17:48:20.882�10=070�
2015-08-31 20:48:26,536 FIXT.1.1:DUMMY_INC->ANOTHER_INC: 8=FIXT.1.1�9=189�35=W�34=5�49=DUMMY_INC�52=20150831-20:48:26.535�56=ANOTHER_INC�22=99�48=ITRZ21�262=req_A�268=2�269=0�270=0.01005�271=10�272=20150831�273=20:48:26.514�269=1�270=0.0101�271=2�272=20150831�273=20:48:26.514�10=123�

The FIX Message filter plugin can read the FIX log as an input and turn it into something like this:

Installation

$ /opt/logstash/bin/plugin install logstash-filter-fix_protocol

Plugin Configuration

Setting	Input type	Required	Default Value
fix_message	string/variable	Yes	"message"
data_dictionary_path	string	Yes	"/PATH/TO/YOUR/DD"
session_dictionary_path	string	No	nil

fix_message

value type is a string
required

Should be the actual fix message passed to the filter. You might need to use a separate filter, like grok, to parse a log and set a fix string variable.

data_dictionary_path

value type is a string
required

Should be the absolute path to your data dictionary xml file.

session_dictionary_path

value type is a string
Not required

Should be the absolute path to your session dictionary xml file for FIX versions > 5.0. Note, if you do not set this but are using FIX 5.0, the filter will still work, but admin messages won't be correctly parsed - you'll lose data. The filter ignores key-value pairs that it doesn't parse correctly.

Sample Config File

Note: For FIX < 5.0, simply omit the session_dictionary_path.

input {
  file {
    path => "/PATH/TO/YOUR/FIX-MESSAGE.log"
    start_position => "beginning"
  }
}
filter {
  grok {
    match => ["message","%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:fix_session}: %{GREEDYDATA:fix_string}"]
  }
  fix_protocol {
    fix_message => fix_string
    session_dictionary_path => "/PATH/TO/FIX/5.0/SESSION/DICTIONARY/FIX.xml"
    data_dictionary_path => "/PATH/TO/FIX/5.0/DATA/DICTIONARY/FIX.xml"
  }
}
output {
  stdout { codec => rubydebug }
}

Sample Config File For Multiple FIX Versions

input {
  file {
    path => "/path/to/fix.log"
    start_position => "beginning"
  }
}
filter {
  grok {
    match => ["message","%{TIMESTAMP_ISO8601:timestamp} %{GREEDYDATA:fix_session}: %{GREEDYDATA:fix_string}"]
  }
  if [message] =~ "=FIX.4.2" {
    fix_protocol {
      fix_message => fix_string
      data_dictionary_path => "/path/to/datadict/FIX42.xml"
    }

  if [message] =~ "=FIX.4.4" {
    fix_protocol {
      fix_message => fix_string
      data_dictionary_path => "/path/to/datadict/FIX44.xml"
    }
  }
  if [message] =~ "=FIX.5.0" {
    fix_protocol {
      fix_message => fix_string
      data_dictionary_path => "/path/to/datadict/FIX50.xml"
    }
  }

  }
}
output {
  stdout { codec => rubydebug }
}

Notice, we're using the Grok filter to create a fix_message variable from a theoretical FIX Message log file. Then, we're passing that variable to our filter. You can see this emulated behavior in our specs.

Development Environment

To get set up quickly, we recommend using Vagrant with the Ansible provisioning available in this source repository.

Setup with Vagrant

Install Ansible
Install VirtualBox
Install Vagrant

Then,

vagrant up

Manual Setup (OSX)

rvm install jruby
rvm use jruby
bundle install
brew install logstash

Then, run bin/console for an interactive prompt that will allow you to experiment.

To release a new version, update the version number in logstash-filter-fix_protocol.gemspec, and then run bundle exec rake release to create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Note: If you get an error message about metadata, you'll need to update to ruby gems > 2.0. Run gem update --system

Running Tests

$ ./bin/rspec rspec

Logstash 2x vs 5x

Remove any installed versions of logstash and install your desired version.

After you've completed the 'Manual change' or 'Ansible provisioning change' below, follow instructions for 'Development Logstash Installation'

Logstash 6.x and 7.x

The error we see when trying to use the plugin in logstash is the following:

[2019-11-15T16:39:20,984][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- i18n/core_ext/string/interpolate>, :path=>"logstash/filters/fix_protocol", :type=>"filter", :name=>"fix_protocol"}

It is not clear what the plug-in's dependency to i18n is since we were able to workaround the issue by simply placing an empty file in this location:

/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/i18n-x.x.x/lib/i18n/core_ext/string/interpolate.rb

Alternatively the source it may be looking for is from an older version of the i18n gem... you can find it here: https://github.com/ruby-i18n/i18n/tree/0.6/lib/i18n/core_ext/string

Everything appears to be working by adding either a blank file or the old gem's interpolate.rb file. Your mileage may vary of course, we've only used this workaround in a development environment and have no familiarity with this plug-ins source code.

Manual change:

Change the version number in lib/logstash/filters/version.rb

module Logstash
  VERSION = '2.x'
end

Ansible provisioning change:

Change the version number in provision/group_vars/all.yml

logstash_version: 5.x # -> 2.x

Run vagrant provision:

vagrant provision

Development Logstash Installation

Add the filter to your installation of LogStash

# /opt/logstash/Gemfile
#...
gem "logstash-output-kafka"
gem "logstash-input-http_poller"
gem "logstash-filter-fix_protocol", :path => "/PATH/TO/YOUR/FORK"

Install the filter plugin

$ /opt/logstash/bin/plugin install --no-verify

Start logstash installation with a LogStash configuration file.
```
$ /opt/logstash/bin/logstash -f /PATH/TO/logstash.conf
```

Contributing

Contributions are welcome! Please see the Contribution Guidelines for details.

FIX Message Logstash Filter is maintained and funded by Connamara Systems, llc.

The names and logos for Connamara Systems are trademarks of Connamara Systems, llc.

Licensing

This software is available under the Apache license and a commercial license. Please see the LICENSE file for the terms specified by the Apache license. The commercial license offers more flexible licensing terms compared to the Apache license, and includes support services. Contact us for more information on the Connamara commercial license, what it enables, and how you can start commercial development with it.

This product includes software developed by quickfixengine.org (http://www.quickfixengine.org/). Please see the QuickFIX Software LICENSE for the terms specified by the QuickFIX Software License.

diogouchoas/logstash-filter-fix_protocol