This consumes the Twitter streaming API, processing and feeding the Redis instance for the rest of Emojitracker.
-
Make sure you have Ruby 2.1.x installed.
-
Get the repository and basic dependencies going:
git clone mroth/emojitrack-feeder cd emojitrack-feeder bundle install --without=production -
Copy
.env-sampleto.envand configure required variables. -
Make sure you have Redis installed and running. The rules in
lib/config.rbcurrently dictate the order a redis server instance is looked for. -
Run all processes via
foreman startorforego start(depending on which you have installed).
Be sure to note that while the processing power is fairly managable, the feeder component of emojitrack requires on it's own about 1MB/s of downstream bandwith, and ~250KB/s of upstream. You can use the MAX_TERMS environment variable to process less emoji chars if you don't have the bandwidth where you are.
Note, DO NOT run the feeder process with REDIS_URL configured to the production server, ever.
This is but a small part of emojitracker's infrastructure. Major components of the project include:
- emojitrack-web the web frontend and application server
- emojitrack-feeder consumes the Twitter Streaming API and feeds our data pipeline
- emojitrack-streamer handles streaming updates to clients via SSE
- ruby version (deprecated)
- nodejs version(deprecated)
- go version (production)
- streamer API spec defines the streamer spec & tests servers in staging
Additionally, many of the libraries emojitrack uses have also been carved out into independent emoji-related open-source projects, see the following:
- emoji_data.rb utility library for handling the Emoji vs Unicode nightmare (Ruby)
- emoji-data-js utility library for handling the Emoji vs Unicode nightmare (Nodejs port)
- exmoji utility library for handling the Emoji vs Unicode nightmare (Elixir/Erlang port)
- emojistatic generates static emoji assets for a public CDN
As well as some general purpose libraries: