RESTBase was built to provide a low-latency & high-throughput API for Wikipedia / Wikimedia content. It is basically a storing proxy, which presents a coherent API powered by Swagger specs to the outside, and backs up many of these entry points with storage. The default table storage backend is based on Cassandra, which helps it to perform well at Wikimedia's scale without placing undue burden on operations.
As a proxy, RESTBase does not perform any significant content processing itself. Instead, it requests content transformations from backend services when needed, and typically (depending on configuration) stores it back for later retrieval. For high-volume static end points most requests will be satisfied directly from storage.
The table storage backends conform to a RESTful table storage API similar to Amazon DynamoDB and Google DataStore. The primary implementation uses Apache Cassandra. Notable features include automatically maintained secondary indexes and some lightweight transaction support. A SQLite backend is under development.
RESTBase systematically emits statsd metrics about storage and backend requests. Especially the systematic metric production for backend services provides a good baseline level of performance and error instrumentation in a micro-service architecture.
We use Phabricator to track issues. See the list of current issues in RESTBase.
From the restbase project directory, install the Node dependencies:
npm installStart RESTBase:
node serverThe defaults without a config file should work for a local Cassandra installation with the default passwords. To customize RESTBase's behavior, copy the example config to its default location:
cp config.example.yaml config.yamlYou can also pass in the path to another file with the -c commandline option
to server.js. If you're running a single Cassandra instance (e.g. a local
development environment), set defaultConsistency to one in
config.yaml.
See the Wikimedia REST content API sandbox for a fine example of what RESTBase can do.
To run all the tests from a clean slate, first make sure Cassandra is running locally, then fire up the tests with npm:
npm test
To run tests from a single file, e.g. test/features/pagecontent/rerendering.js, run mocha with the file as an argument:
mocha test/features/pagecontent/rerendering.js
Note that this might require some setup (e.g. creating the necessary domain and buckets), which is currently done by test/buckets.js.
This also works for a directory, e.g. test/features/pagecontent/:
mocha test/features/pagecontent
To check the test coverage, use npm, then browse the report:
npm run-script coverage
The coverage report can now be found in <project>/coverage/lcov-report/index.html.