Bring your Rubix ML models into production by serving them with one of our stand-alone model inference servers. Servers wrap your trained estimator in an API that can be queried locally or over the network in real-time using standard protocols. In addition, the library provides client implementations for making queries from your applications.
Install Rubix Server using Composer:
$ composer require rubix/server- PHP 7.2 or above
- Tensor extension for fast Matrix/Vector computing
- Event extension for high-volume servers
Rubix model servers are stand-alone processes that wrap an estimator in an API that can be queried over a network connection. Since servers implements their own networking stack, they can be run directly from the PHP command line interface (CLI) without the need for an intermediary server such as Nginx or Apache. By utilizing concurrency, each server instance is able to handle thousands of connections at the same time. Need more inference throughput? Model servers scale by adding more instances behind a load balancer.
To boot up a server, pass a trained estimator instance to the serve() method:
public function serve(Estimator $estimator) : voiduse Rubix\Server\HTTPServer;
use Rubix\ML\Classifiers\KNearestNeighbors;
$server = new HTTPServer('127.0.0.1', 8000);
$estimator = new KNearestNeighbors(5);
// Import a dataset
$estimator->train($dataset);
$server->serve($estimator);Or, you can load a previously trained estimator from storage and serve it like in the example below.
use Rubix\ML\PersistentModel;
use Rubix\ML\Persisters\Filesystem;
$estimator = PersistentModel::load(new Filesystem('example.model'));
$server->serve($estimator);Note: The server will stay running until the process is terminated. It is a good practice to use a process monitor such as Supervisor to start and autorestart the server in case of a failure.
To gracefully shut down the server, send a quit signal (SIGQUIT) to the process. To shut down immediately, without waiting for current connections to close, you can either send a terminate (SIGTERM) or interrupt (SIGINT) signal.
Note: Signal handling does not work in Windows environments.
For example, to shut down gracefully, first identify the server's process ID (PID) and then send the QUIT signal to it.
$ kill -s QUIT 1234Servers that implement the Verbose interface accept any PSR-3 compatible logger instance and begin logging critical information such as errors and start/stop events. To set a logger pass the PSR-3 logger instance to the setLogger() method on the server instance.
use Rubix\ML\Other\Loggers\Screen;
$server->setLogger(new Screen());A JSON over HTTP server exposing Representational State Transfer (REST) and GraphQL APIs. The HTTP Server operates with ubiquitous standards making it compatible with a wide range of systems. In addition, it provides its own web-based user interface for real-time server monitoring.
| # | Param | Default | Type | Description |
|---|---|---|---|---|
| 1 | host | '127.0.0.1' | string | The host address to bind the server to. Use '0.0.0.0' to bind to all interfaces. |
| 2 | port | 8000 | int | The network port to run the HTTP services on. |
| 3 | cert | null | string | The path to the certificate used to authenticate and encrypt the HTTP channel. |
| 4 | middlewares | [] | array | The stack of server middleware to run on each request/response. |
| 5 | max concurrent requests | 20 | int | The maximum number of requests that can be handled concurrently. |
| 6 | sse reconnect buffer | 50 | int | The maximum number of events to store in the server-sent events (SSE) reconnect buffer. |
Example
use Rubix\Server\HTTPServer;
use Rubix\Server\HTTP\Middleware\Server\AccessLogGenerator;
use Rubix\ML\Other\Loggers\Screen;
use Rubix\Server\HTTP\Middleware\Server\BasicAuthenticator;
$server = new HTTPServer('127.0.0.1', 443, '/cert.pem', [
new AccessLogGenerator(new Screen()),
new BasicAuthenticator([
'morgan' => 'secret',
'taylor' => 'secret',
]),
], 100, 100);The HTTP server exposes the following resources and their methods.
| Method | URI | Description |
|---|---|---|
| GET | / | The web user interface. |
| POST | /graphql | Query the server using GraphQL. |
| GET | /model | Return the properties of the model. |
| POST | /model/predictions | Make a set of predictions on a dataset. |
| POST | /model/probabilities | Return the joint probabilities of each sample in a dataset. |
| POST | /model/anomaly_scores | Return the anomaly scores of each sample in a dataset. |
| GET | /server/dashboard | Return the properties of the server dashboard. |
| GET | /server/dashboard/events | Subscribe to the dashboard events stream. |
| GET | /ui/server | The server dashboard interface. |
The HTTP server provides its own high-level user interface to the GraphQL API it exposes under the hood. To access the on-demand web UI, navigate to http://hostname:port (or https://hostname:port if using a secure socket connection) using your web browser. The example below is a screen capture of the server dashboard in dark mode.
This server respects the following php.ini configuration variables.
| Name | Default | Description |
|---|---|---|
| memory_limit | 128M | The maximum amount of memory the server is allowed to consume. |
| post_max_size | 8M | The maximum size of a request body the server can buffer. |
- R. Fielding et al. (2014). Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content.
HTTP middleware are processors of the incoming HTTP requests and outgoing responses produced by the request handler (or Controller). They allow the user to hook into the HTTP request/response cycle by inserting additional logic into the pipeline.
Generates an HTTP access log using a format similar to the Apache log format.
| # | Param | Default | Type | Description |
|---|---|---|---|---|
| 1 | logger | LoggerInterface | A PSR-3 logger instance. |
Example
use Rubix\Server\HTTP\Middleware\Server\AccessLog;
use Rubix\ML\Other\Loggers\Screen;
$middleware = new AccessLog(new Screen());[2020-11-04 23:10:57] INFO: 127.0.0.1 "POST /predictions HTTP/1.1" 200 140 - "Rubix ML REST Client/0.2.3"
[2020-11-04 23:11:54] INFO: 127.0.0.1 "POST /predictions/sample HTTP/1.1" 200 96 - "Rubix ML REST Client/0.2.3"An implementation of HTTP Basic Auth as described in RFC7617.
Note: This authorization strategy is only secure over an encrypted communication channel such as HTTPS with SSL or TLS.
| # | Param | Default | Type | Description |
|---|---|---|---|---|
| 1 | passwords | array | An associative map from usernames to their passwords. | |
| 2 | realm | 'auth' | string | The unique name given to the scope of permissions required for this server. |
Example
use Rubix\Server\HTTP\Middleware\Server\BasicAuthenticator;
$middleware = new BasicAuthenticator([
'morgan' => 'secret',
'taylor' => 'secret',
], 'ml models');Authenticates incoming requests using a shared key that is kept secret between the client and server. It uses the Authorization header with the Bearer prefix to indicate the shared key.
Note: This authorization strategy is only secure over an encrypted communication channel such as HTTPS with SSL or TLS.
| # | Param | Default | Type | Description |
|---|---|---|---|---|
| 1 | tokens | array | The shared secret keys (bearer tokens) used to authorize requests. | |
| 2 | realm | 'auth' | string | The unique name given to the scope of permissions required for this server. |
Example
use Rubix\Server\HTTP\Middleware\Server\SharedTokenAuthenticator;
$middleware = new SharedTokenAuthenticator([
'secret', 'another-secret',
], 'ml models');A whitelist of clients that can access the server - all other connections will be dropped.
| # | Param | Default | Type | Description |
|---|---|---|---|---|
| 1 | ips | ['127.0.0.1'] | array | An array of trusted client ip addresses. |
Example
use Rubix\Server\HTTP\Middleware\Server\TrustedClients;
$middleware = new TrustedClients([
'127.0.0.1', '192.168.4.1', '45.63.67.15',
]);Clients allow you to communicate directly with a model server using a friendly object-oriented interface inside your PHP applications. Under the hood, clients handle all the networking communication and content negotiation for you so you can write programs as if the model was directly accessible in your applications.
Return the predictions from the model:
public predict(Dataset $dataset) : arrayuse Rubix\Server\RESTClient;
$client = new RESTClient('127.0.0.1', 8080);
// Import a dataset
$predictions = $client->predict($dataset);Calculate the joint probabilities of each sample in a dataset:
public proba(Dataset $dataset) : arrayCalculate the anomaly scores of each sample in a dataset:
public score(Dataset $dataset) : arrayClients that implement the Async Client interface have asynchronous versions of all the standard client methods. All asynchronous methods return a Promises/A+ object that resolves to the return value of the response. Promises allow you to perform other work while the request is processing or to execute multiple requests in parallel. Calling the wait() method on the promise will block until the promise is resolved and return the value.
public predictAsync(Dataset $dataset) : Promise$promise = $client->predictAsync($dataset);
// Do something else
$predictions = $promise->wait();Return a promise for the probabilities predicted by the model:
public probaAsync(Dataset $dataset) : PromiseReturn a promise for the anomaly scores predicted by the model:
public scoreAsync(Dataset $dataset) : PromiseThe REST Client communicates with the HTTP Server through the JSON REST API it exposes.
Interfaces: Client, AsyncClient
| # | Param | Default | Type | Description |
|---|---|---|---|---|
| 1 | host | '127.0.0.1' | string | The IP address or hostname of the server. |
| 2 | port | 8000 | int | The network port that the HTTP server is running on. |
| 3 | secure | false | bool | Should we use an encrypted HTTP channel (HTTPS)? |
| 4 | middlewares | array | The stack of client middleware to run on each request/response. | |
| 5 | timeout | float | The number of seconds to wait before giving up on the request. | |
| 6 | verify SSL certificate | true | bool | Should we verify the server's SSL certificate? |
Example
use Rubix\Server\RESTClient;
use Rubix\Server\HTTP\Middleware\Client\BasicAuthenticator;
use Rubix\Server\HTTP\Middleware\Client\CompressRequestBody;
use Rubix\Server\HTTP\Encoders\Gzip;
$client = new RESTClient('127.0.0.1', 443, true, [
new BasicAuthenticator('user', 'password'),
new CompressRequestBody(new Gzip(1)),
], 0.0, true);Similarly to Server middleware, client middlewares are functions that hook into the request/response cycle but from the client end. Some of the server middlewares have accompanying client middleware such as Basic Authenticator and Shared Token Authenticator.
Adds the necessary authorization headers to the request using the Basic scheme.
| # | Param | Default | Type | Description |
|---|---|---|---|---|
| 1 | username | string | The username. | |
| 2 | password | string | The password. |
Example
use Rubix\Server\HTTP\Middleware\Client\BasicAuthenticator;
$middleware = new BasicAuthenticator('morgan', 'secret');Apply a compression algorithm (Encoder) to the request body.
Note: The request body must be larger than the standard max MTU (maximum transfer unit) otherwise the body will be sent uncompressed.
| # | Param | Default | Type | Description |
|---|---|---|---|---|
| 1 | encoder | Gzip | object | The encoder used to compress the request body. |
Example
use Rubix\Server\HTTP\Middleware\Client\CompressRequestBody;
use Rubix\Server\HTTP\Encoders\Deflate;
$middleware = new CompressRequestBody(new Deflate(5));Adds the necessary authorization headers to the request using the Bearer scheme.
| # | Param | Default | Type | Description |
|---|---|---|---|---|
| 1 | token | string | The shared token to authenticate the request. |
Example
use Rubix\Server\HTTP\Middleware\Client\SharedtokenAuthenticator;
$middleware = new SharedTokenAuthenticator('secret');Here you will find answers to the most frequently asked questions.
All model servers are designed to be run from the PHP command line interface (CLI). Model servers are long-running asynchronous processes that handle concurrent requests and implement their own networking stack avoiding the need for a third-party web server such as Nginx or Apache.
To run the server, you can execute your script containing the server code by entering the following on the command line.
$ php server.phpYes, model server are designed to coexist with other web servers (including other model servers) seamlessly. Just make sure that each server runs on its own unique port.
Since model servers are inference-only (i.e. they only support queries), they scale horizontally by adding more instances behind a load balancer such as Nginx.
Yes, the HTTP Server supports both Gzip and Deflate compression schemes applied to the request bodies. However, since response bodies tend to be considerably smaller compared to requests for this application, they do not support outbound message compression.
The code is licensed MIT and the documentation is licensed CC BY-NC 4.0.