/reddit-proxy

Handles coordinating the reddit requests from each service.

Primary LanguagePythonOtherNOASSERTION

Reddit Proxy

This package is used to route requests to the reddit server while staying within a rate-limit. This also handles request retrying and authorization management (staying logged in).

Environment Variables

  • USER_AGENT: The user agent string to pass for all requests
  • APPNAME: The application name for logging
  • PGHOST: Host for logging
  • PGPORT: Port for logging
  • PGDATABASE: Database name for logging
  • PGUSER: Username to login as for logging
  • PGPASSWORD: Password to login with for logging
  • AMQP_HOST: The AMQP host to connect to
  • AMQP_PORT: The AMQP port to connect on
  • AMQP_USERNAME: The username for the AMQP connection
  • AMQP_PASSWORD: The password for the AMQP connection
  • AMQP_VHOST: The AMQP virtual host
  • AMQP_QUEUE: Which AMQP queue to listen on
  • MIN_TIME_BETWEEN_REQUESTS_S: The number of seconds between requests to reddit.
  • REDDIT_USERNAME: The username for reddit
  • REDDIT_PASSWORD: The password for reddit
  • REDDIT_CLIENT_ID: The client id for the app in reddit
  • REDDIT_CLIENT_SECRET: THe client secret for the app in reddit

Folder Structure

  • main.py: The main entrypoint
  • endpoints/: Contains the requests to reddit
  • handlers/: Contains the queue request handlers

Packet Structure

Requests

{
    "type": "request_type",
    "response_queue": "my_response_queue",
    "uuid": "7c07f3c0-f62c-43a0-badc-ce89869547e2",
    "version_utc_seconds": 1581255042.707,
    "sent_at": 1581255043.000,
    "args": {},
    "style": {
        "2xx": { "operation": "copy" },
        "4xx": { "operation": "failure" },
        "5xx": { "operation": "retry", "ignore_version": false }
    },
    "ignore_version": false
}

Responses

{
    "uuid": "7c07f3c0-f62c-43a0-badc-ce89869547e2",
    "type": "copy",
    "status": 200,
    "info": {}
}

The request_type typically corresponds to an alias for a reddit endpoint (i.e., user_listing). Request types much match one of the fixed set of keys known by the reddit proxy. Request types which are prefixed with an underscore don't actually correspond to reddit API calls and are instead for configuration and monitoring purposes - these only give success and failure responses.

The style argument may be omitted to get the above behavior. The response queue may be omitted for no response. The valid operations are as follows:

  • copy: The response queue object will include the status code and the handlers response.
  • success: The response queue object will just indicate success, without the status code or body.
  • failure: The response queue object will just indicate failure, without the status code or body.
  • retry: The operation will be requeued. By default, the operation will be retried either until success, a newer version of the application connects to the given response queue, or the response is explicitly cleared. The operation will be retried with "ignore_version" set to false, unless the style specifies "ignore_version": true.

By default all operations are logged. copy and success default to TRACE level, whereas failure and retry default to WARN. Logging for a request for a given status code can be silenced by setting log_level to the special value NONE. For example, the following disables logging except on failure, which uses the INFO level:

{
    "__comment": "other fields ommitted for simplicity",
    "style": {
        "2xx": { "operation": "copy", "log_level": "NONE" },
        "4xx": { "operation": "failure", "log_level": "INFO" },
        "5xx": { "operation": "retry", "log_level": "NONE" }
    }
}

The top-level ignore_version field should be false if the request should be nacked without retries when the version is older than the newest version and true if the version field should not be checked. As a general rule of thumb, information requests should have ignore_version false whereas action requests should have ignore_version true but the retry ignore version should always be false for non-critical sections.

For example, when fetching a listing it would be complicated for the client to recall the context for why it was requesting that listing through restarts and there is no disadvantage to simply requeueing the request. However, when posting comments it's complicated for it to requeue the request compared to knowing that once the packet is sent to the queue, the bot will at least attempt to post that comment once, however if a request has previously failed we'd rather drop it than retry through restarts (in case it is failing due to a programming error).

Setting ignore version to true on retries should only happen if the code is extremely resilient, hence we're very confident only temporary issues would cause it to fail.

Prefixing the response queue with void will cause the reddit proxy to never send a response. There will be no way to confirm the success of the request.

Special Request Types

Request types prefixed with an underscore have no "style" argument as they only provide either success or failure and it's not configurable when these are returned. These operations typically affect the state of the reddit proxy and don't impact reddit itself.

  • _connect: Essentially a no-op used to change the version number. Responds success if the connecter is at least as new as the previous newest version and failure otherwise. No arguments.
  • _ping: Always returns success, useful for measuring ping or checking liveliness. Also used internally as a notification we have reached the tail of the queue since some previous event. No arguments.