/newrelic-python-agent

Multi-Plugin python-based Agent for NewRelic's Platform

Primary LanguagePythonBSD 3-Clause "New" or "Revised" LicenseBSD-3-Clause

NewRelic Python Agent

An agent that polls supported backend systems and submits the results to the NewRelic platform. Currently supported backend systems are:

  • Alternative PHP Cache
  • Apache HTTP Server
  • CouchDB
  • Elasticsearch
  • HAProxy
  • Memcached
  • MongoDB
  • Nginx
  • pgBouncer
  • PHP FPM
  • PostgreSQL
  • RabbitMQ
  • Redis
  • Riak
  • uWSGI

Base Requirements

The agent requires Python 2.6 or 2.7 and pip for installation. Individual plugin backends may require additional libraries and are detailed below.

Configuration File Note

The configuration file uses YAML as its format. Most tickets for non-working installs are due to configuration file formatting errors. Please make sure you are properly formatting your configuration file prior to submitting a ticket. YAML is a whitespace dependent markup format. More information on writing proper YAML can be found at http://yaml.org.

Installation Instructions

  1. Install via pip:
$ pip install newrelic-python-agent
  1. Create a newrelic user and config folder (NOTE: this step is not needed if you already have an official NewRelic agent installed)
$ useradd -m newrelic
$ NRLOGDIR=/etc/newrelic; if [ ! -d "$NRLOGDIR" ]; then mkdir $NRLOGDIR && chown -R newrelic:newrelic $NRLOGDIR && chmod -R 755 $NRLOGDIR; fi
  1. Copy the configuration file example from /opt/newrelic-python-agent/newrelic-python-agent.cfg to /etc/newrelic/newrelic-python-agent.cfg and edit the configuration in that file. (You need to add your API key and enable your agents)
$ NRCFGFILE=/etc/newrelic/newrelic-python-agent.cfg; if [[ ! -f "$NRCFGFILE" ]] ; then cp /opt/newrelic-python-agent/newrelic-python-agent.cfg $NRCFGFILE; fi
$ nano /etc/newrelic/newrelic-python-agent.cfg
  1. Make a /var/log/newrelic directory and make sure it is writable by the newrelic user:
$ NRLOGDIR=/var/log/newrelic; if [ ! -d "$NRLOGDIR" ]; then mkdir $NRLOGDIR && chown -R newrelic:newrelic $NRLOGDIR && chmod -R 755 $NRLOGDIR; fi
  1. Make a /var/run/newrelic directory and make sure it is writable by the newrelic user:
$ NRRUNDIR=/var/run/newrelic; if [ ! -d "$NRRUNDIR" ]; then mkdir $NRRUNDIR && chown -R newrelic:newrelic $NRRUNDIR && chmod -R 755 $NRRUNDIR; fi
  1. Run the app:
$ newrelic-python-agent -c PATH-TO-CONF-FILE [-f]

Where -f is to run it in the foreground instead of as a daemon.

Sample configuration and init.d scripts are installed to /opt/newrelic-python-agent in addition to a PHP script required for APC monitoring.

/opt/newrelic-python-agent/ should also contain init.d scripts for Debian (Ubuntu) and REHL. New distro version with Systemd may require differnet init scripts. On Ubuntu systems 16.04 and newer, you can try running systemctl enable newrelic-python-agent from /etc/init.d to automatically generate a Systemd service file.

Installing Additional Requirements

To use the MongoDB the mongodb library is required. For the pgBouncer or PostgreSQL plugin you must install the psycopg2 library. To easily do this, make sure you have the latest version of pip installed (https://pip.pypa.io/). This should be done after installing the agent itself:

$ pip install newrelic-python-agent[mongodb]

or:

$ pip install newrelic-python-agent[pgbouncer]

or:

$ pip install newrelic-python-agent[postgresql]

If this does not work for you, make sure you are running a recent copy of pip (>= 1.3).

Plugin Configuration Stanzas

Each plugin can support gathering data from a single or multiple targets. To support multiple targets for a plugin, you create a list of target stanzas:

plugin_name:
  - name: target_name
    host: localhost
    foo: bar
  - name: target_name
    host: localhost
    foo: bar

While you can use the multi-target format for a plugin's configuration stanza like:

plugin_name:
  - name: target_name
    host: localhost
    foo: bar

You can also use a single mapping like follows:

plugin_name:
  name: target_name
  host: localhost
  foo: bar

The fields for plugin configurations can vary due to a plugin's configuration requirements. The name value in each stanza is only required when using multiple targets in a plugin. If it is only a single target, the name will be taken from the server's hostname.

You can also use an optional description with the plugin name to define multiple discrete blocks that use the same plugin:

plugin_name:local:
  - name: local_target
    host: localhost
    foo: bar
plugin_name:remote:
  - name: remote_target
    host: remotehost
    foo: bar

This can be very useful when using dynamic config blocks. You could specify a block that's statically defined and a separate set that's dynamically changed, even though both might use the same plugin type.

Dynamic Plugins

If a plugin is a subclass of the base.ConfigPlugin class, then it is considered a config plugin, rather than a metric plugin. Instead of returning metric data, it returns config blocks to apply to the running config. The result is a dictionary that looks like:

timestamp: unix timestamp
application:
    plugin_name:desc1:
        - name: config1
          foo: bar
        - name: config2
          foo: bar
    plugin_name:desc2:

This will replace the config blocks as specified by plugin_name:desc1 and plugin_name:desc2 in the running config. If the new config for a block is empty (like plugin_name:desc2 in this example), then the config block will be removed from the running config. The entire config block is replaced by these results.

This entire result is saved and presented to the plugin the next time it runs so it can know what the previous results were. A base ConfigPlugin plugin does not require any parameters to run, but can optionally take a refresh_interval setting to determine how often it should run. It uses the timestamp of the previous results to know how long it's been since the last successful run. Specify the number of seconds to wait between runs. An example config that sets a 5 minute refresh interval:

exampleConfigPlugin:
    name: somename
    refresh_interval: 300

APC Installation Notes

Copy the apc-nrp.php script to a directory that can be served by your web server or php-fpm application. Edit the newrelic-python-agent configuration to point to the appropriate URL.

Apache HTTPd Installation Notes

Enable the HTTPd server status page in the default virtual host. The following example configuration snippet for Apache HTTPd 2.2 demonstrates how to do this:

<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
</Location>

For HTTPd 2.4, it should look something like:

<Location /server-status>
    SetHandler server-status
    Require ip 127.0.0.1
</Location>

The agent requires the extended information to parse metrics. If you are not seeing any metrics on your graphs for Apache verify that you have enabled ExtendedStatus, the default is off so you must enable it. In your global Apache HTTP configuration you need to enable exetended status using:

ExtendedStatus On

If you are monitoring Apache HTTPd via a HTTPS connection you can use the verify_ssl_cert configuration value in the httpd configuration section to disable SSL certificate verification.

Memcached Installation Notes

The memcached plugin can communicate either over UNIX domain sockets using the path configuration variable or TCP/IP using the host and port variables. Do not include both.

MongoDB Installation Notes

You need to install the pymongo driver, either by running pip install pymongo or by following the "Installing Additional Requirements" above. Each database you wish to collect metrics for must be enumerated in the configuration.

There are two configuration stanza formats for MongoDB. You must use one or the other, they can not be mixed. For non-authenticated polling, you can simply enumate the databases you would like stats from as a list:

mongodb:
  name: hostname
  host: localhost
  port: 27017
  #admin_username: foo
  #admin_password: bar
  #ssl: False
  #ssl_keyfile: /path/to/keyfile
  #ssl_certfile: /path/to/certfile
  #ssl_cert_reqs: 0  # Should be 0 for ssl.CERT_NONE, 1 for ssl.CERT_OPTIONAL, 2 for ssl.CERT_REQUIRED
  #ssl_ca_certs: /path/to/cacerts file
  databases:
    - database_name_1
    - database_name_2

If your MongoDB server requires authentication, you must provide both admin credentials and database level credentials and the stanza is formatted as a nested array:

mongodb:
  name: hostname
  host: localhost
  port: 27017
  #admin_username: foo
  #admin_password: bar
  #ssl: False
  #ssl_keyfile: /path/to/keyfile
  #ssl_certfile: /path/to/certfile
  #ssl_cert_reqs: 0  # Should be 0 for ssl.CERT_NONE, 1 for ssl.CERT_OPTIONAL, 2 for ssl.CERT_REQUIRED
  #ssl_ca_certs: /path/to/cacerts file
  databases:
    database_name_1:
      username: foo
      password: bar
    database_name_2:
      username: foo
      password: bar

Nginx Installation Notes

Enable the Nginx stub_status setting on the default site in your configuration. The following example configuration snippet for Nginx demonstates how to do this:

location /nginx_stub_status {
  stub_status on;
  allow 127.0.0.1;
  deny all;
}

If you are monitoring Nginx via a HTTPS connection you can use the verify_ssl_cert configuration value in the httpd configuration section to disable SSL certificate verification.

pgBouncer Installation Notes

The user specified must be a stats user.

PostgreSQL Installation Notes

By default, the specified user must be superuser to get PostgreSQL directory listings. To skip those checks that require superuser permissions, use the superuser: False setting in the configuration file.

Several of the checks take O(N) time where N is the number of relations in the database. If you need to use this on a database with a very large number of relations, you can skip these, using relation_stats: False.

E.g.:

postgresql:
  host: localhost
  port: 5432
  user: newrelic
  dbname: postgres
  password: newrelic
  superuser: False
  relation_stats: False

RabbitMQ Installation Notes

The user specified must have access to all virtual hosts you wish to monitor and should have either the Administrator tag or the Monitoring tag.

If you are monitoring RabbitMQ via a HTTPS connection you can use the verify_ssl_cert configuration value in the httpd configuration section to disable SSL certificate verification.

Redis Installation Notes

For Redis daemons that are password protected, add the password configuration value, otherwise omit it. The Redis configuration section allows for multiple redis servers. The syntax to poll multiple servers is in the example below.

The Redis plugin can communicate either over UNIX domain sockets using the path configuration variable or TCP/IP using the host and port variables. Do not include both.

Riak Installation Notes

If you are monitoring Riak via a HTTPS connection you can use the verify_ssl_cert configuration value in the httpd configuration section to disable SSL certificate verification.

UWSGI Installation Notes

The UWSGI plugin can communicate either over UNIX domain sockets using the path configuration variable or TCP/IP using the host and port variables. Do not include both.

Make sure you have enabled stats server in your uwsgi config.

Configuration Example

%YAML 1.2
---
Application:
  license_key: REPLACE_WITH_REAL_KEY
  poll_interval: 60

  # optional settings (and their defaults) when talking to newrelic:
  #newrelic_api_timeout: 10
  #verify_ssl_cert: true
  #skip_newrelic_upload: true
  #endpoint: https://platform-api.newrelic.com/platform/v1/metrics
  #proxy: http://localhost:8080

  apache_httpd:
     -  name: hostname1
        scheme: http
        host: localhost
        port: 80
        path: /server-status
        #verify_ssl_cert: true
     -  name: hostname2
        scheme: http
        host: localhost
        port: 80
        path: /server-status
        #verify_ssl_cert: true

  couchdb:
     -  name: localhost
        host: localhost
        port: 5984
        #verify_ssl_cert: true
        #username: foo
        #password: bar
     -  name: localhost
        host: localhost
        port: 5984
        #verify_ssl_cert: true
        #username: foo
        #password: bar

  elasticsearch:
    name: clustername
    host: localhost
    port: 9200

  haproxy:
    name: my-haproxy-server
    host: localhost
    port: 80
    path: /haproxy?stats;csv
    scheme: http
    #verify_ssl_cert: true
    #username: foo
    #password: bar

  mongodb:
    name: hostname
    host: localhost
    port: 27017
    admin_username: foo
    admin_password: bar
    databases:
      database_name_1:
        username: foo
        password: bar
      database_name_2:
        username: foo
        password: bar

  memcached:
    - name: localhost
      host: localhost
      port: 11211
      path: /path/to/unix/socket
    - name: localhost
      host: localhost
      port: 11211
      path: /path/to/unix/socket

  nginx:
    - name: hostname
      host: localhost
      port: 80
      path: /nginx_stub_status
      #verify_ssl_cert: true
    - name: hostname
      host: localhost
      port: 80
      path: /nginx_stub_status
      #verify_ssl_cert: true

  pgbouncer:
    - host: localhost
      port: 6000
      user: stats

  php_apc:
     scheme: http
     host: localhost
     port: 80
     path: /apc-nrp.php
     #username: foo
     #password: bar
     #verify_ssl_cert: t

  php_fpm:
    - name: fpm-pool
      scheme: https
      host: localhost
      port: 443
      path: /fpm_status
      query: json

  postgresql:
    - host: localhost
      port: 5432
      user: postgres
      dbname: postgres
      superuser: True

  rabbitmq:
    - name: rabbitmq@localhost
      host: localhost
      port: 15672
      username: guest
      password: guest
      #verify_ssl_cert: true
      api_path: /api

  redis:
    - name: localhost
      host: localhost
      port: 6379
      db_count: 16
      password: foobar
      #path: /var/run/redis/redis.sock
    - name: localhost
      host: localhost
      port: 6380
      db_count: 16
      password: foobar
      #path: /var/run/redis/redis.sock

  riak:
    - name: localhost
      host: localhost
      port: 8098
      #verify_ssl_cert: true

Daemon:
  user: newrelic
  pidfile: /var/run/newrelic/newrelic-python-agent.pid

Logging:
  formatters:
    verbose:
      format: '%(levelname) -10s %(asctime)s %(process)-6d %(processName) -15s %(threadName)-10s %(name) -25s %(funcName) -25s L%(lineno)-6d: %(message)s'
  handlers:
    file:
      class : logging.handlers.RotatingFileHandler
      formatter: verbose
      filename: /var/log/newrelic/newrelic-python-agent.log
      maxBytes: 10485760
      backupCount: 3
  loggers:
    newrelic_python_agent:
      level: INFO
      propagate: True
      handlers: [console, file]
    requests:
      level: ERROR
      propagate: True
      handlers: [console, file]

Troubleshooting

  • If the installation does not install the newrelic-python-agent application in /usr/bin then it is likely that setuptools or distribute is not up to date. The following commands can be run to install distribute and pip for installing the application:
$ curl http://python-distribute.org/distribute_setup.py | python
$ curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python
  • If the application installs but doesn't seem to be submitting status, check the logfile which at /tmp/newrelic-python-agent.log if the default example logging configuration is used.
  • If the agent starts but dies shortly after ensure that /var/log/newrelic and /var/run/newrelic are writable by the same user specified in the daemon section of the configuration file.
  • If the agent has died and won't restart, remove any files found in /var/run/newrelic/
  • If using the Apache HTTP plugin and your stats are blank, ensure the ExtendedStatus directive is on.