lambda-python: A Python repository from SignalFx

End of Support (EOS) Notice ⚠️

The SignalFx Python Lambda Wrapper has reached End of Support and has been permanently archived.

Splunk OpenTelemetry Lambda is the successor. To learn more about the Splunk OTel Lambda Layer, see Instrument your Lambda function.

SignalFx Python Lambda Wrapper

Overview

You can use this document to add a SignalFx wrapper to your AWS Lambda for Python.

The SignalFx Python Lambda Wrapper wraps around an AWS Lambda Python function handler, which allows metrics and traces to be sent to SignalFx.

At a high-level, to add a SignalFx Python Lambda wrapper, you can package the code yourself, or you can use a Lambda layer containing the wrapper and then attach the layer to a Lambda function.

To learn more about Lambda Layers, please visit the AWS documentation site and see [AWS Lambda Layers](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html).

Step 1: Add the Lambda wrapper in AWS

To add the SignalFx wrapper, you have the following options:

Option 1: In AWS, create a Lambda function, then attach a SignalFx-hosted layer with a wrapper.

If you are already using Lambda layers, then SignalFx recommends that you follow this option.

In this option, you will use a Lambda layer created and hosted by SignalFx.

Option 2: In AWS, create a Lambda function, then create and attach a layer based on a SignalFx SAM (Serverless Application Model) template.

If you are already using Lambda layers, then SignalFx also recommends that you follow this option.

In this option, you will choose a SignalFx template, and then deploy a copy of the layer.

Option 3: Use the wrapper as a regular dependency, and then create a Lambda function based on your artifact containing both code and dependencies.

Option 1: Create a Lambda function, then attach the SignalFx-hosted Lambda layer

In this option, you will use a Lambda layer created and hosted by SignalFx.

To verify compatibility, review the list of supported regions. See [Lambda Layer Versions](https://github.com/signalfx/lambda-layer-versions/blob/master/python/PYTHON.md).
Open your AWS console.
In the landing page, under Compute, click Lambda.
Click Create function to create a layer with SignalFx's capabilities.
Click Author from scratch.
In Function name, enter a descriptive name for the wrapper.
In Runtime, select the desired language.
Click Create function.
Click on Layers, then add a layer.
Mark Provide a layer version.
Enter an ARN number.

To locate the ARN number, see [Lambda Layer Versions](https://github.com/signalfx/lambda-layer-versions/blob/master/python/PYTHON.md).

Option 2: Create a Lambda function, then create and attach a layer based on a SignalFx template

In this option, you will choose a SignalFx template, and then deploy a copy of the layer.

Open your AWS console.
In the landing page, under Compute, click Lambda.
Click Create function to create a layer with SignalFx's capabilities.
Click Browse serverless app repository.
Click Public applications.
In the search field, enter and select wrapper-python-sfx-app.
Review the template, permissions, licenses, and then click Deploy.
- A copy of the layer will now be deployed into your account.
Return to the previous screen to add a layer to the function, select from list of runtime compatible layers, and then select the name of the copy.

Option 3: Install the wrapper package with pip

Run the following installation script in your command line:

pip install signalfx_lambda

Step 2: Locate and set the ingest endpoint

By default, this function wrapper will send data to the us0 realm. As a result, if you are not in us0 realm and you want to use the ingest endpoint directly, then you must explicitly set your realm. To set your realm, use a subdomain, such as ingest.us1.signalfx.com or ingest.eu0.signalfx.com.

To locate your realm:

Open SignalFx and in the top, right corner, click your profile icon.
Click My Profile.
Next to Organizations, review the listed realm.

Step 3: Set environment variables

Set SIGNALFX_ACCESS_TOKEN with your correct access token. Review the following example.

SIGNALFX_ACCESS_TOKEN=access token

If you use Smart Gateway, or want to ingest directly from a realm other than us0, then you must set at least one endpoint variable. (For environment variables, SignalFx defaults to the us0 realm. As a result, if you are not in the us0 realm, you may need to set your environment variables.) There are two options:

Option 1

Set SIGNALFX_TRACING_URL and SIGNALFX_METRICS_URL to configure where traces and metrics will be sent to. The following example will send traces to the gateway and metrics directly to the ingest endpoint.

SIGNALFX_METRICS_URL=https://ingest.us0.signalfx.com
SIGNALFX_TRACING_URL=http://<my_gateway>:8080/v1/trace

To learn more, see:

[Deploying the SignalFx Smart Gateway](https://docs.signalfx.com/en/latest/apm/apm-deployment/smart-gateway.html)

(Optional) Set additional environment variables. Review the following examples.

SIGNALFX_SEND_TIMEOUT=timeout in seconds for sending datapoint [ default: 0.3 ]
SIGNALFX_TRACING_URL=tracing endpoint [ default: https://ingest.signalfx.com/v1/trace ]

Step 4: Wrap a function

There are two wrappers provided.

The decorators can be used individually or together.

For metrics, decorate your handler with @signalfx_lambda.emits_metrics. Review the following example.

import signalfx_lambda

@signalfx_lambda.emits_metrics()
def handler(event, context):
    # your code

For tracing, decorate your handler with @signalfx_lambda.is_traced. Review the following example.

import signalfx_lambda

@signalfx_lambda.is_traced()
def handler(event, context):
    # your code

Optionally, you can tell the wrapper to not auto-create a span but still initialize tracing for manual usage.

This is useful when processing SQS messages and you want each message to tie to the trace from producer that emitted the message.

import signalfx_lambda

@signalfx_lambda.is_traced(with_span=False)
def handler(event, context):
    for record in event.get('Records', []):
        with signalfx_lambda.tracing.create_span(record, context):
            # your code to process record

Step 5: Send custom metrics from a Lambda function

To send custom metrics from a Lambda function, include the following code in your function:

import signalfx_lambda

# sending application_performance metric with value 100 and dimension abc:def
signalfx_lambda.send_gauge('application_performance', 100, {'abc':'def'})

# sending counter metric with no dimension
signalfx_lambda.send_counter('database_calls', 1)

Step 6: Add tracing to the Lambda function

To trace critical parts of your handler function, include the following code in your function:

import opentracing

@signalfx_lambda.is_traced()
def some_function():
    # opentracing.tracer must be referenced from within
    # a function decorated with the is_traced() decorator
    # or it'll not reference the correct tracer initialized
    # for the lambda function.
    tracer = opentracing.tracer
    with tracer.start_active_span("span_name", tags=tags) as scope:

        # do some work

        span = scope.span
        span.set_tag("example_tag", "example_value")

To review more examples and usage details, see [Jaeger Python Tracer](https://github.com/signalfx/jaeger-client-python>).

Propagating trace context to outgoing requests or lambda response

The library ships a helper function to inject tracing context headers into a dictionary like object. The function accepts two arguments. First argument must be a dictionary like object that the trace context is injected into. The second argument is optional and must be a OpenTracing span context. If one is not provided, the function uses the currently active span. Example:

import signalfx_lambda

@signalfx_lambda.is_traced()
def handler(event, context):
    headers = {}

    # inject trace context into the headers dictionary
    signalfx_lambda.tracing.inject(headers)

    # Or inject traces from a specific span context instead of the
    # one from the active scope.
    # signalfx_lambda.tracing.inject(headers, span.context)

    request = urllib.request.Request('http://some-service', headers=headers)
    response = urllib.request.urlopen(request)

    # your code

Additional information

Metrics and dimensions sent by the metrics wrapper

The Lambda wrapper sends the following metrics to SignalFx:

Metric Name	Type	Description
function.invocations	Counter	Count number of Lambda invocations
function.cold_starts	Counter	Count number of cold starts
function.errors	Counter	Count number of errors from underlying Lambda handler
function.duration	Gauge	Milliseconds in execution time of underlying Lambda handler

The Lambda wrapper adds the following dimensions to all data points sent to SignalFx:

Dimension	Description
lambda_arn	ARN of the Lambda function instance
aws_region	AWS Region
aws_account_id	AWS Account ID
aws_function_name	AWS Function Name
aws_function_version	AWS Function Version
aws_function_qualifier	AWS Function Version Qualifier (version or version alias if it is not an event source mapping Lambda invocation)
event_source_mappings	AWS Function Name (if it is an event source mapping Lambda invocation)
aws_execution_env	AWS execution environment (e.g. AWS_Lambda_python3.6)
function_wrapper_version	SignalFx function wrapper qualifier (e.g. signalfx_lambda_0.0.2)
metric_source	The literal value of ‘lambda_wrapper’

Tags sent by the tracing wrapper

The tracing wrapper creates a span for the wrapper handler. This span contains the following tags:

Tag	Description
aws_request_id	AWS Request ID
lambda_arn	ARN of the Lambda function instance
aws_region	AWS Region
aws_account_id	AWS Account ID
aws_function_name	AWS Function Name
aws_function_version	AWS Function Version
aws_function_qualifier	AWS Function Version Qualifier (version or version alias if it is not an event source mapping Lambda invocation)
event_source_mappings	AWS Function Name (if it is an event source mapping Lambda invocation)
aws_execution_env	AWS execution environment (e.g. AWS_Lambda_python3.6)
function_wrapper_version	SignalFx function wrapper qualifier (e.g. signalfx_lambda_0.0.2)
component	The literal value of ‘python-lambda-wrapper’

Auto-instrumentation packages

The SignalFx Python Lambda Wrapper can automatically instrument supported packages. All you need to do is to install instrumentations you need in addition to signalfx_lambda. Below is a list of all instrumentation packages supported:

Library/Framework	Instrumentation Package
celery	`signalfx-instrumentation-celery`
django	`signalfx-instrumentation-django`
elasticsearch	`signalfx-instrumentation-elasticsearch`
flask	`signalfx-instrumentation-flask`
psycopg	`signalfx-instrumentation-dbapi`
pymongo	`signalfx-instrumentation-pymongo`
pymysql	`signalfx-instrumentation-dbapi`
redis	`signalfx-instrumentation-redis`
requests	`signalfx-instrumentation-requests`
tornado	`signalfx-instrumentation-tornado`

For example, if your Lambda function uses requests, then you should add https://github.com/signalfx/python-requests/archive/v0.2.0post1.zip#egg=requests-opentracing to your requirements.txt file or make sure it gets installed into the Lambda environment.

Test locally

If you would like to test changes to a wrapper, run the following commands in your command line:

pip install python-lambda-local

python-lambda-local tests/test.py tests/event.json -a 'arn:aws:lambda:us-east-1:accountId:function:functionNamePython:$LATEST'

Publish a new version

If you would like to publish a new version, run the following command in your command line to install a new Python package (build a wheel):

python setup.py bdist_wheel --universal

signalfx/lambda-python