logshare

logshare is a client library for Cloudflare's Enterprise Log Share (ELS) REST API. ELS allows Cloudflare customers to:

Fetch logs for a specific ray ID.
Fetch logs between timestamps.
Save logs to files.
Push log data into other tools, such as ElasticSearch or jq, to sort/filter results.

The logshare library is designed to be fast, and will stream gigabytes of logs from Cloudflare with minimal memory usage on the application side.

A CLI program called logshare-cli is also included, and is the recommended way of interacting with the library. The examples below will primarily focus on logshare-cli.

Install & Download

With a correctly installed Go environment:

# Install it onto your $GOPATH:
$ go get github.com/cloudflare/logshare/...
# Run the CLI:
$ logshare-cli <options>

You can also download pre-built Linux binaries of logshare-cli from the Releases tab on GitHub for Linux, Windows and macOS (nee OS X).

Support

Please raise an issue on this repository, and include:

The versions of logshare and logshare-cli that you are using (note: try the latest versions first).
Any error output from logshare-cli.
The expected behaviour.
Make sure to redact any API keys, email addresses and/or zone IDs when submitting an issue.

Cloudflare's support team may not be able to resolve issues with the logshare library/client.

Available Options

You can check the available options by running logshare-cli --help:

logshare-cli --help
NAME:
   logshare-cli - Fetch request logs from Cloudflare's Enterprise Log Share API

USAGE:
   logshare-cli [global options] command [command options] [arguments...]

COMMANDS:
     help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --api-key value                Your Cloudflare API key
   --api-email value              The email address associated with your Cloudflare API key and account
   --zone-id value                The zone ID of the zone you are requesting logs for
   --zone-name value              The name of the zone you are requesting logs for. logshare will automatically fetch the ID of this zone from the Cloudflare API
   --ray-id value                 The ray ID to request logs from (instead of a timestamp)
   --start-time value             The timestamp (in Unix seconds) to request logs from. Defaults to 30 minutes behind the current time (default: 1504137645)
   --end-time value               The timestamp (in Unix seconds) to request logs to. Defaults to 20 minutes behind the current time (default: 1504138245)
   --count value                  The number (count) of logs to retrieve. Pass '-1' to retrieve all logs for the given time period (default: 1)
   --by-received                  Retrieve logs by the processing time on Cloudflare. This mode allows you to fetch all available logs vs. based on the log timestamps themselves.
   --fields value                 Select specific fields to retrieve in the log response. Pass a comma-separated list to fields to specify multiple fields.
   --list-fields                  List the available log fields for use with the --fields flag
   --google-storage-bucket value  Full URI to a Google Cloud Storage Bucket to upload logs to
   --google-project-id value      Project ID of the Google Cloud Storage Bucket to upload logs to

Typically you will need the zone ID from the Cloudflare API to retrieve logs from the ELS REST API. In order to make retrieving logs more straightforward, you can provide the zone name via the --zone-name= option, and logshare-cli will fetch the relevant zone ID for this zone before retrieving logs.

Useful Tips

Although logshare-cli can be used in multiple ways, and for ingesting logs into a larger system, a common use-case is ad-hoc analysis of logs when troubleshooting or analyzing traffic. Here are a few examples that leverage jq to parse log output.

Distribution of Origin Response Status Codes

$ logshare-cli --api-key=<snip> --api-email=<snip> --zone-name=example.com --start-time=1453307871 --count=20000 | jq '.[] | .originResponse.status // empty' | sort -rn | uniq -c | sort -rn

Top 10 Visitor Countries

logshare-cli --api-key=<snip> --api-email=<snip> --zone-name=example.com --start-time=1453307871
--count=20000 | jq '. | .client.country' | uniq -c | sort -rn | head -n 10

39384 "us"
1276 "de"
 933 "ie"
 743 "gb"
 597 "ca"
 587 "in"
 528 "id"
 476 "au"
 464 "jp"
 437 "fr"

Client IPs and endpoints that were impacted by rate-limiting rules

logshare-cli --zone-name example.com --api-key $CF_API_KEY --api-email $CF_API_EMAIL --start-time 1453307871 --count 20000| jq 'select(.edge.rateLimit.processedRules | length > 0)| {ts: .timestamp, ruleID: .edge.rateLimit.processedRules[], url: .clientRequest.uri, srcIP: .client.ip}'

{
  "ts": 1503002722080000000,
  "ruleID": {
    "ruleId": 96612,
    "ruleSrc": "user",
    "status": "allow",
    "ruleType": "ban"
  },
  "url": "/download",
  "srcIP": "45.56.123.254"
}

Uploading ELS Logs to Google Cloud Storage (GCS)

logshare-cli can be used to upload logs directly to GCS. In order to do so both --google-storage-bucket and --google-project-id must be provided. This will reroute log output to a file named cloudflare_els_<zone-id>_<unix-ts>.json in the bucket/project selected. The bucket will be created if it was not already, but the project must already exist.

logshare-cli --api-key=<snip> --api-email=<snip> --zone-name=example.com --start-time 1502438905 
--count 500 --google-storage-bucket=my-bucket --google-project-id=my-project-id

Dependencies to upload to GCS

The Google Cloud SDK must be installed including the beta component along with the go library. Google Application Default Credentials should be enabled so that logshare does not need credential access to talk to GCS.

TODO:

In rough order of importance:

Tests.
More examples.
Support multiple destinations (construct a MultiWriter to allow writing to stdout, files & HTTP simultaneously)
Add a --els-bulk={url} flag that allows a bulk import into Elasticsearch.
Provide a pseudo-daemon mode that allows the client to run as a service and poll at intervals, checkpointing progress.

Feature requests and PRs are welcome.

License

BSD 3-clause licensed. See the LICENSE file for details.

derek9gag/logshare