A little tool to pull/download HTTP, Cloudflare Access and Audit logs from Cloudflare and save it on local storage.
-
For HTTP Logs:
- You must have an active Cloudflare Enterprise zone in order to use Cloudflare Enterprise Log Share (ELS).
- Make sure the zone that you want to pull logs from already has Log Retention enabled - refer to this link on how to enable it.
- Make sure your Cloudflare user account has the permission to access Zone logs (particularly, Log Share Reader role). If you are unsure about that, contact your Administrator. If you are an Administrator already, no further action is required.
-
For Cloudflare Access Logs:
- You must have an active Cloudflare Zero Trust subscription (formerly known as Cloudflare for Teams) which includes Cloudflare Access.
- Make sure your Cloudflare user account has the permission to read Account - Access: Audit Logs. If you are unsure about that, contact your Administrator. If you are an Administrator already, no further action is required.
-
For Cloudflare Audit Logs:
- Make sure your Cloudflare user account has the permission to read Account - Account Settings. If you are unsure about that, contact your Administrator. If you are an Administrator already, no further action is required.
-
You need to create an API Token from the Cloudflare Dashboard to allow access to logs.
-
Requires root access to your local machine.
-
Currently only supports Linux. Windows isn't supported yet.
-
Download the latest version of the binary file from the GitHub Releases section.
-
Unzip the file. You should see 3 files:
cf_logs_downloader
,schema.yml
andsampleconfig.yml
. -
Make "cf_logs_downloader" executable -
chmod +x cf_logs_downloader
. -
Verify the script is working by executing
./cf_logs_downloader -v
. You should see this:Version 2.8.1
Follow the instructions below to generate an API token for HTTP Logs:
- Go to https://dash.cloudflare.com/profile/api-tokens (you need to login first, of course!)
- Click Create Token.
- Under Custom Token section, click Get Started.
- Give your API Token a name.
- Under Permissions, choose Zone - Logs - Read.
- Under Zone Resources, choose the specific zone that you want to read the logs from. Allowing access to all zones is not recommended.
- Under IP Address Filtering, enter the source IP address that will call the Cloudflare API (recommended).
- Provide a TTL to define how long this token can stay active (recommended).
- Click Continue to Summary, then click Create Token.
- Keep the generated token in a safe location.
Follow the instructions below to generate an API token for Cloudflare Access Logs:
- Go to https://dash.cloudflare.com/profile/api-tokens (you need to login first, of course!)
- Click Create Token.
- Under Custom Token section, click Get Started.
- Give your API Token a name.
- Under Permissions, choose Account - Access: Audit Logs - Read.
- Under Account Resources, choose the specific account that you want to read the logs from. Allowing access to all accounts is not recommended.
- Under IP Address Filtering, enter the source IP address that will call the Cloudflare API (recommended).
- Provide a TTL to define how long this token can stay active (recommended).
- Click Continue to Summary, then click Create Token.
- Keep the generated token in a safe location.
Follow the instructions below to generate an API token for Cloudflare Audit Logs:
- Go to https://dash.cloudflare.com/profile/api-tokens (you need to login first, of course!)
- Click Create Token.
- Under Custom Token section, click Get Started.
- Give your API Token a name.
- Under Permissions, choose Account - Account Settings - Read.
- Under Account Resources, choose the specific account that you want to read the logs from. Allowing access to all accounts is not recommended.
- Under IP Address Filtering, enter the source IP address that will call the Cloudflare API (recommended).
- Provide a TTL to define how long this token can stay active (recommended).
- Click Continue to Summary, then click Create Token.
- Keep the generated token in a safe location.
Here are the list of parameters that you can leverage within the tool:
-h, --help show this help message and exit
-c config.yml, --config config.yml
Specify the path to the YAML configuration file.
-a ACCOUNT_ID, --account ACCOUNT_ID
Specify the Cloudflare Account ID, if CF_ACCOUNT_ID
environment variable not set. This will override
CF_ACCOUNT_ID variable. Use only with 'access' or
'audit' log type.
-z ZONE_ID, --zone ZONE_ID
Specify the Cloudflare Zone ID, if CF_ZONE_ID
environment variable not set. This will override
CF_ZONE_ID variable. Use only with 'http' log type.
-t TOKEN, --token TOKEN
Specify your Cloudflare API Token, if CF_TOKEN
environment variable not set. This will override
CF_TOKEN variable.
-r RATE, --rate RATE Specify the log sampling rate from 0.01 to 1. Default
is 1. Only applicable for 'http' log type.
-i INTERVAL, --interval INTERVAL
Specify the interval between each logpull in seconds.
Default is 60 seconds.
-n NICE, --nice NICE Specify the niceness of the logpull process from -20
(highest priority) to 19 (lowest priority). Default is
-10.
--type TYPE Specify the type of logs that you would like to pull.
Possible values: http (for HTTP logs), access (for
Cloudflare Access logs), audit (for Cloudflare Audit
logs)
--path /log/path/ Specify the path to store logs. By default, it will
save to /var/log/cf_logs/.
--prefix PREFIX Specify the prefix name of the logfile being stored on
local storage. By default, the file name will begins
with cf_logs.
--no-organize Instruct the program to store raw logs as is, without
organizing them into date and time folder.
--no-gzip Do not compress the raw logs.
--hide-user-logs Enable this option if you prefer not to include user
activity logs inside the audit logs, which includes
user login histories, API-related events, etc. Only
use this parameter with '--type audit'.
--one-time Only pull logs from Cloudflare for one time, without
scheduling capability. You must specify the start time
and end time of the logs to be pulled from Cloudflare.
--start-time START_TIME
Specify the start time of the logs to be pulled from
Cloudflare. The start time is inclusive. You must
follow the ISO 8601 (RFC 3339) date format, in UTC
timezone. Example: 2020-12-31T12:34:56Z
--end-time END_TIME Specify the end time of the logs to be pulled from
Cloudflare. The end time is exclusive. You must follow
the ISO 8601 (RFC 3339) date format, in UTC timezone.
Example: 2020-12-31T12:35:00Z
--exclude field1,field2
Specify the list of log fields to be excluded from
Logpull. Separate each field by comma without spaces.
Only applicable for 'http' log type.
--available-fields TYPE
Specify the log type to display the list of available
log fields used by the program. These fields are also
included in the logpull by default (unless field
exclusion is configured). Possible values: http |
access.
--install-service Install the program as a systemd service. The service
will execute the program from the path where you
install the service.
--uninstall-service Uninstall the systemd service.
--list-queue List all the pending tasks in the queue which has
failed before, without beautifying the result (raw
JSON).
--list-queue-beauty List all the pending tasks in the queue which has
failed before, with beautifying the result.
--queue-size Display the number of pending tasks in the queue which
has failed before.
--debug Enable debugging functionality.
-v, --version Show program version.
This tool supports specifying the settings via YAML configuration file. Refer to the list below for the supported settings:
type
(string, optional) - Specify the log type. Valid values: http | access | audithide_user_logs
(boolean, optional) - Enable this option if you prefer not to include user activity logs (user login histories, API-related events, etc.) inside the audit logs. Only use this parameter with "audit" log type.cf_account_id
(string, optional) - Specify the Cloudflare Account ID (when you specify "access" or "audit" log type).cf_zone_id
(string, optional) - Specify the Cloudflare Zone ID (when you specify "http" log type).cf_token
(string, optional) - Specify the Cloudflare API Token.rate
(float, optional) - Specify log sampling rate from 0.01 to 1. Default is 1. Only applicable for "http" log type.interval
(int, optional) - Specify the interval between each logpull in seconds. Default is 60 seconds.nice
(int, optional) - Specify the niceness of the logpull process from -20 (highest priority) to 19 (lowest priority). Default is -10.debug
(boolean, optional) - Enable debugging functionality. Acceptable values:true
orfalse
.log_dest
(list, optional) - Specify this to further configure the settings for the destination of the logs. This includes multiple options as shown below:name
(string, required) - Give a unique name of the log destination configuration. Useful to identify in activity log.path
(string, required) - Specify the path to store logs. By default, it will save to /var/log/cf_logs/prefix
(string, required) - Specify the prefix name of the logfile being stored on local storage. By default, the file name will begins with cf_logs.no_organize
(boolean, required) - Instruct the program to store raw logs as is, without organizing them into date and time folder. Acceptable values:true
orfalse
.no_gzip
(boolean, required) - Do not compress the raw logs. Acceptable values:true
orfalse
.
fields.exclude
(list, optional) - Specify the list of fields you want to exclude from logpull. Only applicable for "http" log type. You can execute./cf-logs-downloader --available-fields
to retrieve the list of fields which are available to logpull.
You may refer to schema.yml for more information.
Here's the sample of the configuration settings:
type: http
# cf_account_id: your_account_id_here
cf_zone_id: your_zone_id_here
cf_token: your_token_here
rate: 0.5
interval: 30
nice: -10
debug: true
log_dest:
- name: first_dest
path: /var/log/first_path
prefix: number_one
no_organize: false
no_gzip: true
- name: second_dest
path: /var/log/second_path
prefix: number_two
no_organize: true
no_gzip: false
fields.exclude:
- ZoneID
- WAFProfile
Here are some environment variables that you can create while using this tool:
CF_LOG_TYPE
- Specify the log type.CF_ZONE_ID
- Specify the Cloudflare Zone ID.CF_ACCOUNT_ID
- Specify the Cloudflare Account ID.CF_TOKEN
- Specify the Cloudflare API Token.
Usually command line arguments will take the highest priority among the others. However, depends on the settings, some of them might have different order of precedence:
- For Log Type, Cloudflare Zone ID, Account ID and API Token: command line arguments - environment variable - configuration file
- For sample rate, logpull interval and niceness: command line arguments - configuration file - default value
- For debug option: the option will be turned on when the user specifies it either as command line arguments or inside the configuration file.
- For log path and log file name prefix: specifying this option as command line arguments will override everything specified under
log_dest
inside the configuration file. - For no organize and no gzip: specifying this option as command line arguments will override
no_gzip
andno_organize
option in each item underlog_dest
inside the configuration file.
-
At a bare minimum, you must specify the log type, Cloudflare Zone ID (for HTTP log type), Cloudflare Account ID (for Cloudflare Access / Audit log type) and API Token while using the tool. By doing so, this tool will use default values for the below configurations:
a. For HTTP log type:
- Log sampling rate: 100% (represented by 1 in Cloudflare Logpull API)
- Logpull interval: 60 seconds
- Logpull storage path:
/var/log/cf_logs/
- Log filename prefix:
cf_logs
- Enable folder organize by date and time
- Enable Gzip compression
- Niceness: -10
- No debugging
b. For Cloudflare Access / Audit log type:
- Logpull interval: 60 seconds
- Logpull storage path:
/var/log/cf_logs/
- Log filename prefix:
cf_logs
- Enable folder organize by date and time
- Enable Gzip compression
- Niceness: -10
- No debugging
-
To use the default configurations, you can use this command for logpull:
a. For HTTP log type:
$ sudo ./cf_logs_downloader.py --type http -z YOUR_ZONE_ID -t YOUR_API_TOKEN
b. For Cloudflare Access log type:
$ sudo ./cf_logs_downloader.py --type access -a YOUR_ACCOUNT_ID -t YOUR_API_TOKEN
c. For Cloudflare Audit log type:
$ sudo ./cf_logs_downloader.py --type audit -a YOUR_ACCOUNT_ID -t YOUR_API_TOKEN
Or, if you wish to use the environment variable to specify Cloudflare Zone ID and API Token, just export the environment variable with the value assigned to it (take HTTP log type as example):
$ sudo su # export CF_LOG_TYPE=http # export CF_ZONE_ID=YOUR_ZONE_ID # export CF_TOKEN=YOUR_API_TOKEN # ./cf_logs_downloader.py
Expected outcome: your logs will be stored in
/var/log/cf_logs/1970-01-01/1800/cf_logs_1970-01-01T18:00:00Z~1970-01-01T18:01:00Z.json.gz
initially. Subsequent logs will be stored in their respective folder based on date and time. -
To pull HTTP logs with 10% sampling rate and 10 seconds of interval:
$ sudo ./cf_logs_downloader.py --type http -z YOUR_ZONE_ID -t YOUR_API_TOKEN -r 0.1 -i 10
Expected outcome: your logs will be stored in
/var/log/cf_logs/1970-01-01/1800/cf_logs_1970-01-01T18:00:00Z~1970-01-01T18:00:10Z.json.gz
initially. Subsequent logs will be stored in their respective folder based on date and time. -
To instruct the tool not to save Cloudflare Access logs in compressed (gzip) format, and store the logs in a different folder:
$ sudo ./cf_logs_downloader.py --type access -a YOUR_ACCOUNT_ID -t YOUR_API_TOKEN --no-gzip --path /root/Downloads/my_cloudflare_log/
Expected outcome: your logs will be stored in
/root/Downloads/my_cloudflare_log/1970-01-01/1800/cf_logs_1970-01-01T18:00:00Z~1970-01-01T18:01:00Z.json
initially. Subsequent logs will be stored in their respective folder based on date and time. -
To pull HTTP logs at 2 minutes of interval, and instruct the tool not to organize the logs in date/time folder and use a different prefix for the log filename:
$ sudo ./cf_logs_downloader.py --type http -z YOUR_ZONE_ID -t YOUR_API_TOKEN -i 120 --no-organize --prefix my_site_log
Expected outcome: your logs will be stored in
/var/log/cf_logs/my_site_log_1970-01-01T18:00:00Z~1970-01-01T18:02:00Z.json.gz
initially. Subsequent logs will be stored in their respective folder based on date and time. -
To pull Cloudflare Access logs for just one time (without scheduling) and without organizing the log file into date/time folder:
$ sudo ./cf_logs_downloader.py --type access -a YOUR_ACCOUNT_ID -t YOUR_API_TOKEN --one-time --no-organize --start-time 2021-02-02T18:00:00Z --end-time 2021-02-02T18:30:00Z
Expected outcome: your log will be stored in
/var/log/cf_logs/cf_logs_2021-02-02T18:00:00Z~2021-02-02T18:29:59Z.json.gz
. Note that this tool will deduct 1 second from the end date specified. -
To pull HTTP logs with 50% sampling rate, 30 seconds of interval, store them in a different folder with different log filename prefix, without gzip compression and do not organize the logs into date/time folder:
$ sudo ./cf_logs_downloader.py --type http -z YOUR_ZONE_ID -t YOUR_API_TOKEN -r 0.5 -i 30 --path /home/user/cf_logging/ --prefix example_com --no-gzip --no-organize
Expected outcome: your logs will be stored in
/home/user/cf_logging/example_com_1970-01-01T18:00:00Z~1970-01-01T18:00:30Z.json
initially. Subsequent logs will be stored in their respective folder based on date and time. -
To exclude certain log fields in logpull task (e.g. ZoneID and WAFProfile), use this command:
$ sudo ./cf_logs_downloader.py -z YOUR_ZONE_ID -t YOUR_API_TOKEN --exclude ZoneID,WAFProfile
-
Specifying
--list-queue
as the parameter will display the list of failed tasks waiting for retry, but without any formatting and sorting. The output will look like this:[{"id":99,"data":{"folder_time":"2021-04-22 13:22:00","log_start_time_utc":"2021-04-22 05:22:00","log_end_time_utc":"2021-04-22 05:22:30","log_type":"http","reason":"Logpull error (HTTP 429, Cloudflare 10000 - Rate limited. Please wait and consider throttling your request speed)"},"timestamp":1619069053.252742},{"id":100,"data":{"folder_time":"2021-04-22 13:22:30","log_start_time_utc":"2021-04-22 05:22:30","log_end_time_utc":"2021-04-22 05:23:00","log_type":"http","reason":"Logpull error (HTTP 401, Cloudflare 10000 - Authentication error)"},"timestamp":1619069082.8486648},{"id":101,"data":{"folder_time":"2021-04-22 13:21:30","log_start_time_utc":"2021-04-22 05:21:30","log_end_time_utc":"2021-04-22 05:22:00","log_type":"http","reason":"Logpull error (HTTP 401, Cloudflare 10000 - Authentication error)"},"timestamp":1619069105.1588771},{"id":102,"data":{"folder_time":"2021-04-22 13:23:00","log_start_time_utc":"2021-04-22 05:23:00","log_end_time_utc":"2021-04-22 05:23:30","log_type":"http","reason":"Logpull error (HTTP 401, Cloudflare 10000 - Authentication error)"},"timestamp":1619069112.8409271}]
-
Specifying
--list-queue-beauty
as the parameter will display the list of failed tasks waiting for retry. This option provides formatting as well as sorting based on the log start time. The output will look like this:[ { "id": 99, "data": { "folder_time": "2021-04-22 13:22:00", "log_start_time_utc": "2021-04-22 05:22:00", "log_end_time_utc": "2021-04-22 05:22:30", "log_type": "http", "reason": "Logpull error (HTTP 429, Cloudflare 10000 - Rate limited. Please wait and consider throttling your request speed)" }, "timestamp": 1619069053.252742 }, { "id": 100, "data": { "folder_time": "2021-04-22 13:22:30", "log_start_time_utc": "2021-04-22 05:22:30", "log_end_time_utc": "2021-04-22 05:23:00", "log_type": "http", "reason": "Logpull error (HTTP 401, Cloudflare 10000 - Authentication error)" }, "timestamp": 1619069082.8486648 }, { "id": 101, "data": { "folder_time": "2021-04-22 13:21:30", "log_start_time_utc": "2021-04-22 05:21:30", "log_end_time_utc": "2021-04-22 05:22:00", "log_type": "http", "reason": "Logpull error (HTTP 401, Cloudflare 10000 - Authentication error)" }, "timestamp": 1619069105.1588771 }, { "id": 102, "data": { "folder_time": "2021-04-22 13:23:00", "log_start_time_utc": "2021-04-22 05:23:00", "log_end_time_utc": "2021-04-22 05:23:30", "log_type": "http", "reason": "Logpull error (HTTP 401, Cloudflare 10000 - Authentication error)" }, "timestamp": 1619069112.8409271 } ]
-
Specifying
--queue-size
as the parameter will display the number of items inside the queue. Useful to know how many failed tasks pending for retry. -
If you specify any of the parameters as listed above, all other parameters that you specified (e.g.
-z
or-t
) will be ignored.
- Currently it does not support separating queues based on Zone ID (domain). You may get unexpected behavior when you try to change the Zone ID while there are items in the queue, which is not bind to any Zone IDs.
- Currently only Cloudflare API Token can be used to authenticate against Cloudflare APIs. Global API key is not supported, as this is a more insecure option.
- All the logpull activity logs will be written in
/var/log/cf_logs_downloader/
folder. Make sure you have the appropriate permission (root) to run the script. - Each successful logpull activity will be written in
succ.log
file. - If a logpull task failed, the failed task will be put inside a queue. A separate thread will keep checking the queue for new items, and reattempt the logpull process. The thread will pick up new items after each logpull activity for every 3 seconds. If there's 3 consequtive failed logpull activities, then the thread will wait for 60 seconds before performing the next logpull activity.
- Some logpull tasks can't be retried because of known error (for example, requesting bot management field from a zone which does not have bot management enabled). In this case, the failed logpull activity will be written in
fail.log
. - If you specify
--one-time
parameter, you must specify--start-time
and--end-time
at the same time and vice versa. - For HTTP log type, the
--start-date
must be no more than 7 days earlier than now (according to Cloudflare Developers Docs). - For HTTP log type, the
--end-date
must be at least 1 minute earlier than now and later than--start-date
(according to Cloudflare Developers Docs). - For HTTP log type, the maximum range between
--start-time
and--end-time
must be 1 hour only. Otherwise, Cloudflare API calls will fail (according to Cloudflare Developers Docs).