alp is Access Log Profiler
You can pick your download here, and install it as follows:
sudo install <downloaded file> /usr/local/bin/alp
Install alp with Homebrew
brew install alp
Install alp with asdf and asdf-alp
asdf plugin-add alp https://github.com/asdf-community/asdf-alp.git
asdf install alp <version>
asdf global alp <version>
See: The difference between v0.4.0 and v1.0.0
$ alp --help
usage: alp [<flags>] <command> [<args> ...]
alp is the access log profiler for LTSV, JSON, and others.
Flags:
--help Show context-sensitive help (also try --help-long and --help-man).
-c, --config=CONFIG The configuration file
--file=FILE The access log file
-d, --dump=DUMP Dump profiled data as YAML
-l, --load=LOAD Load the profiled YAML data
--sort=count Output the results in sorted order
-r, --reverse Sort results in reverse order
-q, --query-string Include the URI query string.
--format=table The output format (table, markdown, tsv and csv)
--noheaders Output no header line at all (only --format=tsv, csv)
--show-footers Output footer line at all (only --format=table, markdown)
--limit=5000 The maximum number of results to display.
--location=Local Location name for the timezone
-o, --output=all Specifies the results to display, separated by commas
-m, --matching-groups=PATTERN,...
Specifies URI matching groups separated by commas
-f, --filters=FILTERS Only the logs are profiled that match the conditions
--pos=POSITION_FILE The position file
--nosave-pos Do not save position file
--version Show application version.
Commands:
help [<command>...]
Show help.
ltsv [<flags>]
Profile the logs for LTSV
json [<flags>]
Profile the logs for JSON
regexp [<flags>]
Profile the logs that match a regular expression
- Parses a log in LTSV format
- By default, the following labels are parsed:
time
- datetime
method
- HTTP Method
uri
- URI
status
- HTTP Status Code
apptime
- Response time from the upstream server
reqtime
- Request Processing Time (Response time after receiving a request)
- The
--xxx-label
option can you change the name to any label
$ cat example/logs/ltsv_access.log
time:2015-09-06T05:58:05+09:00 method:POST uri:/foo/bar?token=xxx&uuid=1234 status:200 size:12 apptime:0.057
time:2015-09-06T05:58:41+09:00 method:POST uri:/foo/bar?token=yyy status:200 size:34 apptime:0.100
time:2015-09-06T06:00:42+09:00 method:GET uri:/foo/bar?token=zzz status:200 size:56 apptime:0.123
time:2015-09-06T06:00:43+09:00 method:GET uri:/foo/bar status:400 size:15 apptime:-
time:2015-09-06T05:58:44+09:00 method:POST uri:/foo/bar?token=yyy status:200 size:34 apptime:0.234
time:2015-09-06T05:58:44+09:00 method:POST uri:/hoge/piyo?id=yyy status:200 size:34 apptime:0.234
time:2015-09-06T05:58:05+09:00 method:POST uri:/foo/bar?token=xxx&uuid=1234 status:200 size:12 apptime:0.057
time:2015-09-06T05:58:41+09:00 method:POST uri:/foo/bar?token=yyy status:200 size:34 apptime:0.100
time:2015-09-06T06:00:42+09:00 method:GET uri:/foo/bar?token=zzz status:200 size:56 apptime:0.123
time:2015-09-06T06:00:43+09:00 method:GET uri:/foo/bar status:400 size:15 apptime:-
time:2015-09-06T06:00:43+09:00 method:GET uri:/diary/entry/1234 status:200 size:15 apptime:0.135
time:2015-09-06T06:00:43+09:00 method:GET uri:/diary/entry/5678 status:200 size:30 apptime:0.432
time:2015-09-06T06:00:43+09:00 method:GET uri:/foo/bar/5xx status:504 size:15 apptime:60.000
$ cat example/logs/ltsv_access.log | alp ltsv
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD | URI | MIN | MAX | SUM | AVG | P1 | P50 | P99 | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 2 | 0 | 2 | 0 | 0 | 0 | GET | /foo/bar | 0.123 | 0.123 | 0.246 | 0.123 | 0.123 | 0.123 | 0.123 | 0.000 | 56.000 | 56.000 | 112.000 | 56.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/1234 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
| 5 | 0 | 5 | 0 | 0 | 0 | POST | /foo/bar | 0.057 | 0.234 | 0.548 | 0.110 | 0.057 | 0.100 | 0.057 | 0.065 | 12.000 | 34.000 | 126.000 | 25.200 |
| 1 | 0 | 1 | 0 | 0 | 0 | POST | /hoge/piyo | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.000 | 34.000 | 34.000 | 34.000 | 34.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/5678 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.000 | 30.000 | 30.000 | 30.000 | 30.000 |
| 1 | 0 | 0 | 0 | 0 | 1 | GET | /foo/bar/5xx | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 11 | 0 | 10 | 0 | 0 | 1 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
- Parse a log with one JSON per line
- By default, the following keys are parsed:
time
- datetime
method
- HTTP Method
uri
- URI
status
- HTTP Status Code
response_time
- Response time from the upstream server
request_time
- Request Processing Time (Response time after receiving a request)
- The
--xxx-key
option can you change the name to any key
$ cat example/logs/json_access.log
{"time":"2015-09-06T05:58:05+09:00","method":"POST","uri":"/foo/bar?token=xxx&uuid=1234","status":200,"body_bytes":12,"response_time":0.057}
{"time":"2015-09-06T05:58:41+09:00","method":"POST","uri":"/foo/bar?token=yyy","status":200,"body_bytes":34,"response_time":0.100}
{"time":"2015-09-06T06:00:42+09:00","method":"GET","uri":"/foo/bar?token=zzz","status":200,"body_bytes":56,"response_time":0.123}
{"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/foo/bar","status":400,"body_bytes":15,"response_time":"-"}
{"time":"2015-09-06T05:58:44+09:00","method":"POST","uri":"/foo/bar?token=yyy","status":200,"body_bytes":34,"response_time":0.234}
{"time":"2015-09-06T05:58:44+09:00","method":"POST","uri":"/hoge/piyo?id=yyy","status":200,"body_bytes":34,"response_time":0.234}
{"time":"2015-09-06T05:58:05+09:00","method":"POST","uri":"/foo/bar?token=xxx&uuid=1234","status":200,"body_bytes":12,"response_time":0.057}
{"time":"2015-09-06T05:58:41+09:00","method":"POST","uri":"/foo/bar?token=yyy","status":200,"body_bytes":34,"response_time":0.100}
{"time":"2015-09-06T06:00:42+09:00","method":"GET","uri":"/foo/bar?token=zzz","status":200,"body_bytes":56,"response_time":0.123}
{"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/foo/bar","status":400,"body_bytes":15,"response_time":"-"}
{"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/diary/entry/1234","status":200,"body_bytes":15,"response_time":0.135}
{"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/diary/entry/5678","status":200,"body_bytes":30,"response_time":0.432}
{"time":"2015-09-06T06:00:43+09:00","method":"GET","uri":"/foo/bar/5xx","status":504,"body_bytes":15,"response_time":60.000}
$ cat example/logs/json_access.log | alp json
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD | URI | MIN | MAX | SUM | AVG | P1 | P50 | P99 | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 2 | 0 | 2 | 0 | 0 | 0 | GET | /foo/bar | 0.123 | 0.123 | 0.246 | 0.123 | 0.123 | 0.123 | 0.123 | 0.000 | 56.000 | 56.000 | 112.000 | 56.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/1234 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
| 5 | 0 | 5 | 0 | 0 | 0 | POST | /foo/bar | 0.057 | 0.234 | 0.548 | 0.110 | 0.057 | 0.100 | 0.057 | 0.065 | 12.000 | 34.000 | 126.000 | 25.200 |
| 1 | 0 | 1 | 0 | 0 | 0 | POST | /hoge/piyo | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.000 | 34.000 | 34.000 | 34.000 | 34.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/5678 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.000 | 30.000 | 30.000 | 30.000 | 30.000 |
| 1 | 0 | 0 | 0 | 0 | 1 | GET | /foo/bar/5xx | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 11 | 0 | 10 | 0 | 0 | 1 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
- Parses the log to match the regular expression
- By default, the following named capture groups are parsed:
time
- datetime
method
- HTTP Method
uri
- URI
status
- HTTP Status Code
response_time
- Response time from the upstream server
request_time
- Request Processing Time (Response time after receiving a request)
- The
--xxx-subexp
option can you change the name to any named capture groups
$ cat example/logs/combined_access.log
127.0.0.1 - - [06/Sep/2015:05:58:05 +0900] "POST /foo/bar?token=xxx&uuid=1234 HTTP/1.1" 200 12 "-" "curl/7.54.0" "-" 0.057
127.0.0.1 - - [06/Sep/2015:05:58:41 +0900] "POST /foo/bar?token=yyy HTTP/1.1" 200 34 "-" "curl/7.54.0" "-" 0.100
127.0.0.1 - - [06/Sep/2015:06:00:42 +0900] "GET /foo/bar?token=zzz HTTP/1.1" 200 56 "-" "curl/7.54.0" "-" 0.123
127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /foo/bar HTTP/1.1" 400 15 "-" "curl/7.54.0" "-" -
127.0.0.1 - - [06/Sep/2015:05:58:44 +0900] "POST /foo/bar?token=yyy HTTP/1.1" 200 34 "-" "curl/7.54.0" "-" 0.234
127.0.0.1 - - [06/Sep/2015:05:58:44 +0900] "POST /hoge/piyo?id=yyy HTTP/1.1" 200 34 "-" "curl/7.54.0" "-" 0.234
127.0.0.1 - - [06/Sep/2015:05:58:05 +0900] "POST foo/bar?token=xxx&uuid=1234 HTTP/1.1" 200 12 "-" "curl/7.54.0" "-" 0.057
127.0.0.1 - - [06/Sep/2015:05:58:41 +0900] "POST /foo/bar?token=yyy HTTP/1.1" 200 34 "-" "curl/7.54.0" "-" 0.100
127.0.0.1 - - [06/Sep/2015:06:00:42 +0900] "GET /foo/bar?token=zzz HTTP/1.1" 200 56 "-" "curl/7.54.0" "-" 0.123
127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /foo/bar HTTP/1.1" 400 15 "-" "curl/7.54.0" "-" -
127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /diary/entry/1234 HTTP/1.1" 200 15 "-" "curl/7.54.0" "-" 0.135
127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /diary/entry/5678 HTTP/1.1" 200 30 "-" "curl/7.54.0" "-" 0.432
127.0.0.1 - - [06/Sep/2015:06:00:43 +0900] "GET /foo/bar/5xx HTTP/1.1" 504 15 "-" "curl/7.54.0" "-" 60.000
$ cat example/logs/combined_access.log | alp regexp
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD | URI | MIN | MAX | SUM | AVG | P1 | P50 | P99 | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 1 | 0 | 1 | 0 | 0 | 0 | POST | foo/bar | 0.057 | 0.057 | 0.057 | 0.057 | 0.057 | 0.057 | 0.057 | 0.000 | 12.000 | 12.000 | 12.000 | 12.000 |
| 2 | 0 | 2 | 0 | 0 | 0 | GET | /foo/bar | 0.123 | 0.123 | 0.246 | 0.123 | 0.123 | 0.123 | 0.123 | 0.000 | 56.000 | 56.000 | 112.000 | 56.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/1234 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
| 4 | 0 | 4 | 0 | 0 | 0 | POST | /foo/bar | 0.057 | 0.234 | 0.491 | 0.123 | 0.057 | 0.100 | 0.234 | 0.067 | 12.000 | 34.000 | 114.000 | 28.500 |
| 1 | 0 | 1 | 0 | 0 | 0 | POST | /hoge/piyo | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.234 | 0.000 | 34.000 | 34.000 | 34.000 | 34.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/5678 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.000 | 30.000 | 30.000 | 30.000 | 30.000 |
| 1 | 0 | 0 | 0 | 0 | 1 | GET | /foo/bar/5xx | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 60.000 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
| 11 | 0 | 10 | 0 | 0 | 1 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+--------+--------+--------+--------+--------+--------+--------+--------+-----------+-----------+-----------+-----------+
See: Usage samples
-c, --config
- The configuration file
- YAML
--file=FILE
- The access log file
-d, --dump=DUMP
- File path for creating the profile results to a file
-l, --load=LOAD
- File path to read the results of the profile created with the
-d, --dump
option - Can expect it to work fast if you change the
--sort
and--reverse
options for the same profile results
- File path to read the results of the profile created with the
--sort=count
- Output the results in sorted order
- Sort in ascending order
max
,min
,sum
,avg
max-body
,min-body
,sum-body
,avg-body
p1
,p50
,p99
,stddev
uri
method
count
- The default is
count
-r, --reverse
- Sort in desecending order
-q, --query-string
- URIs up to and including query strings are included in the profile
--format=table
- Print the profile results in a table, Markdown, TSV and CSV format
- The default is table format
--noheaders
- Print no header when TSV and CSV format
--show-footers
- Print the total number of each 1xx ~ 5xx in the footer of the table or Markdown format
--limit=5000
- Maximum number of profile results to be printed
- This setting is to avoid using too much memory
- The default is 5000 lines
--location="Local"
- The timezone of the time specified in the filter condition.
- Default is localhost timezone
-o, --output="all"
- Specify the profile results to be print, separated by commas
count
,1xx
,2xx
,3xx
,4xx
,5xx
,method
,uri
,min
,max
,sum
,avg
,p1
,p50
,p99
,stddev
,min_body
,max_body
,sum_body
,avg_body
- The default is
all
-m, --matching-groups=PATTERN,...
- Treat URIs that match regular expressions as the same URI
- See URI matching groups
-f, --filters=FILTERS
- Filters the targets for profile
- See Filter
--pos=POSITION_FILE
- Stores the number of bytes to which the file has been read.
- If the number of bytes is stored in the POSITION_FILE, the data after that number of bytes will be profiled
- You can profile without truncating the file
- Also, it is expected to work fast because it seeks and skips files
--nosave-pos
- Data after the number of bytes specified by
--pos
is profiled, but the number of bytes reads is not stored
- Data after the number of bytes specified by
Consider the following cases like /diary/entry/1234
and /diary/entry/5678
.
If you simply profile URIs with different parameters on the same route, they will be profiled by parameter, but you may want to profile them by the route.
$ cat example/logs/ltsv_access.log | alp ltsv --filters "Uri matches '^/diary/entry'"
+-------+-----+-----+-----+-----+-----+--------+-------------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
| COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD | URI | MIN | MAX | SUM | AVG | P1 | P50 | P99 | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
+-------+-----+-----+-----+-----+-----+--------+-------------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/1234 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.135 | 0.000 | 15.000 | 15.000 | 15.000 | 15.000 |
| 1 | 0 | 1 | 0 | 0 | 0 | GET | /diary/entry/5678 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.432 | 0.000 | 30.000 | 30.000 | 30.000 | 30.000 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
| 2 | 0 | 2 | 0 | 0 | 0 |
+-------+-----+-----+-----+-----+-----+--------+-------------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
$ cat example/logs/ltsv_access.log | alp ltsv --filters "Uri matches '^/diary/entry'" -m "/diary/entry/.+"
+-------+-----+-----+-----+-----+-----+--------+-----------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
| COUNT | 1XX | 2XX | 3XX | 4XX | 5XX | METHOD | URI | MIN | MAX | SUM | AVG | P1 | P50 | P99 | STDDEV | MIN(BODY) | MAX(BODY) | SUM(BODY) | AVG(BODY) |
+-------+-----+-----+-----+-----+-----+--------+-----------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
| 2 | 0 | 2 | 0 | 0 | 0 | GET | /diary/entry/.+ | 0.135 | 0.432 | 0.567 | 0.283 | 0.135 | 0.135 | 0.135 | 0.148 | 15.000 | 30.000 | 45.000 | 22.500 |
+-------+-----+-----+-----+-----+-----+--------+-----------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
| 2 | 0 | 2 | 0 | 0 | 0 |
+-------+-----+-----+-----+-----+-----+--------+-----------------+-------+-------+-------+-------+-------+-------+-------+--------+-----------+-----------+-----------+-----------+
In such a case, there is an option -m, --matching-groups=PATTERN,...
.
You can also specify multiple items separated by commas.
It is a function to include or exclude targets according to the conditions.
Filter on the following variables:.
Uri
- URI
Method
- HTTP Method
Time
- Datetime string
- See: https://github.com/tkuchiki/parsetime
ResponseTime
- Response time
BodyBytes
- Bytes of HTTP Body
Status
- HTTP Status Code
The following operators are available:.
+
,-
,*
,/
,%
,**(pow)
==
,!=
,<
,>
,<=
,>=
not
,!
and
,&&
or
,||
matches
- e.g.
Uri matches "PATTERN"
not(Uri matches "PATTERN")
- e.g.
contains
- e.g.
Uri contains "STRING"
not(Uri contains "STRING")
- e.g.
startsWith
- e.g.
Uri startsWith "PREFIX"
not(Uri startsWith "PREFIX")
- e.g.
endsWith
- e.g.
Uri endsWith "SUFFIX"
not(Uri endsWith "SUFFIX")
- e.g.
in
- e.g.
Method in ["GET", "POST"]
Method not in ["GET", "POST"]
- e.g.
See: https://github.com/antonmedv/expr/blob/master/docs/Language-Definition.md
TimeAgo(duration)
- now -
duration
ns
,us or µs
,ms
,s
,m
,h
- e.g.
Time >= TimeAgo("5m")
- now -
BetweenTime(val, start, end)
- Like SQL's
BETWEEN
, returnsstart <= val && val <= end
- e.g.
BetweenTime(Time, "2019-08-06T00:00:00", "2019-08-06T00:05:00")
- Like SQL's
See: Usage samples
Donations are welcome as always!
My Kyash account is tkuchiki
.