awslabs/amazon-kinesis-agent

How to use W3C_LOG_PARSE to analyse the log sending by kinesis agent

MMichael-S opened this issue · 0 comments

Hi Experts
I use kinesis agent to send Apache Log to a kinesis data stream with simple configuration.

{
"kinesis.endpoint": "kinesis.us-east-1.amazonaws.com",
...
"flows": [
{
"filePattern": "/var/log/httpd/access_log*",
"kinesisStream": "kinesis-stream-test",
"partitionKeyOption": "RANDOM"
}
]
}

So if I want to use W3C_LOG_PARSE function to parse the log data, how can I know the "character-expression" parameter like "log" as belowing?

CREATE OR REPLACE PUMP "INTERMEDIATE_STREAM_PUMP" AS
INSERT INTO "INTERMEDIATE_STREAM"
SELECT STREAM l.r.COLUMN1, l.r.COLUMN2, l.r.COLUMN3, l.r.COLUMN4, l.r.COLUMN5, l.r.COLUMN6, l.r.COLUMN7
FROM (SELECT STREAM W3C_LOG_PARSE("log", 'COMMON') FROM "SOURCE_SQL_STREAM_001" ) AS l(r);

Thanks in advance.