Request Body logging

Question

Request Body logging

Closed this issue 6 years ago · 14 comments

Title: Request Body logging

Description:

how can i log the content of a request / the request-body with Envoy ?
I want to send the body content (decoded) lets say to elastic-search or write in to a file.

One could use lua-script to get the request object and send the request to an upstream cluster !

Or are there better ways for doing this ?

thanks

Answer 1 · 2018-10-15T20:20:16.000Z

I am facing a similar issue. For now I have configured a route with a generic "/*" prefix plus shadow annotation so that all incoming HTTP requests are forwarded from Envoy to a Logstash instance that in turn ingests data to an Elasticsearch cluster:

{
    "clusters": [],
    "prefix": "/*",
    "prefix_rewrite": "/",
    "shadow": {
        "name": "cluster_shadow_logstash_default_svc_cluster_local_8080"
    }

This works fine for HTTP requests, but responses are not forwarded. I would like to avoid custom script execution on Envoy and just forward all incoming request/response messages, ideally delegating any kind of processing (e.g. pair aggregation by header-id) to Logstash.

Thanks in advance.

Answer 2 · 2018-10-27T15:24:05.000Z

Anyone wants to share a lua script that forwards the requests?

Answer 3 · 2018-10-29T16:34:56.000Z

Hi ,

this works for me. I forwarding the request to fluentd service.

           ` inline_code :  

            function envoy_on_request(request_handle)
            
            local body = request_handle:body()
            result = tostring(body)
            request_handle:logDebug(result)
            
            -- Make an HTTP call.
            local headers, body = request_handle:httpCall(
            "fluentd",
            {
             [":method"] = "POST",
             [":path"] = "/",
             [":authority"] = "lua_cluster"
            },
            result,
            24224)
              
            end
            function envoy_on_response(response_handle)
              -- Make an HTTP call here too
            end  `

Answer 4 · 2018-11-06T18:13:22.000Z

How have you defined fluentd? Is it a separate service with a ClusterId (kubernates)? I'm wondering how the envoy.yaml is picking it up.

Answer 5 · 2018-11-09T12:50:23.000Z

Hi marimarkov,

fluentd is defined inside cluster section of static_resources:

clusters:
- name: fluentd
    connect_timeout: 0.25s
    type: strict_dns # static
    lb_policy: round_robin
    hosts:
    - socket_address:
        address: fluentd
        port_value: 24224

Answer 6 · 2018-11-09T12:52:54.000Z

closing this issue because it seems that right now request/response body logging is not supported .

Answer 7 · 2019-05-29T00:56:53.000Z

@githubYasser do we have any updates on it? Can we capture request/response from envoy now?

Answer 8 · 2019-09-20T09:19:42.000Z

@githubYasser

Thanks for the nice suggestion 👏

My question is, below line returns Lua's userdata, which means that has few useful information for debugging since it shows just a single pointer.

            local body = request_handle:body()
            result = tostring(body)
            request_handle:logDebug(result)

This line will show something like this:

[2019-09-20 08:58:47.242][13][debug][lua] [source/extensions/filters/http/lua/lua_filter.cc:589] script log: userdata: 0x418dcf88

And, even if you send the data to fluentd, I though it will be "userdata: 0x418dcf88" string, correct?

I think, what you might want is something like this:

                  local index = 0
                  for chunk in handle:bodyChunks() do
                    local len = chunk:length()
                    local result = chunk:getBytes(index, len)
                    index = index + len

                    handle:logDebug(result)

                    -- Make an HTTP call.
                    local headers, body = handle:httpCall(
                     "fluentd",
                        {
                          [":method"] = "POST",
                          [":path"] = "/",
                          [":authority"] = "lua_cluster"
                        },
                      result,
                      24224)
                  end

You can use getBytes to read the real value from the pointers.

Answer 9 · 2019-09-20T13:40:49.000Z

FYI

I created a reproducible repository to show how to log HTTP POST body using Lua's HTTP Filter.

~~https://github.com/kenju/envoy-lua-filters-sample~~
moved 👉 https://github.com/kenju/service-mesh-patterns/tree/c22ae3850c2b34515c88a91459542af872d55e24/envoy-lua-filters

Hope it helps anyone who has same problems with us.

Answer 10 · 2020-01-16T04:48:59.000Z

@kenju
your repo is private, we could not have access

Answer 11 · 2020-01-16T05:02:01.000Z

@scottfield

I've merged into another repository. Thanks for your comment.

Here is the latest one:

https://github.com/kenju/service-mesh-patterns/tree/c22ae3850c2b34515c88a91459542af872d55e24/envoy-lua-filters

and here is the lua filter

https://github.com/kenju/service-mesh-patterns/blob/c22ae3850c2b34515c88a91459542af872d55e24/envoy-lua-filters/envoy/lib/mylibrary.lua

Answer 12 · 2020-11-16T12:21:03.000Z

@kenju

                  local index = 0
                  for chunk in handle:bodyChunks() do
                    local len = chunk:length()
                    local result = chunk:getBytes(index, len)
                    index = index + len

                    handle:logDebug(result)

                    -- Make an HTTP call.
                    local headers, body = handle:httpCall(
                     "fluentd",
                        {
                          [":method"] = "POST",
                          [":path"] = "/",
                          [":authority"] = "lua_cluster"
                        },
                      result,
                      24224)
                  end

for each chunk from handle:bodyChunks(), read from index to len, why should index update to index + len ? per my understanding index should always be 0, any ref about this?

Answer 13 · 2021-05-19T03:54:47.000Z

local index = 0
                  for chunk in handle:bodyChunks() do
                    local len = chunk:length()
                    local result = chunk:getBytes(index, len)
                    index = index + len

                    handle:logDebug(result)

                    -- Make an HTTP call.
                    local headers, body = handle:httpCall(
                     "fluentd",
                        {
                          [":method"] = "POST",
                          [":path"] = "/",
                          [":authority"] = "lua_cluster"
                        },
                      result,
                      24224)
                  end

I agree with @xring

Answer 14 · 2022-03-11T09:51:05.000Z

@johnzheng1975 i tried the same way to print the requests on the console. it prints nothing and no error also on the console. am I missing anything here ?

               local index = 0
              for chunk in handle:bodyChunks() do
                local len = chunk:length()
                local result = chunk:getBytes(index, len)
                index = index + len

                handle:logInfo(result)
              end