Missing log entries

Question

Missing log entries

draganjovanovic1 opened this issue 5 years ago · 21 comments

Hello,

I've configured the sink and I can see my log entries in the GCP Console. However, very large number of my logs are not there.

Here is an example of my console log output:

{"Timestamp":"2020-01-10T11:04:25.0718940+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"e4145579f55a4894a67c1fa4935f4a04","MachineName":"Dragans-MacBook-Pro","ThreadId":10}}
{"Timestamp":"2020-01-10T11:04:25.1694000+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":200,"headers":["[Date, Fri, 10 Jan 2020 10:04:24 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 72]","[X-Correlation-Id, e4145579f55a4894a67c1fa4935f4a04]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"e4145579f55a4894a67c1fa4935f4a04","MachineName":"Dragans-MacBook-Pro","ThreadId":10}}
{"Timestamp":"2020-01-10T11:04:54.3551770+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"eb2aa0ca99654ba5b74637ed0aeeab1f","MachineName":"Dragans-MacBook-Pro","ThreadId":4}}
{"Timestamp":"2020-01-10T11:04:54.3641420+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:04:53 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, eb2aa0ca99654ba5b74637ed0aeeab1f]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"eb2aa0ca99654ba5b74637ed0aeeab1f","MachineName":"Dragans-MacBook-Pro","ThreadId":4}}
{"Timestamp":"2020-01-10T11:05:20.5734090+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"091fa7d9d00b4f229fff2b8c325f2005","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:20.5748690+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:19 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 091fa7d9d00b4f229fff2b8c325f2005]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"091fa7d9d00b4f229fff2b8c325f2005","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:21.0256940+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"83d208d31a4d49dbbc346560adf2e225","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:21.0313500+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:20 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 83d208d31a4d49dbbc346560adf2e225]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"83d208d31a4d49dbbc346560adf2e225","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:26.0542250+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"edef2fe4b3c34bd38baabecc785452b8","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:26.0562250+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:25 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, edef2fe4b3c34bd38baabecc785452b8]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"edef2fe4b3c34bd38baabecc785452b8","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:26.6089760+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"c22a8340a435417a878a8a6ca7f6325c","MachineName":"Dragans-MacBook-Pro","ThreadId":12}}
{"Timestamp":"2020-01-10T11:05:26.6104780+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:25 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, c22a8340a435417a878a8a6ca7f6325c]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"c22a8340a435417a878a8a6ca7f6325c","MachineName":"Dragans-MacBook-Pro","ThreadId":12}}
{"Timestamp":"2020-01-10T11:05:27.2000710+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"3dc394d58e044d2599b491203306a697","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:27.2015120+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:26 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 3dc394d58e044d2599b491203306a697]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"3dc394d58e044d2599b491203306a697","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:28.3005700+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"fa0d8d5de32044cc8987d9cbac81c00d","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:28.3020780+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:27 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, fa0d8d5de32044cc8987d9cbac81c00d]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"fa0d8d5de32044cc8987d9cbac81c00d","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:38.3667470+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"a832a8f417dc455090b4f54a8f6db8e1","MachineName":"Dragans-MacBook-Pro","ThreadId":21}}
{"Timestamp":"2020-01-10T11:05:38.3712410+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:37 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, a832a8f417dc455090b4f54a8f6db8e1]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"a832a8f417dc455090b4f54a8f6db8e1","MachineName":"Dragans-MacBook-Pro","ThreadId":21}}
{"Timestamp":"2020-01-10T11:05:39.0143940+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"324e3804d78a4581aa87520a6d0efe75","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:39.0161660+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:38 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 324e3804d78a4581aa87520a6d0efe75]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"324e3804d78a4581aa87520a6d0efe75","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:39.3237510+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"c781c524470e405d9f71f6c8bc278b0e","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:39.3249350+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:38 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, c781c524470e405d9f71f6c8bc278b0e]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"c781c524470e405d9f71f6c8bc278b0e","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:39.5585740+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"2a1e92a95cc54a4ea3bf9dfdf6a19f3d","MachineName":"Dragans-MacBook-Pro","ThreadId":18}}
{"Timestamp":"2020-01-10T11:05:39.5599370+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:38 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 2a1e92a95cc54a4ea3bf9dfdf6a19f3d]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"2a1e92a95cc54a4ea3bf9dfdf6a19f3d","MachineName":"Dragans-MacBook-Pro","ThreadId":18}}
{"Timestamp":"2020-01-10T11:05:39.7495670+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"e2c61b94e28548d0bc581db4c31c0deb","MachineName":"Dragans-MacBook-Pro","ThreadId":20}}
{"Timestamp":"2020-01-10T11:05:39.7506290+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:38 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, e2c61b94e28548d0bc581db4c31c0deb]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"e2c61b94e28548d0bc581db4c31c0deb","MachineName":"Dragans-MacBook-Pro","ThreadId":20}}
{"Timestamp":"2020-01-10T11:05:39.9603800+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"c0884d90d73747929b46b9d33352e01d","MachineName":"Dragans-MacBook-Pro","ThreadId":15}}
{"Timestamp":"2020-01-10T11:05:39.9611280+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:38 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, c0884d90d73747929b46b9d33352e01d]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"c0884d90d73747929b46b9d33352e01d","MachineName":"Dragans-MacBook-Pro","ThreadId":15}}
{"Timestamp":"2020-01-10T11:05:40.2113760+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"5f2891dafb51485d8b2518691a671cb4","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:40.2122700+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:39 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 5f2891dafb51485d8b2518691a671cb4]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"5f2891dafb51485d8b2518691a671cb4","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:40.4606170+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"44e9fe24f5ad4d12acdf60e530019156","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:40.4614860+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:39 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 44e9fe24f5ad4d12acdf60e530019156]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"44e9fe24f5ad4d12acdf60e530019156","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:40.6800060+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"6014c148f7be405f813a8131c0058069","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:40.6808140+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:39 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 6014c148f7be405f813a8131c0058069]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"6014c148f7be405f813a8131c0058069","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:40.9107450+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"9d2bc13ac98f4813a2f36a8d4a6092f1","MachineName":"Dragans-MacBook-Pro","ThreadId":18}}
{"Timestamp":"2020-01-10T11:05:40.9115300+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:39 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 9d2bc13ac98f4813a2f36a8d4a6092f1]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"9d2bc13ac98f4813a2f36a8d4a6092f1","MachineName":"Dragans-MacBook-Pro","ThreadId":18}}
{"Timestamp":"2020-01-10T11:05:41.1436090+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"a1f1c19ccf284b64858abec5211fdd14","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:41.1441430+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:40 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, a1f1c19ccf284b64858abec5211fdd14]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"a1f1c19ccf284b64858abec5211fdd14","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:41.3763180+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"863a7755fcc7494f8bedbe105478d0c4","MachineName":"Dragans-MacBook-Pro","ThreadId":21}}
{"Timestamp":"2020-01-10T11:05:41.3777770+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:40 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 863a7755fcc7494f8bedbe105478d0c4]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"863a7755fcc7494f8bedbe105478d0c4","MachineName":"Dragans-MacBook-Pro","ThreadId":21}}
{"Timestamp":"2020-01-10T11:05:41.6278620+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"c08bf78848d540dc9b2ba2a5de26122a","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:41.6284430+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:40 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, c08bf78848d540dc9b2ba2a5de26122a]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"c08bf78848d540dc9b2ba2a5de26122a","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:41.8885000+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"dad4d12f4d244799935f0dde459b853d","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:41.8906800+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:40 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, dad4d12f4d244799935f0dde459b853d]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"dad4d12f4d244799935f0dde459b853d","MachineName":"Dragans-MacBook-Pro","ThreadId":16}}
{"Timestamp":"2020-01-10T11:05:42.1979820+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"82c2265843764655b7994570543854d0","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:42.1984540+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:41 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 82c2265843764655b7994570543854d0]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"82c2265843764655b7994570543854d0","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:42.6854430+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/ss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"6afa4bd2f15d45fab3ab6f3f1d5a66bf","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:42.6861520+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:41 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 6afa4bd2f15d45fab3ab6f3f1d5a66bf]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"6afa4bd2f15d45fab3ab6f3f1d5a66bf","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:45.5274610+01:00","Level":"Debug","MessageTemplate":"HTTP Request; {method} {url}; {headers}; {body}","Properties":{"method":"GET","url":"http://localhost:5002/sssss","headers":["[Accept, */*]","[Host, localhost:5002]","[User-Agent, curl/7.65.3]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"2d5871b524e84501aadf4c53b8a3c8b2","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}
{"Timestamp":"2020-01-10T11:05:45.5295870+01:00","Level":"Debug","MessageTemplate":"HTTP Response; {statusCode}; {headers}; {body}","Properties":{"statusCode":401,"headers":["[Date, Fri, 10 Jan 2020 10:05:44 GMT]","[Content-Type, application/json; charset=utf-8]","[Server, Kestrel]","[Content-Length, 87]","[X-Correlation-Id, 2d5871b524e84501aadf4c53b8a3c8b2]"],"body":"n/a","SourceContext":"MyProject.ServiceLib.Web.RequestResponseLoggingMiddleware","CorrelationId":"2d5871b524e84501aadf4c53b8a3c8b2","MachineName":"Dragans-MacBook-Pro","ThreadId":17}}

and my GCP Logs viewer:

You can see that only the first entry is there and nothing more. I have tried to log much more entries and see if they will appear in the GCP but no success.

I tried to enable Serilog internal logging to check for potential errors but nothing logged there.

Here is my configuration:

    let configureLogger googleCredentialsJsonFile =
        let file = File.CreateText ("serilog_internal.log")
        Serilog.Debugging.SelfLog.Enable(TextWriter.Synchronized(file))

        let versionInfo = getVersionInfo ()
        let applicationName = versionInfo.ProductName
        let applicationVersion = versionInfo.ProductVersion

        LogContext.PushProperty("Application", applicationName) |> ignore
        LogContext.PushProperty("ApplicationVersion", applicationVersion) |> ignore

        let googleCredentialsJson = File.ReadAllText (googleCredentialsJsonFile)

        let projectId =
            Json.deserialize<{| project_id: string |}> googleCredentialsJson
            |> fun x -> x.project_id

        Log.Logger <-
          LoggerConfiguration()
            .Destructure.FSharpTypes()
            .MinimumLevel.Debug()
            .WriteTo.Console(JsonFormatter ())
            .WriteTo.GoogleCloudLogging(
                GoogleCloudLoggingSinkOptions(
                    ProjectId=projectId,
                    UseJsonOutput=true,
                    ServiceName=applicationName,
                    ServiceVersion=applicationVersion,
                    GoogleCredentialJson=googleCredentialsJson
                )
            )
            .Enrich.FromLogContext()
            .Enrich.WithMachineName()
            .Enrich.WithThreadId()
            .CreateLogger()

The googleCredentialsJson is path to the file with credentials for service account which has admin rights and I would assume it is good as long as it manages to log at least one entry.

Please advise.

Regards,
Dragan

Answer 1 · 2020-01-10T20:22:29.000Z

Are you looking at the correct project in the console? Is there any log filter or exclusion setup in stackdriver?

Try the test project in this repo and see if that works.

Answer 2 · 2020-01-10T22:00:53.000Z

It's the one and only project I have made in this GCP account, no exclusions made in stackdriver. What bothers me is that I run service, see one item logged both in console and GCP log viewer and all new items are logged in the console output only, nothing in the GCP. I would assume it should be all or none, not one and then nothing. You can see both outputs in the original post (console output and GCP screen shot).

I will give test project a try and see if there are any differences.

Answer 3 · 2020-01-12T04:09:46.000Z

This library has SourceLink enabled. You can debug the logging code in Visual Studio.

https://www.hanselman.com/blog/ExploringNETCoresSourceLinkSteppingIntoTheSourceCodeOfNuGetPackagesYouDontOwn.aspx

Answer 4 · 2020-01-13T21:32:46.000Z

@manigandham I spent quite a while time with this for now.

I have tried the test project and modified both my project and the test project to bootstrap logging with exactly the same values, service account, etc. Double checked, it's 100% the same. Test project logs all entries all the time, mine is not. Mine logs 4 entries on startup and then nothing more ☹️ No output in Serilog internal log, nothing... All Serilog NuGet packages are exactly the same versions on both projects.

The difference between the test project and mine is in netcore version, the test project is netcoreapp2.1, mine is netcoreapp3.1. The test project is C#, mine is F#.

Unfortunately I wasn't able to debug through the source since I am on macOS and VS Code. No Windows with Visual Studio available.

Answer 5 · 2020-01-13T22:42:58.000Z

This library is only a few files so copy them into your project and remove the nuget reference. it should still build and run and then you can debug directly.

Answer 6 · 2020-01-14T10:19:34.000Z

Hi,

I have found what the issue is:

Grpc.Core.RpcException: Status(StatusCode=InvalidArgument, Detail="Log name contains illegal character")
   at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Serilog.Sinks.GoogleCloudLogging.GoogleCloudLoggingSink.EmitBatchAsync(IEnumerable`1 events) in /Users/dragan/Projects/GitHub/serilog-sinks-googlecloudlogging/src/Serilog.Sinks.GoogleCloudLogging/GoogleCloudLoggingSink.cs:line 88

I solved it by setting UseSourceContextAsLogName=false but I would assume that the library needs a fix.

Interesting is that I have logged the _logNameToWrite value in the EmitBatchAsync just before _client.WriteLogEntriesAsync and it was projects/sustained-works-252716/logs/Default the same as when it initially worked (first 4 entries).

Answer 7 · 2020-01-14T15:56:03.000Z

One more thing. The UseSourceContextAsLogName=false hack worked only if I use library as a ProjectReference. When I revert to NuGet package it does not work. Unfortunately I can't debug it as a package.

Answer 8 · 2020-01-15T19:04:27.000Z

The log name has rules on allowed characters and url encoding. Try the latest version 2.3.0 which handles the formatting automatically:

https://www.nuget.org/packages/Serilog.Sinks.GoogleCloudLogging/2.3.0

Answer 9 · 2020-01-22T08:34:35.000Z

Did the latest version resolve this?

Answer 10 · 2020-01-22T08:48:14.000Z

Sorry, totally missed it. Will try and post the results.

Answer 11 · 2020-01-23T14:04:49.000Z

Hi,
I'm also facing the same issue and update to 2.3.0 didn't help.

Answer 12 · 2020-01-23T17:05:23.000Z

Thanks @Jeremiad for reporting. I am in complete mess these days and can't manage to test it.

Answer 13 · 2020-01-25T22:39:05.000Z

Hey everyone - I'm unsure what I can do to help since I've been unable to reproduce this issue and there's no response output from Stackdriver to lookup either.

Here are some options:

This library uses Serilog's BatchingSink to asynchronously flush logs in the background. Is your project running for at least a few minutes? Are you calling Log.CloseAndFlush(); at the end of your app lifecycle to ensure logs are sent?
Download this code and run the TestWeb project using the same settings and see if all the logs are available. If so then the problem is at least isolated to your project.
Test with text and json output (the UseJsonOutput setting) to see if it's specific to a format.
Enable SourceLink support and debug this library while running your project. Here's a guide to the enable the correct settings: http://www.aaronstannard.com/visual-studio-sourcelink-setup/
If you don't want to debug and need logs right away then you can also install the Stackdriver logging agent to capture any console stdout/stderr output. This agent is already on GKE and Cloud Run and you can install it for any GCE or other VMs yourself: https://cloud.google.com/logging/docs/agent/installation

Answer 14 · 2020-01-26T22:36:06.000Z

@manigandham I already migrated my services to App Engine so I get stdout logged in StackDriver for free. I stopped using the lib since I don't need it anymore. However, it used to work with UseSourceContextAsLogName=false. The problem is probably with cumbersome fully qualified class names in .NET, especially generics, and if you leave Serilog to make context name it is not good for Google Cloud Logging. The issues is definitely in the incompatible Context Name as I posted earlier (the Log name contains illegal character exception).

So the workaround for now is to use UseSourceContextAsLogName=false.

Answer 15 · 2020-01-27T07:39:37.000Z

@draganjovanovic1

That should be fixed with ver 2.3.0. From my testing, even class names with generic will log fine as seen below:

var log = _loggerFactory.CreateLogger<Dictionary<string, int>>();
log.LogWarning("testing diff source context");

If anyone else has anything to add then I can look at this some more.

Answer 16 · 2020-01-27T08:23:26.000Z

@manigandham I will test again with my project in a few days. I am currently not in the position to do that. I will post my findings and if it works I'll close the issue.

Answer 17 · 2020-01-27T08:42:43.000Z

For me it helped that i added Log.CloseAndFlush() on the code. For some reason i forgot to add it before.

Answer 18 · 2020-01-29T16:17:25.000Z

@manigandham one thing to check is that F# type names tend to have + (for classes in modules) and @ (for closures) sigils in them, so any escaping you do needs to take that into account. Those are all valid type name characters via the CIL spec, etc, etc.

EDIT: I see here that you have a whitelist of good characters, so you should be compatible already. Nice!

Answer 19 · 2020-02-14T15:04:50.000Z

Hey, I finally found some time to try it out. Tried both 2.3.0 and 2.4.0 but unfortunately it does not work for me and my F# project. Will try to checkout the latest source from your repo and debug locally as soon as I find some spare time, hopefully next week.

Answer 20 · 2020-05-13T07:38:29.000Z

Hey @draganjovanovic1 - are you still having trouble with this or did it get resolved?

Answer 21 · 2020-05-17T09:19:28.000Z

Closing for lack of reply.