CSV stats export /stats/realm/composite.csv gives 500 response

Question

CSV stats export /stats/realm/composite.csv gives 500 response

bschlaman opened this issue 3 years ago · 2 comments

TL;DR

/stats/realm/composite.csv gives a 500 response in production environment; composite.json working fine.

From the logs:

caller: "middleware/recovery.go:40"
logger: "middleware.Recover"
message: "http handler panic"
panic: "runtime error: invalid memory address or nil pointer dereference"

Reproduction

Went to /stats/realm/composite.csv directly or clicked on "Export as: CSV" option under the stats page.

Answer 1 · 2021-08-02T20:18:39.000Z

@mikehelmick

Answer 2 · 2021-08-02T20:20:49.000Z

Posting full log below.

{
  "insertId": "61084e8c000e06778ad92223",
  "jsonPayload": {
    "message": "internal error",
    "request_id": "0be05d5d-63ac-4c9b-b5c9-ead2fefec24a",
    "error": "panic: runtime error: invalid memory address or nil pointer dereference",
    "stacktrace": "github.com/google/exposure-notifications-verification-server/pkg/controller.InternalError\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/controller.go:78\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.Recovery.func1.1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/recovery.go:41\nruntime.gopanic\n\truntime/panic.go:965\nruntime.panicmem\n\truntime/panic.go:212\nruntime.sigpanic\n\truntime/signal_unix.go:734\ngithub.com/google/exposure-notifications-verification-server/pkg/database.CompositeStats.MarshalCSV\n\tgithub.com/google/exposure-notifications-verification-server/pkg/database/composite_stats.go:193\ngithub.com/google/exposure-notifications-verification-server/pkg/render.(*Renderer).RenderCSV\n\tgithub.com/google/exposure-notifications-verification-server/pkg/render/csv.go:35\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/stats.(*Controller).HandleComposite.func1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/stats/composite.go:88\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/ratelimit/limitware.(*Middleware).Handle.func1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/ratelimit/limitware/middleware.go:147\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.RequireMFA.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/mfa.go:67\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.RequireEmailVerified.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/email_verified.go:66\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.ProcessFirewall.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/firewall.go:65\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.RequireMembership.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/membership.go:120\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.LoadCurrentMembership.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/membership.go:88\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.RequireAuth.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/auth.go:130\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.HandleCSRF.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/csrf.go:112\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.InjectCurrentPath.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/current_path.go:43\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.buildHandler.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/sessions.go:100\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.ProcessDebug.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/debug.go:40\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/unrolled/secure.(*Secure).Handler.func1\n\tgithub.com/unrolled/secure@v1.0.9/secure.go:182\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.Recovery.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/recovery.go:46\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.PopulateLogger.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/logger.go:75\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.PopulateRequestID.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/request_id.go:44\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.ProcessLocale.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/i18n.go:148\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.PopulateTemplateVariables.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/template.go:50\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.WithObservability.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/observability.go:34\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1\n\tgithub.com/NYTimes/gziphandler@v1.1.1/gzip.go:336\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\tgithub.com/gorilla/mux@v1.8.0/mux.go:210\ngithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware.MutateMethod.func1.1\n\tgithub.com/google/exposure-notifications-verification-server/pkg/controller/middleware/method.go:37\nnet/http.HandlerFunc.ServeHTTP\n\tnet/http/server.go:2049\nnet/http.(*ServeMux).ServeHTTP\n\tnet/http/server.go:2428\ngo.opencensus.io/plugin/ochttp.(*Handler).ServeHTTP\n\tgo.opencensus.io@v0.23.0/plugin/ochttp/server.go:92\nnet/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2867\nnet/http.(*conn).serve\n\tnet/http/server.go:1932",
    "caller": "controller/controller.go:78",
    "timestamp": "2021-08-02T19:59:07.848750838Z"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "project_id": "encv-prod",
      "location": "us-central1",
      "revision_name": "server-00151-leh",
      "service_name": "server",
      "configuration_name": "server"
    }
  },
  "timestamp": "2021-08-02T19:59:08.919159Z",
  "severity": "ERROR",
  "labels": {
    "instanceId": "00bf4bf02d69b7cb23725b77aacc7dc1b528745b254dea747967360645bcec95b14daa588b9cd16f49db2fd48fdc37878577dc3543ea57a16df7222f5b"
  },
  "logName": "projects/encv-prod/logs/run.googleapis.com%2Fstderr",
  "trace": "projects/encv-prod/traces/fccfc151ec100fdde4173d8ac5994791",
  "receiveTimestamp": "2021-08-02T19:59:09.183931783Z"
}