wandb/server

Invalid memory error since docker image v0.19.0

byteSamurai opened this issue · 3 comments

Hey,
I use a Watchtower to get the latest versions for my images. Since v0.19. I get the attached error message. I have struggles to debug it. The only fix to truncate the data directory and start fresh. Basically, I have only reports in my instance. Submitting them again, would be a very annoying. Is there a way to debug this/fix this?

{
    "level": "INFO",
    "time": "2022-09-17T23:26:58.415804766Z",
    "info": {
        "program": "gorilla",
        "source": "mnt/ramdisk/core/services/gorilla/cmd/setup.go:1313",
        "pid": 54
    },
    "data": {
        "variables": {
            "query": ""
        },
        "rule": "",
        "locations": [],
        "userID": 4,
        "username": "censored",
        "path": [
            "reportSearch"
        ],
        "query": "query ReportSearch($query: String!) {\n  reportSearch(query: $query) {\n    edges {\n      node {\n        entityName\n        displayName\n        id\n        projectName\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n}\n",
        "operationname": "ReportSearch"
    },
    "message": "runtime error: invalid memory address or nil pointer dereference",
    "logging.googleapis.com/trace": ""
}
{
    "level": "INFO",
    "time": "2022-09-17T23:27:11.29224889Z",
    "info": {
        "program": "gorilla",
        "source": "mnt/ramdisk/core/services/gorilla/cmd/setup.go:1313",
        "pid": 54
    },
    "data": {
        "stack": [
            "goroutine 661886 [running]:",
            "github.com/wandb/core/services/gorilla.(*GraphLogger).LogPanic(0xc0002b5bf0, {0x300000000?, 0xc00104ea30?}, {0x220fd60?, 0x408dd60})",
            "\t/mnt/ramdisk/core/services/gorilla/logging.go:57 +0x71",
            "github.com/graph-gophers/graphql-go/internal/exec.execFieldSelection.func2.1()",
            "\t/mnt/ramdisk/core/services/gorilla/internal/graphql-go/internal/exec/exec.go:248 +0x8c",
            "panic({0x220fd60, 0x408dd60})",
            "\t/usr/local/go/src/runtime/panic.go:884 +0x212",
            "github.com/wandb/core/services/gorilla/api/resolver.(*QueryResolver).ReportSearch(0xc000d86700, {0x2c00818, 0xc000fefc50}, {{0x0?, 0x0?}})",
            "\t/mnt/ramdisk/core/services/gorilla/api/resolver/query.go:1501 +0x50d",
            "github.com/wandb/core/services/gorilla/api/resolver.(*cachingQuery).ReportSearch(0xc0005989c0, {0x2c00818?, 0xc000fefc50?}, {{0x0?, 0x0?}})",
            "\t/mnt/ramdisk/core/services/gorilla/api/resolver/cache_gen.go:348 +0x3f",
            "github.com/wandb/core/services/gorilla/api/resolver.(*queryAuth).ReportSearch(0xc000cb2f00, {0x2c00818?, 0xc000fefc50?}, {{0x0?, 0x228c6a0?}})",
            "\t/mnt/ramdisk/core/services/gorilla/api/resolver/auth_gen.go:628 +0x42",
            "github.com/wandb/core/services/gorilla/api/resolver.(*paramValidationQuery).ReportSearch(0xc0005943f0, {0x2c00818, 0xc000fefc50}, {{0x0?, 0x7f512f772230?}})",
            "\t/mnt/ramdisk/core/services/gorilla/api/resolver/validation_gen.go:168 +0xa7",
            "github.com/wandb/core/services/gorilla/api/resolver.(*impersonatingQuery).ReportSearch(0x10?, {0x2c00818?, 0xc000fefc50?}, {{0x0?, 0x4a04be?}})",
            "\t/mnt/ramdisk/core/services/gorilla/api/resolver/impersonation_gen.go:169 +0x33",
            "reflect.Value.call({0x2695320?, 0xc00059e360?, 0x30?}, {0x26fb995, 0x4}, {0xc000fefcb0, 0x2, 0x49baee?})",
            "\t/usr/local/go/src/reflect/value.go:584 +0x8c5",
            "reflect.Value.Call({0x2695320?, 0xc00059e360?, 0x20fb7e0?}, {0xc000fefcb0?, 0x20c4280?, 0xc0005946a8?})",
            "\t/usr/local/go/src/reflect/value.go:368 +0xbc",
            "github.com/graph-gophers/graphql-go/internal/exec.execFieldSelection.func2(0x203000?, {0x2c00818?, 0xc000fef950?}, 0xc000594648, 0xc001310e00, 0xc00156bec0, 0xc00058eeb0)",
            "\t/mnt/ramdisk/core/services/gorilla/internal/graphql-go/internal/exec/exec.go:278 +0x553",
            "github.com/graph-gophers/graphql-go/internal/exec.execFieldSelection({0x2c00818, 0xc000fef950}, 0xc0012f62a0, 0x10d50cb7915712?, 0xc001310e00, 0x0?, 0x1)",
            "\t/mnt/ramdisk/core/services/gorilla/internal/graphql-go/internal/exec/exec.go:298 +0x1d0",
            "github.com/graph-gophers/graphql-go/internal/exec.(*Request).execSelections.func1(0xc001310e00)",
            "\t/mnt/ramdisk/core/services/gorilla/internal/graphql-go/internal/exec/exec.go:102 +0x1b8",
            "created by github.com/graph-gophers/graphql-go/internal/exec.(*Request).execSelections",
            "\t/mnt/ramdisk/core/services/gorilla/internal/graphql-go/internal/exec/exec.go:98 +0x1d0",
            ""
        ],
        "value": "runtime error: invalid memory address or nil pointer dereference"
    },
    "message": "graphql: panic occurred: runtime error: invalid memory address or nil pointer dereference",
    "logging.googleapis.com/trace": ""
}
{
    "level": "INFO",
    "time": "2022-09-17T23:27:11.292810903Z",
    "info": {
        "program": "gorilla",
        "source": "mnt/ramdisk/core/services/gorilla/cmd/setup.go:1313",
        "pid": 54
    },
    "data": {
        "userID": 4,
        "username": "censored",
        "path": [
            "reportSearch"
        ],
        "query": "query ReportSearch($query: String!) {\n  reportSearch(query: $query) {\n    edges {\n      node {\n        entityName\n        displayName\n        id\n        projectName\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n}\n",
        "operationname": "ReportSearch",
        "variables": {
            "query": ""
        },
        "rule": "",
        "locations": []
    },
    "message": "runtime error: invalid memory address or nil pointer dereference",
    "logging.googleapis.com/trace": ""
}

Thanks for reporting this. We're looking into the root cause and will be releasing a fix this week.

Hey @byteSamurai!

We believe this should be resolved in our latest version - v0.20.0 : Please let us know if you still face this issue!

Thanks,
Ramit

Oh nice job! No further problem on this one. Thank you all :)