fiskaltrust/product-de-bring-your-own-datacenter

Failed to remove queue upload

darek-phorest opened this issue · 6 comments

darek-phorest commented

We are seeing a bunch of following warnings, we would like to know how to fix those and what might be the side effect to this:

warn: fiskaltrust.SignatureCloud.DE.Services.HelipadUploader[0] Failed to remove queue upload for cashbox cashbox. cashboxid: 917a0698-01e9-439f-a255-93f37acc652d

darek-phorest commented

hi can i get an answer for this one ? We are trying to investigate an issue with our pods memory perpetually growing and we wonder if it's related to this log entry.

TSchmiedlechner commented

Hi @darek-phorest,
sorry for the delay. This message isn't critical and should not directly be related to memory consumption. However, is it written to the log very frequently? If yes, this might be a hint for other problems, e.g. Ambassador not working correctly - as the method writing this is called when we dispose a "worker", i.e. when another instance of the Middleware was created on another pod.

If it's only showing up "here and then", I would not consider this an issue. The log level could actually be informational, we will change this in a future release.

darek-phorest commented

Hi @TSchmiedlechner

We have 10 pods in our production cluster. We see that the Failed to remove queue upload entry in the logs shows up quite frequently: almost 12K times in the last 2 days and i assume this is directly proportional to the number of cashboxes and receipts we generate. We have about 700 cashboxes now in production.

Here is a sample log snippet for one of our cashboxes:

logs-insights-results (5).csv

you will see that once worker is created all the other pods are registering the Failed to remove queue upload log entry. The worker seams to be created on the same pod so i assume the ambassador is working fine. I just don't understand what is the lifecycle of the worker per cashbox as it seams they are created multiple times during the day (no pod restarts on our cluster).

darek-phorest commented

hi @TSchmiedlechner do you have any helpful info on this? It looks like a worker for queue can be created multiple times on the same pod, what would be the reason for it ?

mgbarnech commented

Hi
do you have any update about that?
We are seeing the same logs and we would like to know if it is the expected behavior?

We have 3 byodc Pods

  • pod/byodc-7467d7bcf6-b4fnc
  • pod/byodc-7467d7bcf6-kpvxb
  • pod/byodc-7467d7bcf6-zxz9x

and each time we request a cashbox (for example /json/v1/Sign) after some inactivity period we are also seeing the followings logs

{
    "log": "\u001b[40m\u001b[1m\u001b[33mwarn\u001b[39m\u001b[22m\u001b[49m: fiskaltrust.SignatureCloud.DE.Services.HelipadUploader[0]\n      Failed to remove queue upload for cashbox cashbox. cashboxid: 9f36da62-e548-4585-8e3f-67d6008e7743\n",
    "stream": "stdout",
    "kubernetes": {
        "pod_name": "byodc-7467d7bcf6-zxz9x",
        "container_name": "byodc",
        "docker_id": "faa3c92ffd2dcebe6aad0c9553b45a3857841877eb1e90f8860f7209914c96e9",
        "container_hash": "ghcr.io/fiskaltrust/byodc-mysql-fiskaly@sha256:adde70ed0481ff88caa6a27c36541dcab9ab3e59ae8da3403fba85d8c66c2ecd",
        "container_image": "ghcr.io/fiskaltrust/byodc-mysql-fiskaly:1.3-buster"
    }
},
{
    "log": "\u001b[40m\u001b[1m\u001b[33mwarn\u001b[39m\u001b[22m\u001b[49m: fiskaltrust.SignatureCloud.DE.Services.HelipadUploader[0]\n      Failed to remove queue upload for cashbox cashbox. cashboxid: 9f36da62-e548-4585-8e3f-67d6008e7743\n",
    "stream": "stdout",
    "kubernetes": {
        "pod_name": "byodc-7467d7bcf6-kpvxb",
        "container_name": "byodc",
        "docker_id": "9d8cf62e6240a55c22e7c85ed2902ede9c7943396c365a71ff0a02945623a00f",
        "container_hash": "ghcr.io/fiskaltrust/byodc-mysql-fiskaly@sha256:adde70ed0481ff88caa6a27c36541dcab9ab3e59ae8da3403fba85d8c66c2ecd",
        "container_image": "ghcr.io/fiskaltrust/byodc-mysql-fiskaly:1.3-buster"
    }
},
{
    "log": "\u001b[40m\u001b[32minfo\u001b[39m\u001b[22m\u001b[49m: fiskaltrust.SignatureCloud.DE.Services.MiddlewareInstanceFactory[0]\n      Worker created. cashboxid: 9f36da62-e548-4585-8e3f-67d6008e7743\n",
    "stream": "stdout",
    "kubernetes": {
        "pod_name": "byodc-7467d7bcf6-b4fnc",
        "container_name": "byodc",
        "docker_id": "f79f095b67dd4d647f9782c8fc848eeed4f3705c2881b67a692ae71dd6d8acf5",
        "container_image": "ghcr.io/fiskaltrust/byodc-mysql-fiskaly:1.3-buster"
    }
}

Two of the Pods are failing and one creates the worker

StefanKert commented

Hi folks,

sorry for not getting back earlier.

TL;DR
The warning is not indicating issues, it is just noise that we will try to get rid of in one of the next releases. Basically, we are trying to remove something that has never been created.

A few words to what is going on and why this log is showing up:

The logs posted by @darek-phorest are pretty good to explain this case:

In case the Middleware instance hasn't been created on the Pod that is receiving the request (byodc-86578c674b-f6bk5) the creation of this so called worker is triggered. To ensure that the worker isn't running in any other pods (e.g. to make sure that we are not getting into any chaining issues) we are disposing it (or at least trying to do so) in the other pods. In this case we are sending dispose requests to the following pods:

  • byodc-86578c674b-2rw7b
  • byodc-86578c674b-ghm4h
  • byodc-86578c674b-np524

If there is an instance available it is disposed. If there is none we don't have to do anything.

Beside the instances that have to be disposed, we also stop the upload on these pods to ensure that we are not respawn the instances whenever we try to upload on none used pods. This is where we are currently creating this log warning. While trying to remove the instance from our uploadqueue, we are not able to remove it since it has never been created.

image

  • 👍1