Events are missing on newer k8s version

[DaveOHenry]

I spotted some missing events in an AKS cluster:

{
    "action": "Scheduling",
    "apiVersion": "v1",
    "eventTime": "2021-11-22T13:23:54.299870Z",
    "firstTimestamp": null,
    "involvedObject": {
        "apiVersion": "v1",
        "kind": "Pod",
        "name": "dummy-app-5d7b5796cf-zdz7z",
        "namespace": "foobar",
        "resourceVersion": "48010215",
        "uid": "e339d139-a27e-4ccc-8737-0db3a9620d6c"
    },
    "kind": "Event",
    "lastTimestamp": null,
    "message": "0/3 nodes are available: 3 node(s) didn't match pod affinity/anti-affinity rules, 3 node(s) didn't match pod anti-affinity rules.",
    "metadata": {
        "creationTimestamp": "2021-11-22T13:23:54Z",
        "name": "dummy-app-5d7b5796cf-zdz7z.16b9e1185e1665b8",
        "namespace": "foobar",
        "resourceVersion": "48010220",
        "uid": "205d3081-293e-473b-bf1a-8bb96c179606"
    },
    "reason": "FailedScheduling",
    "reportingComponent": "default-scheduler",
    "reportingInstance": "default-scheduler-kube-scheduler-v2-1234567890-hmccv",
    "source": {},
    "type": "Warning"
}

and

{
    "action": "Binding",
    "apiVersion": "v1",
    "eventTime": "2021-11-22T13:23:59.144478Z",
    "firstTimestamp": null,
    "involvedObject": {
        "apiVersion": "v1",
        "kind": "Pod",
        "name": "dummy-app-5d7b5796cf-zdz7z",
        "namespace": "foobar",
        "resourceVersion": "48010219",
        "uid": "e339d139-a27e-4ccc-8737-0db3a9620d6c"
    },
    "kind": "Event",
    "lastTimestamp": null,
    "message": "Successfully assigned foobar/dummy-app-5d7b5796cf-zdz7z to aks-system00-12345678-vmss000815",
    "metadata": {
        "creationTimestamp": "2021-11-22T13:23:59Z",
        "name": "dummy-app-5d7b5796cf-zdz7z.16b9e1197ed94a53",
        "namespace": "foobar",
        "resourceVersion": "48010246",
        "uid": "55631036-8044-4f91-8ffc-088e6dd88e2e"
    },
    "reason": "Scheduled",
    "reportingComponent": "default-scheduler",
    "reportingInstance": "default-scheduler-kube-scheduler-v2-1234567890-hmccv",
    "source": {},
    "type": "Normal"
}

Note the null values on firstTimestamp and lastTimestamp. I guess this is the issue, because other events that are correctly exported contain values for these fields. As far as I understand these fields are deprecated in favour of the eventTime field (that is most likely not available in older k8s versions).

Tested v0.9 and it's catching events correctly for me. Looks like the issue was introduced here:
f50b41b#diff-e296c9ee9b82d141eedf4896aa413ddff6906c72affb8f10159cee74dd911f71 with the change from CreationTimestamp to LastTimestamp.
The watcher is filtering out precious events when LastTimestamp is null.

[aflag]

I am also seeing this issue in 0.10. My lastTimestamp is also null. I agree with @DaveOHenry there. In my particular use case I don't really care if sometimes we get duplicated events, Deduplicating down the pipeline is easy for us. If I may suggest a feature, I'd suggest adding a configuration field that allows you to toggle that filtering off. I may have the time to contribute to the project if we agree on the fix.

In my experience, that affects all events from the kubernetes' scheduler. I'm running k8s 1.16.15, if that's relevant.

[maithal-netapp]

Hi,

We are also hitting this issue with version 0.10. Please let us know if there is any timeline when the fix might be available, so that we can plan.

Thanks & Regards,
Manoj

[sasati-tibco]

Facing this issue in v0.11 too. I hope this gets fixed in next release.