investigate ways to show binding tests working
Closed this issue · 4 comments
Looked into this, but am being hindered by some computer issues (with pair being down). Will have some data to back this up soon, but my current finding is that we can find the events dealing with a pod binding request uri and the endpoint involved, which is the endpoint we are looking for. When we grab the namespace involved in this event, and then look at all useragents that hit this namespace, ordered by the timestamp of their event, then you can see the 'delete collection of pods' test stephen wrote interspersed with these bindings. This feels like good (though still indirect) evidence of the binding happening as part of the test.
However, the way we show when an endpoint is tested is to look at the test that hit it, which is taken by grabbing all audit events where the test field is not null, then joining it to the tests table to get the test's release, then grabbing the earliest released test and its release date. The binding endpoint will not have its test field filled. We can build a statement that shows it is hit by a test, but we'll need to do some additional logic if we want to figure out what that test is.
We could write a procedure that updates the audit events table so that these binding endpoints get their test field properly filled out? This mikght be the cleanest, and involves the least amount of changes to the existing code. It does mean we are mutating these records after they've been loaded, which raises the possibility of errors. Keen to discuss.
Given an e2e test
And the namespace created by it running it
When we iterate over all e2e created namespaces
And take note of each test created within
Then we can have a test to namespace mapping
And for particularly difficult to test endpoints
And mark specific endpoints as tested using this test->namespace mappingInvestigating the binding endpoint
Intro
We are in the final stages of conformance, with 98% of the conformant-eligible endpoints covered by a test as reported by APISnoop. The endpoints that remain are historically hard to target. We recently discovered one of these endpoints, createCoreV1NamespacedPodBinding might already be tested—in the broad spirit of conformance testing—but not showing in coverage reports due to how APIsnoop is structured.
We’d like to expand APISnoop’s logic to handle edge cases like this endpoint. Since this affects our coverage reports, we wanted to get agreement from stakeholders before doing the work. This document gives the context on the endpoint and our reasoning, and offers ways we could expand our reporting to accurately show that it has been tested.
How we report on test coverage
Put broadly, we consider an endpoint tested if it is hit by a test during the kubernetes conformance test suite. More specifically, the test suite outputs a an audit log of events showing the request and responses to various k8s endpoints. These events include metadata like the useragent that made the request. All tests in the suite have a useragent that starts with ’e2e.test’. In snoopdb, we look at the endpoints of an audit event and whether they were hit by a useragent that starts with e2e. If so, we consider it tested.
This logic is defined in the sql function loadauditevents, line 55.
As an example, here are the audit events for the test “[sig-node] Pods should delete a collection of pods [Conformance]”.
In this, I group it by endpoint and order it by the stagetimestamp of the event.
NOTE: The test will often hit an endpoint multiple times in a row as part of its step. I use a lag function to drop repeats in a group, for the sake of clarity. So if a test hits endpoints AA,BBB,CC,AA,CC it shows in our result as A,B,C,A,C.
with test_run as (
select data->>'stageTimestamp' as stagetime,
endpoint,
data->'objectRef'->>'namespace' as namespace,
left(useragent,16)||'...' as useragent,
(lag(endpoint,1) over (order by data->>'stageTimestamp') != endpoint) as first_hit_in_group
from audit_event
where test = '[sig-node] Pods should delete a collection of pods [Conformance]'
)
select stagetime, endpoint, namespace, useragent
from test_run
where first_hit_in_group is true
or first_hit_in_group is null;
stagetime | endpoint | namespace | useragent
-----------------------------+-------------------------------------------+-----------+---------------------
2023-01-11T17:45:50.807074Z | createCoreV1Namespace | | e2e.test/v1.27.0...
2023-01-11T17:45:50.847505Z | listCoreV1NamespacedServiceAccount | pods-873 | e2e.test/v1.27.0...
2023-01-11T17:45:51.008398Z | listCoreV1NamespacedConfigMap | pods-873 | e2e.test/v1.27.0...
2023-01-11T17:45:51.095629Z | createCoreV1NamespacedPod | pods-873 | e2e.test/v1.27.0...
2023-01-11T17:45:51.228863Z | listCoreV1NamespacedReplicationController | pods-873 | e2e.test/v1.27.0...
2023-01-11T17:45:51.269978Z | listAppsV1NamespacedReplicaSet | pods-873 | e2e.test/v1.27.0...
2023-01-11T17:45:51.318735Z | listCoreV1NamespacedPod | pods-873 | e2e.test/v1.27.0...
2023-01-11T17:45:53.364865Z | listCoreV1NamespacedReplicationController | pods-873 | e2e.test/v1.27.0...
2023-01-11T17:45:53.411451Z | listAppsV1NamespacedReplicaSet | pods-873 | e2e.test/v1.27.0...
2023-01-11T17:45:53.453038Z | listCoreV1NamespacedPod | pods-873 | e2e.test/v1.27.0...
2023-01-11T17:45:53.513536Z | deleteCoreV1CollectionNamespacedPod | pods-873 | e2e.test/v1.27.0...
2023-01-11T17:45:53.556143Z | listCoreV1NamespacedPod | pods-873 | e2e.test/v1.27.0...
2023-01-11T17:45:56.642930Z | listCoreV1Node | | e2e.test/v1.27.0...
2023-01-11T17:45:56.694666Z | deleteCoreV1Namespace | pods-873 | e2e.test/v1.27.0...
2023-01-11T18:10:20.787488Z | createCoreV1Namespace | | e2e.test/v1.27.0...
2023-01-11T18:10:20.804685Z | listCoreV1NamespacedServiceAccount | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:20.822545Z | listCoreV1NamespacedConfigMap | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:20.829066Z | createCoreV1NamespacedPod | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:20.917047Z | listCoreV1NamespacedReplicationController | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:20.922101Z | listAppsV1NamespacedReplicaSet | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:20.943673Z | listCoreV1NamespacedPod | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:22.947490Z | listCoreV1NamespacedReplicationController | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:22.950580Z | listAppsV1NamespacedReplicaSet | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:22.953350Z | listCoreV1NamespacedPod | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:24.948431Z | listCoreV1NamespacedReplicationController | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:24.951945Z | listAppsV1NamespacedReplicaSet | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:24.954500Z | listCoreV1NamespacedPod | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:24.957664Z | deleteCoreV1CollectionNamespacedPod | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:25.021574Z | listCoreV1NamespacedPod | pods-4348 | e2e.test/v1.27.0...
2023-01-11T18:10:27.042063Z | listCoreV1Node | | e2e.test/v1.27.0...
2023-01-11T18:10:27.046075Z | deleteCoreV1Namespace | pods-4348 | e2e.test/v1.27.0...
2023-01-11T19:48:20.387286Z | createCoreV1Namespace | | e2e.test/v1.27.0...
2023-01-11T19:48:20.434830Z | listCoreV1NamespacedServiceAccount | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:20.597966Z | listCoreV1NamespacedConfigMap | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:20.693773Z | createCoreV1NamespacedPod | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:20.822430Z | listCoreV1NamespacedReplicationController | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:20.865790Z | listAppsV1NamespacedReplicaSet | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:20.914811Z | listCoreV1NamespacedPod | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:22.958034Z | listCoreV1NamespacedReplicationController | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:22.998522Z | listAppsV1NamespacedReplicaSet | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:23.039210Z | listCoreV1NamespacedPod | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:25.019156Z | listCoreV1NamespacedReplicationController | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:25.087706Z | listAppsV1NamespacedReplicaSet | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:25.188758Z | listCoreV1NamespacedPod | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:26.978442Z | listCoreV1NamespacedReplicationController | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:27.029959Z | listAppsV1NamespacedReplicaSet | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:27.097131Z | listCoreV1NamespacedPod | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:28.977032Z | listCoreV1NamespacedReplicationController | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:29.026396Z | listAppsV1NamespacedReplicaSet | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:29.074216Z | listCoreV1NamespacedPod | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:29.202802Z | deleteCoreV1CollectionNamespacedPod | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:29.259008Z | listCoreV1NamespacedPod | pods-4904 | e2e.test/v1.27.0...
2023-01-11T19:48:41.483434Z | listCoreV1Node | | e2e.test/v1.27.0...
2023-01-11T19:48:41.560944Z | deleteCoreV1Namespace | pods-4904 | e2e.test/v1.27.0...
(54 rows)
That test is defined in e2e/common/node/pods.go. As you can see in the test, it creates three pod names, then runs a for loop through that list. This is mirrored in the audit events: for each pod, we see a namespace created that’s named after the pod, a number of relevant endpoints hit, and then the deletion of the namespace.
Our endpoint
It seemed to us that, for this test, there would be a need to create a binding of each pod, yet the endpoint createCoreV1NamespacedPodBinding is not hit. Why is that?
We suspected that the endpoint was being interacted with by a different useragent. If we run through the query again, but now without the test restriction and looking only at a single namespace, we’ll see every endpoint hit and by whom.
This query won’t show events performed by the test outside of the namespace (and so we won’t see createCoreV1Namespace or listCoreV1Node). It’s still useful for seeing the larger pattern.
(I limited it to 20 rows to hopefully not overwhelm.)
with test_run as (
select data->>'stageTimestamp' as stagetime,
endpoint,
data->'objectRef'->>'namespace' as namespace,
left(useragent,16)||'...' as useragent,
(lag(endpoint,1) over (order by data->>'stageTimestamp') != endpoint) as first_hit_in_group
from audit_event
where data->'objectRef'->>'namespace' = 'pods-4134'
)
select
stagetime,
case
when endpoint = 'createCoreV1NamespacedPodBinding'
then '*'||endpoint||'*'
else endpoint
end as endpoint,
namespace,
case
when endpoint = 'createCoreV1NamespacedPodBinding'
then '*'||left(useragent,16)||'...*'
else left(useragent,16)||'...'
end as useragent
from test_run
where first_hit_in_group is true
or first_hit_in_group is null
limit 20;
stagetime | endpoint | namespace | useragent
-----------------------------+--------------------------------------+-----------+-----------------------
2023-01-10T21:47:13.065638Z | listCoreV1NamespacedResourceQuota | pods-4134 | kube-apiserver/v...
2023-01-10T21:47:13.103803Z | createCoreV1NamespacedServiceAccount | pods-4134 | kube-controller-...
2023-01-10T21:47:13.103880Z | createCoreV1NamespacedConfigMap | pods-4134 | kube-controller-...
2023-01-10T21:47:13.104978Z | listCoreV1NamespacedServiceAccount | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.117014Z | createCoreV1NamespacedServiceAccount | pods-4134 | kube-controller-...
2023-01-10T21:47:13.118236Z | listCoreV1NamespacedServiceAccount | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.122208Z | listCoreV1NamespacedConfigMap | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.126881Z | createCoreV1NamespacedConfigMap | pods-4134 | kube-controller-...
2023-01-10T21:47:13.127325Z | listCoreV1NamespacedConfigMap | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.127701Z | createCoreV1NamespacedPod | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.129252Z | listCoreV1NamespacedLimitRange | pods-4134 | kube-apiserver/v...
2023-01-10T21:47:13.143505Z | createCoreV1NamespacedPod | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.145297Z | *createCoreV1NamespacedPodBinding* | pods-4134 | *kube-scheduler/v...*
2023-01-10T21:47:13.159435Z | createCoreV1NamespacedEvent | pods-4134 | kube-scheduler/v...
2023-01-10T21:47:13.159781Z | readCoreV1NamespacedPod | pods-4134 | kubelet/v1.27.0 ...
2023-01-10T21:47:13.159907Z | listCoreV1NamespacedConfigMap | pods-4134 | kubelet/v1.27.0 ...
2023-01-10T21:47:13.168392Z | createCoreV1NamespacedPod | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.169785Z | *createCoreV1NamespacedPodBinding* | pods-4134 | *kube-scheduler/v...*
2023-01-10T21:47:13.177967Z | readCoreV1NamespacedPod | pods-4134 | kubelet/v1.27.0 ...
2023-01-10T21:47:13.179072Z | patchCoreV1NamespacedPodStatus | pods-4134 | kubelet/v1.27.0 ...
(20 rows)
Our targeted endpoint is in these results. createCoreV1NamespacedPodBinding is hit by kube-scheduler immediately after the e2e useragent hits createCoreV1NamespacedPod. It seems like this endpoint is being hit as part of the test.
Looking further into this endpoint and the kube-scheduler, we knew that it would be hitting the endpoint with a requestURI always structured the same way:
/api/v1/namespaces/pods-%/pods/test-pod-%/binding
This uri is based on how the test itself was written, since it named the pods `test-pod` plus a random number.
With this in mind, we can extend our query above with additional logic. What does it look like when we restrict our results to those hit by the test OR by kube-server at that requestURI?
with test_run as (
select data->>'stageTimestamp' as stagetime,
endpoint,
data->'objectRef'->>'namespace' as namespace,
left(useragent,16)||'...' as useragent,
(lag(endpoint,1) over (order by data->>'stageTimestamp') != endpoint) as first_hit_in_group
from audit_event
where useragent like '%e2e.test%[sig-node] Pods should delete a collection of pods [Conformance]%'
or data->>'requestURI' like '/api/v1/namespaces/pods-%/pods/test-pod-%/binding'
)
select stagetime, endpoint, namespace, useragent
from test_run
where first_hit_in_group is true
or first_hit_in_group is null;
stagetime | endpoint | namespace | useragent
-----------------------------+-------------------------------------------+-----------+---------------------
2023-01-10T21:47:13.064246Z | createCoreV1Namespace | | e2e.test/v1.27.0...
2023-01-10T21:47:13.104978Z | listCoreV1NamespacedServiceAccount | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.122208Z | listCoreV1NamespacedConfigMap | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.127701Z | createCoreV1NamespacedPod | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.145297Z | createCoreV1NamespacedPodBinding | pods-4134 | kube-scheduler/v...
2023-01-10T21:47:13.168392Z | createCoreV1NamespacedPod | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.169785Z | createCoreV1NamespacedPodBinding | pods-4134 | kube-scheduler/v...
2023-01-10T21:47:13.188230Z | createCoreV1NamespacedPod | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.188255Z | createCoreV1NamespacedPodBinding | pods-4134 | kube-scheduler/v...
2023-01-10T21:47:13.190187Z | listCoreV1NamespacedReplicationController | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.191118Z | createCoreV1NamespacedPodBinding | pods-4134 | kube-scheduler/v...
2023-01-10T21:47:13.198073Z | listCoreV1NamespacedReplicationController | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.199311Z | listAppsV1NamespacedReplicaSet | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:13.239908Z | createCoreV1NamespacedPodBinding | pods-4134 | kube-scheduler/v...
2023-01-10T21:47:13.240082Z | listCoreV1NamespacedPod | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:15.245650Z | listCoreV1NamespacedReplicationController | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:15.248509Z | listAppsV1NamespacedReplicaSet | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:15.251178Z | listCoreV1NamespacedPod | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:17.252254Z | listCoreV1NamespacedReplicationController | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:17.255153Z | listAppsV1NamespacedReplicaSet | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:17.257818Z | listCoreV1NamespacedPod | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:17.261364Z | deleteCoreV1CollectionNamespacedPod | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:17.327982Z | listCoreV1NamespacedPod | pods-4134 | e2e.test/v1.27.0...
2023-01-10T21:47:20.769904Z | listCoreV1Node | | e2e.test/v1.27.0...
2023-01-10T21:47:20.773236Z | deleteCoreV1Namespace | pods-4134 | e2e.test/v1.27.0...
2023-01-10T23:26:45.565525Z | createCoreV1Namespace | | e2e.test/v1.27.0...
2023-01-10T23:26:45.606155Z | listCoreV1NamespacedServiceAccount | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:45.802240Z | listCoreV1NamespacedConfigMap | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:45.887796Z | createCoreV1NamespacedPod | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:45.893238Z | createCoreV1NamespacedPodBinding | pods-6896 | kube-scheduler/v...
2023-01-10T23:26:45.931611Z | createCoreV1NamespacedPod | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:45.938249Z | createCoreV1NamespacedPodBinding | pods-6896 | kube-scheduler/v...
2023-01-10T23:26:45.975022Z | createCoreV1NamespacedPod | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:45.980357Z | createCoreV1NamespacedPodBinding | pods-6896 | kube-scheduler/v...
2023-01-10T23:26:46.016344Z | listCoreV1NamespacedReplicationController | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:46.057678Z | listAppsV1NamespacedReplicaSet | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:46.099133Z | listCoreV1NamespacedPod | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:48.142674Z | listCoreV1NamespacedReplicationController | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:48.183801Z | listAppsV1NamespacedReplicaSet | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:48.225326Z | listCoreV1NamespacedPod | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:50.141953Z | listCoreV1NamespacedReplicationController | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:50.183386Z | listAppsV1NamespacedReplicaSet | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:50.225117Z | listCoreV1NamespacedPod | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:50.293662Z | deleteCoreV1CollectionNamespacedPod | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:50.335061Z | listCoreV1NamespacedPod | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:26:51.419043Z | listCoreV1Node | | e2e.test/v1.27.0...
2023-01-10T23:26:51.464516Z | deleteCoreV1Namespace | pods-6896 | e2e.test/v1.27.0...
2023-01-10T23:40:52.229492Z | createCoreV1Namespace | | e2e.test/v1.27.0...
2023-01-10T23:40:52.273959Z | listCoreV1NamespacedServiceAccount | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:52.442342Z | listCoreV1NamespacedConfigMap | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:52.535292Z | createCoreV1NamespacedPod | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:52.544991Z | createCoreV1NamespacedPodBinding | pods-3454 | kube-scheduler/v...
2023-01-10T23:40:52.580174Z | createCoreV1NamespacedPod | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:52.591750Z | createCoreV1NamespacedPodBinding | pods-3454 | kube-scheduler/v...
2023-01-10T23:40:52.652084Z | createCoreV1NamespacedPod | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:52.672030Z | createCoreV1NamespacedPodBinding | pods-3454 | kube-scheduler/v...
2023-01-10T23:40:52.700475Z | listCoreV1NamespacedReplicationController | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:52.741057Z | listAppsV1NamespacedReplicaSet | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:52.784156Z | listCoreV1NamespacedPod | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:54.850684Z | listCoreV1NamespacedReplicationController | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:54.911027Z | listAppsV1NamespacedReplicaSet | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:54.965961Z | listCoreV1NamespacedPod | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:56.844611Z | listCoreV1NamespacedReplicationController | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:56.924445Z | listAppsV1NamespacedReplicaSet | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:56.971241Z | listCoreV1NamespacedPod | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:58.932476Z | listCoreV1NamespacedReplicationController | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:59.107012Z | listAppsV1NamespacedReplicaSet | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:40:59.289259Z | listCoreV1NamespacedPod | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:41:00.854884Z | listCoreV1NamespacedReplicationController | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:41:00.926584Z | listAppsV1NamespacedReplicaSet | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:41:00.976802Z | listCoreV1NamespacedPod | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:41:02.969297Z | listCoreV1NamespacedReplicationController | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:41:03.119902Z | listAppsV1NamespacedReplicaSet | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:41:03.246295Z | listCoreV1NamespacedPod | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:41:04.968093Z | listCoreV1NamespacedReplicationController | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:41:05.194611Z | listAppsV1NamespacedReplicaSet | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:41:05.305026Z | listCoreV1NamespacedPod | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:41:05.574675Z | deleteCoreV1CollectionNamespacedPod | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:41:05.640501Z | listCoreV1NamespacedPod | pods-3454 | e2e.test/v1.27.0...
2023-01-10T23:41:17.734483Z | listCoreV1Node | | e2e.test/v1.27.0...
2023-01-10T23:41:17.806464Z | deleteCoreV1Namespace | pods-3454 | e2e.test/v1.27.0...
(81 rows)
A long wall of text, but hopefully the point comes across. It is the same set of endpoints, in the same order, for each created test pod. Our targeted endpoint is hit in the same way each time by kube-scheduler. It seems like our endpoint is being hit as a part of this test, but in a way APISnoop is not currently set up to report.
Where to from here?
If we are in agreement that this endpoint is qualified as “tested”, then we can update APISnoop’s logic to account for this.
There are a few ways we can do this. We could put in an additional update to the db for this test in particular, seeing it as an edge-case. It would essentially be the query given above and would update any auditevents found in that query to be ’tested’ with that test. This would be a conservative change in APISnoop, which could be good, but it also means we have a portion of our reporting dependent on this test. If the test’s text or body changes in any significant way, then our coverage reporting could change arbitrarily. It adds a small, but continual, maintenance fee.
Another option would be to expand our definition of ’hit by a test’ across the board. We could say an endpoint is marked as tested if either of these conditions are true:
- it is hit by a test useragent
- it is part of an event that shares a namespace as one created by a test useragent.
This second option would be easier to maintain, but may be too loose of a definition which could impact the confidence in our coverage report. Perhaps there is another definition that feels better and could cover edge cases like the one outlined here? The upside of this is that our remaining 2% of endpoints may already be covered by tests, once we expand our reporting to find them.
with test_namespace as (
select data->'objectRef'->>'namespace' as namespace
from audit_event
where test is not null
), broader_tested as (
select endpoint
from audit_event ae
join test_namespace t on (t.namespace = ae.data->'objectRef'->>'namespace')
group by endpoint
)
select endpoint,
exists(
select 1
from broader_tested bt
where ue.endpoint = bt.endpoint
) as already_tested
from testing.untested_stable_endpoint ue
where eligible is true;
endpoint | already_tested
---------------------------------------------+----------------
replaceApiregistrationV1APIServiceStatus | f
replaceApiregistrationV1APIService | f
patchApiregistrationV1APIServiceStatus | f
deleteApiregistrationV1CollectionAPIService | f
createCoreV1NamespacedPodBinding | t
connectCoreV1PostNamespacedPodAttach | t
connectCoreV1GetNamespacedPodAttach | f
(7 rows)
If we look at those tested endpoints, we can see the useragents that are hitting it.
with test_namespace as (
select data->'objectRef'->>'namespace' as namespace
from audit_event
where test is not null
group by namespace
), broader_tested as (
select endpoint, useragent
from audit_event ae
join test_namespace t on (t.namespace = ae.data->'objectRef'->>'namespace')
where useragent not like '%e2e.test'
group by endpoint,useragent
)
select ue.endpoint,
useragent
from testing.untested_stable_endpoint ue
join broader_tested bt on (bt.endpoint = ue.endpoint)
where eligible is true;
endpoint | useragent
--------------------------------------+-------------------------------------------------------------------
connectCoreV1PostNamespacedPodAttach | kubectl/v1.27.0 (linux/amd64) kubernetes/564f438
connectCoreV1PostNamespacedPodAttach | kubectl/v1.27.0 (linux/amd64) kubernetes/7e97b4b
createCoreV1NamespacedPodBinding | kube-scheduler/v1.27.0 (linux/amd64) kubernetes/564f438/scheduler
createCoreV1NamespacedPodBinding | kube-scheduler/v1.27.0 (linux/amd64) kubernetes/7e97b4b/scheduler
(4 rows)
Fixed by #661