DaspawnW/vault-crd

fabric8io/kubernetes-client@4.1.0 too old for kubernetes 1.15.6

Closed this issue · 2 comments

gaffneyd4 commented

Upon a recent upgrade from kubernetes 1.12.8 to 1.15.6 vault-crd stopped working within our cluster with the following error:

io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 36053598 (57536768)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onMessage(WatchConnectionManager.java:254) [kubernetes-client-4.1.0.jar!/:na]

According to the compatibility matrix for fabric8io's kubernetes-client, the version vault-crd is using 4.1.0 only supports connecting to kubernetes 1.9.0.

It would appear that in order to support versions of kubernetes from 1.10.0 and later, vault-crd needs to be updated to use the latest available version of the kubernetes-client library.

Versions used when finding bug:

vault-crd: 1.2.1, 1.3.1
kubernetes 1.15.6

Full log output:

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.1.1.RELEASE)

2020-01-23 11:18:47.133  INFO 1 --- [           main] de.koudingspawn.vault.VaultApplication   : Starting VaultApplication v0.0.1-SNAPSHOT on vault-crd-6688df545b-f7dsr with PID 1 (/opt/vault-crd.jar started by root in /opt)
2020-01-23 11:18:47.138  INFO 1 --- [           main] de.koudingspawn.vault.VaultApplication   : No active profile set, falling back to default profiles: default
2020-01-23 11:18:48.802  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2020-01-23 11:18:48.833  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2020-01-23 11:18:48.833  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/9.0.13
2020-01-23 11:18:48.848  INFO 1 --- [           main] o.a.catalina.core.AprLifecycleListener   : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
2020-01-23 11:18:48.941  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2020-01-23 11:18:48.941  INFO 1 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 1738 ms
2020-01-23 11:18:49.399  INFO 1 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'vaultThreadPoolTaskScheduler'
2020-01-23 11:18:49.640  WARN 1 --- [           main] i.f.k.client.internal.VersionUsageUtils  : The client is using resource type 'customresourcedefinitions' with unstable version 'v1beta1'
2020-01-23 11:18:50.464  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
2020-01-23 11:18:50.532  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2020-01-23 11:18:50.534  INFO 1 --- [           main] de.koudingspawn.vault.VaultApplication   : Started VaultApplication in 3.788 seconds (JVM running for 4.301)
2020-01-23 11:18:50.642  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret0 in namespace default
2020-01-23 11:18:50.655  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret1 in namespace default
2020-01-23 11:18:50.661  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret2 in namespace default
2020-01-23 11:18:50.667  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret3 in namespace default
2020-01-23 11:18:50.673  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret4 in namespace default
2020-01-23 11:18:50.681  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret5 in namespace default
2020-01-23 11:18:50.686  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret6 in namespace default
2020-01-23 11:18:50.694  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret7 in namespace default
2020-01-23 11:18:50.700  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret8 in namespace default
2020-01-23 11:18:50.721  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret9 in namespace default
2020-01-23 11:18:50.726  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret10 in namespace default
2020-01-23 11:18:50.739  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret11 in namespace default
2020-01-23 11:18:50.747  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret12 in namespace default
2020-01-23 11:18:50.753  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret13 in namespace default
2020-01-23 11:18:50.759  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret14 in namespace default
2020-01-23 11:18:50.773  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret15 in namespace default
2020-01-23 11:18:50.782  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret16 in namespace default
2020-01-23 11:18:50.792  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret17 in namespace default
2020-01-23 11:18:50.800  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret18 in namespace default
2020-01-23 11:18:50.808  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret19 in namespace default
2020-01-23 11:18:50.813  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret20 in namespace default
2020-01-23 11:18:50.821  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret21 in namespace default
2020-01-23 11:18:50.829  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret22 in namespace default
2020-01-23 11:18:50.835  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret23 in namespace default
2020-01-23 11:18:50.840  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret24 in namespace default
2020-01-23 11:18:50.845  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret25 in namespace default
2020-01-23 11:18:50.849  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret26 in namespace default
2020-01-23 11:18:50.857  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret27 in namespace default
2020-01-23 11:18:50.867  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret28 in namespace default
2020-01-23 11:18:50.880  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret29 in namespace default
2020-01-23 11:19:00.511  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
2020-01-23 11:19:00.660  INFO 1 --- [TaskScheduler-1] o.s.v.a.LifecycleAwareSessionManager     : Scheduling Token renewal
2020-01-23 11:19:01.088  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
2020-01-23 11:19:38.663  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-01-23 11:19:38.664  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2020-01-23 11:19:38.672  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 8 ms
2020-01-23 11:19:58.160 ERROR 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Watch for custom resource failed

io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 36053598 (57536768)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$2.onMessage(WatchConnectionManager.java:254) [kubernetes-client-4.1.0.jar!/:na]
	at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:310) [okhttp-3.9.1.jar!/:na]
	at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:222) [okhttp-3.9.1.jar!/:na]
	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:101) [okhttp-3.9.1.jar!/:na]
	at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:265) [okhttp-3.9.1.jar!/:na]
	at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:204) [okhttp-3.9.1.jar!/:na]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153) [okhttp-3.9.1.jar!/:na]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) [okhttp-3.9.1.jar!/:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_201]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]

2020-01-23 11:20:08.168  INFO 1 --- [       Thread-5] o.s.s.c.ThreadPoolTaskScheduler          : Shutting down ExecutorService 'vaultThreadPoolTaskScheduler'
gaffneyd4 commented

@DaspawnW Thank you so much for taking a look at this in such short notice! I really appreciate it. I rolled out the new version today in some testing environments, and unfortunately believe that I found a regression introduced by this fix. I created an issue for it, and I'm happy to provide more information and perform additional testing to help.

#32