Installation fails due to "java.io.IOException: No schema registered under subject!"
JarkoDubbeldam opened this issue · 1 comments
JarkoDubbeldam commented
Describe the bug
Fresh installation of the datahub chart fails during the datahub-system-update-job Job.
To Reproduce
prerequisites-values.yaml:
elasticsearch:
securityContext:
readOnlyRootFilesystem: false
allowPrivilegeEscalation: false
podSecurityContext:
runAsUser: 65534
runAsNonRoot: false
runAsGroup: 65534
fsGroup: 65534
mysql:
primary:
containerSecurityContext:
allowPrivilegeEscalation: false
datahub-values.yaml
elasticsearchSetupJob:
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
podSecurityContext:
runAsUser: 65534
runAsNonRoot: true
runAsGroup: 65534
fsGroup: 65534
kafkaSetupJob:
securityContext:
readOnlyRootFilesystem: false
allowPrivilegeEscalation: false
runAsUser: 1000
podSecurityContext:
runAsUser: 1000
runAsNonRoot: true
runAsGroup: 1000
fsGroup: 1000
mysqlSetupJob:
securityContext:
readOnlyRootFilesystem: false
allowPrivilegeEscalation: false
podSecurityContext:
runAsUser: 65534
runAsNonRoot: true
runAsGroup: 65534
fsGroup: 65534
kubectl create namespace datahub
kubectl create secret generic mysql-secrets --from-literal=mysql-root-password=datahub -n datahub
kubectl create secret generic neo4j-secrets --from-literal=neo4j-password=datahub -n datahub
# Install prerequisites
helm upgrade -i -n datahub prerequisites datahub/datahub-prerequisites --values prerequisites-values.yaml --wait
# Install datahub
helm upgrade -i -n datahub datahub datahub/datahub --values datahub-values.yaml --wait
Expected behavior
A succesful installation of datahub (or at least more specifically, a succesful run of the datahub-system-update-job Job.
Screenshots
Stack trace from pod logs:
2023-09-01 06:19:39,621 [main] ERROR c.l.d.u.s.e.steps.DataHubStartupStep:40 - DataHubStartupStep failed.
org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
Caused by: java.io.IOException: No schema registered under subject!
at io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient.getLatestVersion(MockSchemaRegistryClient.java:261)
at io.confluent.kafka.schemaregistry.client.MockSchemaRegistryClient.getLatestSchemaMetadata(MockSchemaRegistryClient.java:310)
at io.confluent.kafka.serializers.AbstractKafkaSchemaSerDe.lookupLatestVersion(AbstractKafkaSchemaSerDe.java:181)
at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:77)
at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:59)
at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:902)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:862)
at com.linkedin.metadata.dao.producer.KafkaEventProducer.produceDataHubUpgradeHistoryEvent(KafkaEventProducer.java:171)
at com.linkedin.datahub.upgrade.system.elasticsearch.steps.DataHubStartupStep.lambda$executable$0(DataHubStartupStep.java:37)
at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeStepInternal(DefaultUpgradeManager.java:110)
at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:68)
at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.executeInternal(DefaultUpgradeManager.java:42)
at com.linkedin.datahub.upgrade.impl.DefaultUpgradeManager.execute(DefaultUpgradeManager.java:33)
at com.linkedin.datahub.upgrade.UpgradeCli.run(UpgradeCli.java:80)
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768)
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:164)
at com.linkedin.datahub.upgrade.UpgradeCliApplication.main(UpgradeCliApplication.java:23)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Helm output:
history.go:56: [debug] getting history for release datahub
upgrade.go:123: [debug] preparing upgrade for datahub
upgrade.go:131: [debug] performing update for datahub
upgrade.go:303: [debug] creating upgraded release for datahub
client.go:290: [debug] Starting delete for "datahub-elasticsearch-setup-job" Job
client.go:128: [debug] creating 1 resource(s)
client.go:519: [debug] Watching for changes to Job datahub-elasticsearch-setup-job with timeout of 5m0s
client.go:547: [debug] Add/Modify event for datahub-elasticsearch-setup-job: ADDED
client.go:586: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:586: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:586: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:586: [debug] datahub-elasticsearch-setup-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:290: [debug] Starting delete for "datahub-kafka-setup-job" Job
client.go:128: [debug] creating 1 resource(s)
client.go:519: [debug] Watching for changes to Job datahub-kafka-setup-job with timeout of 5m0s
client.go:547: [debug] Add/Modify event for datahub-kafka-setup-job: ADDED
client.go:586: [debug] datahub-kafka-setup-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-kafka-setup-job: MODIFIED
client.go:586: [debug] datahub-kafka-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-kafka-setup-job: MODIFIED
client.go:586: [debug] datahub-kafka-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-kafka-setup-job: MODIFIED
client.go:586: [debug] datahub-kafka-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-kafka-setup-job: MODIFIED
client.go:586: [debug] datahub-kafka-setup-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-kafka-setup-job: MODIFIED
client.go:290: [debug] Starting delete for "datahub-mysql-setup-job" Job
client.go:128: [debug] creating 1 resource(s)
client.go:519: [debug] Watching for changes to Job datahub-mysql-setup-job with timeout of 5m0s
client.go:547: [debug] Add/Modify event for datahub-mysql-setup-job: ADDED
client.go:586: [debug] datahub-mysql-setup-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-mysql-setup-job: MODIFIED
client.go:586: [debug] datahub-mysql-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-mysql-setup-job: MODIFIED
client.go:586: [debug] datahub-mysql-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-mysql-setup-job: MODIFIED
client.go:586: [debug] datahub-mysql-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-mysql-setup-job: MODIFIED
client.go:586: [debug] datahub-mysql-setup-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-mysql-setup-job: MODIFIED
client.go:290: [debug] Starting delete for "datahub-datahub-system-update-job" Job
client.go:128: [debug] creating 1 resource(s)
client.go:519: [debug] Watching for changes to Job datahub-datahub-system-update-job with timeout of 5m0s
client.go:547: [debug] Add/Modify event for datahub-datahub-system-update-job: ADDED
client.go:586: [debug] datahub-datahub-system-update-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-datahub-system-update-job: MODIFIED
client.go:586: [debug] datahub-datahub-system-update-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-datahub-system-update-job: MODIFIED
client.go:586: [debug] datahub-datahub-system-update-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-datahub-system-update-job: MODIFIED
client.go:586: [debug] datahub-datahub-system-update-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-datahub-system-update-job: MODIFIED
client.go:586: [debug] datahub-datahub-system-update-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-datahub-system-update-job: MODIFIED
client.go:586: [debug] datahub-datahub-system-update-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-datahub-system-update-job: MODIFIED
client.go:586: [debug] datahub-datahub-system-update-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-datahub-system-update-job: MODIFIED
client.go:586: [debug] datahub-datahub-system-update-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-datahub-system-update-job: MODIFIED
client.go:586: [debug] datahub-datahub-system-update-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-datahub-system-update-job: MODIFIED
client.go:586: [debug] datahub-datahub-system-update-job: Jobs active: 1, jobs failed: 2, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-datahub-system-update-job: MODIFIED
client.go:586: [debug] datahub-datahub-system-update-job: Jobs active: 1, jobs failed: 2, jobs succeeded: 0
client.go:547: [debug] Add/Modify event for datahub-datahub-system-update-job: MODIFIED
client.go:586: [debug] datahub-datahub-system-update-job: Jobs active: 1, jobs failed: 2, jobs succeeded: 0
upgrade.go:369: [debug] warning: Upgrade "datahub" failed: pre-upgrade hooks failed: timed out waiting for the condition
Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition
helm.go:88: [debug] pre-upgrade hooks failed: timed out waiting for the condition
UPGRADE FAILED
main.newUpgradeCmd.func2
helm.sh/helm/v3/cmd/helm/upgrade.go:157
github.com/spf13/cobra.(*Command).execute
github.com/spf13/cobra@v1.1.3/command.go:852
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/cobra@v1.1.3/command.go:960
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/cobra@v1.1.3/command.go:897
main.main
helm.sh/helm/v3/cmd/helm/helm.go:87
runtime.main
runtime/proc.go:225
runtime.goexit
runtime/asm_amd64.s:1371
Gist with the full logs of the failing job: https://gist.github.com/JarkoDubbeldam/43c87c66826f77ed4489eb821a04b6e5
JarkoDubbeldam commented
Ah, looks like a duplicate of #347