Domain resource can't be created successfully on Azure kubernetes Service
backwind1233 opened this issue · 18 comments
weblogic-kubernetes-operator:4.1.0 just released.
I followed this domain-on-pv-sample step by step and try to create domain.
It worked when I run the sample with local minikube,
but when I tried to run the sample on AKS, and it failed in the last step -- the step when I run the command kubectl apply -f /tmp/sample/domain-resource.yaml, I tried it for several times, but all failed.
It showed Job has reached the specified backoff limit.
I run the command kubectl get job <job-name> -o yaml > job.yaml to check the value of backoffLimit, and it is 0.
Could anyone tell what should I do if I want create domain with weblogic-kubernetes-operator:4.1.0 on AKS?
Attached is event log file.
Hello @ankedia, could you help to take a look at this issue?
I see that you've already lengthened the timeout of the introspection job. Given that the timeout (activeDeadlineSeconds) is set at 10 minutes, introspection is probably not failing due to a timeout. Can you check the status of the introspection job as described here?: https://oracle.github.io/weblogic-kubernetes-operator/managing-domains/debugging/#check-the-introspector-job.
Likely, we will find that the introspection job is failing for some reason that we can assist with.
@backwind1233 Can you upload the operator logs and events. From the screen snapshot, I see a Failed event for Domain with internal error but I can't see the complete event. If you have captured the introspector job pod logs, please upload that as well.
Thank you @rjeberhard @ankedia,
Please also consult the debugging document for debugging the problem.
I have followed the document trying to debug, but I still can't figure out.
Here are the logs.
-
job-describe-log.txt is the output of
kubectl -n sample-domain1-ns describe job/sample-domain1-introspector -
job-events-log.txt is the event log.
-
job-introspector-log.txt is the output of
kubectl -n sample-domain1-ns logs pod/sample-domain1-introspector-lhc2koutput.
It looks like introspector job pod is waiting to iniatilize. It's possible that the init container operator-aux-container1 is failing. How do you build the domain creation image? Is it built on a different chipset or OS? Can you run the below command to get the init container logs of the intospector job pod?
Run the following command to get the logs ->
kubectl logs <introspector-pod-name> -n <namespace> -c operator-aux-container1
It looks like introspector job pod is waiting to iniatilize. It's possible that the init container
operator-aux-container1is failing. How do you build the domain creation image? Is it built on a different chipset or OS? Can you run the below command to get the init container logs of the intospector job pod?
Yes, I build the domain creation image.
Is it built on a different chipset or OS?
Yes, I build it on my Mac OS with M1 .
Run the following command to get the logs ->
kubectl logs <introspector-pod-name> -n <namespace> -c operator-aux-container1
Yes, here is the result of the command.
exec /weblogic-operator/scripts/auxImage.sh: exec format error
Ok. It looks like your domain creation image was built on ARM processor but the container runtime environment is different (for e.g. AMD).
You can try setting the env variable before building the image ->
export DEFAULT_DOCKER_PLATFORM=linux/amd64
To confirm the image architecture, you can run this command ->
docker image inspect phx.ocir.io/weblogick8s/wdt-domain-image:WLS-v1 | grep -i architecture
"Architecture": "amd64",
Looping in @jshum2479 and @robertpatrick for other suggestions.
You can try setting the following environment variable before building the domain creation image on your Mac or build the image on a Linux VM.
export DEFAULT_DOCKER_PLATFORM=linux/amd64
Ok. It looks like your domain creation image was built on ARM processor but the container runtime environment is different (for e.g. AMD).
You can try setting the env variable before building the image ->
export DEFAULT_DOCKER_PLATFORM=linux/amd64To confirm the image architecture, you can run this command ->
docker image inspect phx.ocir.io/weblogick8s/wdt-domain-image:WLS-v1 | grep -i architecture "Architecture": "amd64",Looping in @jshum2479 and @robertpatrick for other suggestions.
Yes, maybe that's the reason.
You can try setting the following environment variable before building the domain creation image on your Mac or build the image on a Linux VM.
export DEFAULT_DOCKER_PLATFORM=linux/amd64
Ok, I will try that, thank you.
I used this command export DEFAULT_DOCKER_PLATFORM=linux/amd64 to build the image on my Mac M1,
also check the architecture with
docker image inspect wlsgzhcontainer.azurecr.io/wdt-domain-image:WLS-v2| grep -i architecture
"Architecture": "arm64",
It is "amd64", but the job still failed.
I will try to build the image on a virtual machine.
Hi @ankedia, could you run the domain creation sample with AKS successfully? Could you have a try, perhaps I made some silly mistakes.
@backwind1233 I don't have access to an AKS environment to run the domain creation sample. From the docker image inspect command output that you have provided above, the image is still using arm64 architecture and not amd64. Have you tried to build the image on a virtual machine? I'll check with my colleagues tomorrow about why architecture is still arm64 after you set DEFAULT_DOCKER_PLATFORM env variable to linux/amd64.
Thank you @ankedia , I am tring to build the image on VM,
I got an error when I try to build the image
Please set the JAVA_HOME environment variable to match the location of your Java installation. Java 8 or newer is required.
but I do set the JAVA_HOME
azureroot@buildvmimageonubuntu:/tmp/sample/wdt-artifacts/wdt-model-files/WLS-v1$ echo $JAVA_HOME
/home/azureroot/.sdkman/candidates/java/17.0.7-oracle
I am tring to fix it.
Hello @ankedia , I think you are right, it's the DOCKER_PLATFORM issue.
I build the image on a virtual machine, the architecture of the new image is "amd64".
Now it works as expected on AKS.
