triggermesh/knative-lambda-runtime

Runtime created step-export can't find Dockerfile when using private repository

Hi-Fi opened this issue · 11 comments

Hi-Fi commented

At e.g. Python 3.7 runtime Dockerfile is written to /workspace/workspace/$(inputs.params.DIRECTORY) at dockerfilestep, and referred just as --dockerfile=Dockerfile at export step.

This causes Kaniko to look Dockerfile from /kaniko/Dockerfile when registry-secret argument is defined. This produces error:

Error: error resolving dockerfile path: copying dockerfile: creating file: open /kaniko/Dockerfile: permission denied

to logs.

Workaround seems to be to use - --dockerfile=/workspace/workspace/$(inputs.params.DIRECTORY)/Dockerfile in export step. According to Kaniko documentation Dockerfile should be using context as a base, but apparently that's not the case (even with Kaniko 1.2.0) when registry-secret is defined.

tzununbekov commented

@Hi-Fi hi and thanks for the report. I walked through all deployment examples from our readme and they worked perfectly fine for me - kaniko's context parameter does its job (at least at v0.8.0). Could you share some information about your environment, such as which kubernetes platform and version, knative version, and if you're using tm CLI. Also would be helpful if you run python example commands with debugging enabled (tm -d ...) and dump the output here in a comment. Thanks!

tzununbekov commented

@Hi-Fi one more important bit - if you use tm CLI, please make sure that you're on the latest stable version
https://github.com/triggermesh/tm/releases/latest

Hi-Fi commented
  • tm cli: Triggermesh CLI, version v0.6.0
  • Kubernetes:
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T21:51:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:04:18Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Just noticed that it happens only if I define registry-secret to command.

tm deploy service python-test -f https://github.com/serverless/examples \
                              --registry-secret PRIVATE --build-template knative-python37-runtime \
                              --build-argument DIRECTORY=aws-python-simple-http-endpoint \
                              --build-argument HANDLER=handler.endpoint \
                              --wait --debug
DEBU[2020-10-19T09:39:14+03:00] cloning task to a new object "lambda/knative-python37-runtime" 
DEBU[2020-10-19T09:39:14+03:00] setting registry secret "PRIVATE" for task "lambda/" 
DEBU[2020-10-19T09:39:14+03:00] taskrun "lambda/python-test" output image will be "PRIVATE/python-test:furPRD" 
DEBU[2020-10-19T09:39:14+03:00] taskrun "lambda/python-test-rlkx2" created   
DEBU[2020-10-19T09:39:14+03:00] setting task "lambda/knative-python37-runtime-qt8q6" owner to TaskRun/python-test-rlkx2 
DEBU[2020-10-19T09:39:14+03:00] setting pipelineresource owner               
INFO[2020-10-19T09:39:14+03:00] Waiting for taskrun "python-test-rlkx2" ready state 
DEBU[2020-10-19T09:39:14+03:00] got new event:                               
DEBU[2020-10-19T09:39:15+03:00] got new event:                               
DEBU[2020-10-19T09:39:15+03:00] got new event:                               
DEBU[2020-10-19T09:39:15+03:00]  condition: "Succeeded", status: "Unknown", message: "Pending" 
DEBU[2020-10-19T09:39:15+03:00] got new event:                               
DEBU[2020-10-19T09:39:15+03:00]  condition: "Succeeded", status: "Unknown", message: "pod status \"Initialized\":\"False\"; message: \"containers with incomplete status: [place-tools]\"" 
DEBU[2020-10-19T09:39:15+03:00] got new event:                               
DEBU[2020-10-19T09:39:15+03:00]  condition: "Succeeded", status: "Unknown", message: "pod status \"Initialized\":\"False\"; message: \"containers with incomplete status: [place-tools]\"" 
DEBU[2020-10-19T09:39:17+03:00] got new event:                               
DEBU[2020-10-19T09:39:17+03:00]  condition: "Succeeded", status: "Unknown", message: "pod status \"Ready\":\"False\"; message: \"containers with unready status: [step-git-source-sources-lnzzv step-dockerfile step-export]\"" 
DEBU[2020-10-19T09:39:18+03:00] got new event:                               
DEBU[2020-10-19T09:39:18+03:00]  condition: "Succeeded", status: "Unknown", message: "Not all Steps in the Task have finished executing" 
DEBU[2020-10-19T09:39:23+03:00] got new event:                               
DEBU[2020-10-19T09:39:23+03:00]  condition: "Succeeded", status: "Unknown", message: "Not all Steps in the Task have finished executing" 
DEBU[2020-10-19T09:39:24+03:00] got new event:                               
DEBU[2020-10-19T09:39:24+03:00]  condition: "Succeeded", status: "False", message: "\"step-export\" exited with code 1 (image: \"docker-pullable://gcr.io/kaniko-project/executor@sha256:a49506bad46fa4a8ce412239b25ebc2577290cc21ca04a0618d6157e379f811e\"); for logs run: kubectl -n lambda logs python-test-rlkx2-pod-szhgn -c step-export\n" 
DEBU[2020-10-19T09:39:25+03:00] setting taskrun "lambda/python-test-rlkx2" owner to Configuration/ 
FATA[2020-10-19T09:39:25+03:00] Deploying builder: taskrun "python-test-rlkx2" deployment failed: "step-export" exited with code 1 (image: "docker-pullable://gcr.io/kaniko-project/executor@sha256:a49506bad46fa4a8ce412239b25ebc2577290cc21ca04a0618d6157e379f811e"); for logs run: kubectl -n lambda logs python-test-rlkx2-pod-szhgn -c step-export

And logs from that command:

kubectl -n lambda logs python-test-rlkx2-pod-szhgn -c step-export
Error: error resolving dockerfile path: copying dockerfile: open /kaniko/Dockerfile: permission denied
Usage:
  executor [flags]

Flags:
      --build-arg multi-arg type                  This flag allows you to pass in ARG values at build time. Set it repeatedly for multiple values.
      --cache                                     Use cache when building image
      --cache-dir string                          Specify a local directory to use as a cache. (default "/cache
...

Authentication secret created as:

tm set registry-auth --registry PRIVATE.SERVER.URL --project klr-poc/images PRIVATE
tzununbekov commented

Error: error resolving dockerfile path: copying dockerfile: open /kaniko/Dockerfile: permission denied

I'd say that it's kaniko permission issue (missing root privileges) if it happens not only when registry-secret is set.

Just noticed that it happens only if I define registry-secret to command

Could you elaborate on what happens without the registry-secret parameter?

Hi-Fi commented

Error: error resolving dockerfile path: copying dockerfile: open /kaniko/Dockerfile: permission denied

I'd say that it's kaniko permission issue (missing root privileges) if it happens not only when registry-secret is set.

I run Kaniko as privileged and as UID 0, so I don't think it's permission issue. The Dockerfile is never written to /kaniko, so I don't see how it can be ever found from there. And as said, if that registry-secret is not set, things work as you also mentioned.

Just noticed that it happens only if I define registry-secret to command

Could you elaborate on what happens without the registry-secret parameter?

Task tries to push to that default knative.registry.svc.cluster.local, and fails as it's not found. So task works fine against default registry at least.

tzununbekov commented

I still don't see how setting the registry secret could break the export step in such an uninformative way. Could you show what taskrun pod looks like?

kubectl -n lambda describe pod python-test-rlkx2-pod-szhgn
Hi-Fi commented

Describe with private registry in use:

Name:         python-test-rlkx2-pod-szhgn
Namespace:    lambda
Priority:     0
Node:         nodes-z1-1-update-test-k8s-local/10.1.32.9
Start Time:   Mon, 19 Oct 2020 09:39:15 +0300
Labels:       app.kubernetes.io/managed-by=tekton-pipelines
              tekton.dev/task=knative-python37-runtime-qt8q6
              tekton.dev/taskRun=python-test-rlkx2
Annotations:  cni.projectcalico.org/podIP: 
              cni.projectcalico.org/podIPs: 
              kubernetes.io/limit-ranger:
                LimitRanger plugin set: cpu, memory limit for container step-git-source-sources-lnzzv; cpu, memory limit for container step-dockerfile; cp...
              kubernetes.io/psp: 00-privileged
              pipeline.tekton.dev/release: v0.17.1
              tekton.dev/ready: READY
Status:       Failed
IP:           100.110.73.178
IPs:
  IP:           100.110.73.178
Controlled By:  TaskRun/python-test-rlkx2
Init Containers:
  place-tools:
    Container ID:  docker://ca8929ffd20d2eb3b260b4640fc548ba7e059bbb0045d19333f6f631471d5453
    Image:         gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/entrypoint:v0.17.1@sha256:ec3db704fd4fd6007f41cb8949f14f1ab0d9d76d472f1942677a9e245a1c35af
    Image ID:      docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/entrypoint@sha256:ec3db704fd4fd6007f41cb8949f14f1ab0d9d76d472f1942677a9e245a1c35af
    Port:          <none>
    Host Port:     <none>
    Command:
      /ko-app/entrypoint
      cp
      /ko-app/entrypoint
      /tekton/tools/entrypoint
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 19 Oct 2020 09:39:16 +0300
      Finished:     Mon, 19 Oct 2020 09:39:17 +0300
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  100Mi
    Requests:
      cpu:        25m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /tekton/tools from tekton-internal-tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ng6s8 (ro)
Containers:
  step-git-source-sources-lnzzv:
    Container ID:  docker://fa272943adf433dd072bfdd930e5037111b90e2f18b4116759cb94a8ef93d392
    Image:         gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.17.1@sha256:a39b8c647d6eaf609dc04bc7b0fc0f787b3b25fd6b1c9b182066a5c29a64a18c
    Image ID:      docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init@sha256:a39b8c647d6eaf609dc04bc7b0fc0f787b3b25fd6b1c9b182066a5c29a64a18c
    Port:          <none>
    Host Port:     <none>
    Command:
      /tekton/tools/entrypoint
    Args:
      -wait_file
      /tekton/downward/ready
      -wait_file_content
      -post_file
      /tekton/tools/0
      -termination_path
      /tekton/termination
      -entrypoint
      /ko-app/git-init
      --
      -url
      https://github.com/serverless/examples
      -path
      /workspace/workspace
      -revision
      master
    State:          Terminated
      Reason:       Completed
      Message:      [{"key":"commit","value":"9eef07b09ee67c33e99c89c73b830a45b7da6ddb","resourceName":"sources","resourceRef":{"name":"sources"}},{"key":"url","value":"https://github.com/serverless/examples","resourceName":"sources","resourceRef":{"name":"sources"}},{"key":"StartedAt","value":"2020-10-19T06:39:19.970Z","type":"InternalTektonResult"}]
      Exit Code:    0
      Started:      Mon, 19 Oct 2020 09:39:17 +0300
      Finished:     Mon, 19 Oct 2020 09:39:23 +0300
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  100Mi
    Requests:
      cpu:                0
      ephemeral-storage:  0
      memory:             0
    Environment:
      HOME:                  /tekton/home
      TEKTON_RESOURCE_NAME:  sources
      HOME:                  /tekton/home
    Mounts:
      /tekton/creds from tekton-creds-init-home-6pgfl (rw)
      /tekton/downward from tekton-internal-downward (rw)
      /tekton/home from tekton-internal-home (rw)
      /tekton/results from tekton-internal-results (rw)
      /tekton/tools from tekton-internal-tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ng6s8 (ro)
      /workspace from tekton-internal-workspace (rw)
  step-dockerfile:
    Container ID:  docker://f985a9788dc3c9838d40a3f58e3542b566afd9384566f068bbe8b683b746cee0
    Image:         gcr.io/kaniko-project/executor:debug-v0.8.0
    Image ID:      docker-pullable://gcr.io/kaniko-project/executor@sha256:a49506bad46fa4a8ce412239b25ebc2577290cc21ca04a0618d6157e379f811e
    Port:          <none>
    Host Port:     <none>
    Command:
      /tekton/tools/entrypoint
    Args:
      -wait_file
      /tekton/tools/0
      -post_file
      /tekton/tools/1
      -termination_path
      /tekton/termination
      -entrypoint
      /busybox/sh
      --
      -c
      cd /workspace/workspace/aws-python-simple-http-endpoint
      cat <<EOF > Dockerfile
        FROM gcr.io/triggermesh/knative-lambda-python37
      
        ENV _HANDLER "handler.endpoint"
      
        COPY . .
        RUN if [ -f requirements.txt ]; then pip3.7 install -r requirements.txt ;fi
      
        ENTRYPOINT ["/opt/aws-custom-runtime"]
      EOF
      
    State:          Terminated
      Reason:       Completed
      Message:      [{"key":"StartedAt","value":"2020-10-19T06:39:24.246Z","type":"InternalTektonResult"}]
      Exit Code:    0
      Started:      Mon, 19 Oct 2020 09:39:18 +0300
      Finished:     Mon, 19 Oct 2020 09:39:24 +0300
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  100Mi
    Requests:
      cpu:                0
      ephemeral-storage:  0
      memory:             0
    Environment:
      HOME:           /tekton/home
      DOCKER_CONFIG:  /PRIVATE
    Mounts:
      /PRIVATE from PRIVATE (ro)
      /tekton/creds from tekton-creds-init-home-2rblp (rw)
      /tekton/home from tekton-internal-home (rw)
      /tekton/results from tekton-internal-results (rw)
      /tekton/tools from tekton-internal-tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ng6s8 (ro)
      /workspace from tekton-internal-workspace (rw)
  step-export:
    Container ID:  docker://487c554aecb926a750ff1a1c7e4d1f745f4d4e8bdc8370dec27dcba17f2bc1a3
    Image:         gcr.io/kaniko-project/executor@sha256:a49506bad46fa4a8ce412239b25ebc2577290cc21ca04a0618d6157e379f811e
    Image ID:      docker-pullable://gcr.io/kaniko-project/executor@sha256:a49506bad46fa4a8ce412239b25ebc2577290cc21ca04a0618d6157e379f811e
    Port:          <none>
    Host Port:     <none>
    Command:
      /tekton/tools/entrypoint
    Args:
      -wait_file
      /tekton/tools/1
      -post_file
      /tekton/tools/2
      -termination_path
      /tekton/termination
      -entrypoint
      /kaniko/executor
      --
      --context=/workspace/workspace/aws-python-simple-http-endpoint
      --dockerfile=Dockerfile
      --destination=PRIVATE.SERVER.URL/klr-poc/images/python-test:furPRD
    State:          Terminated
      Reason:       Error
      Message:      [{"key":"StartedAt","value":"2020-10-19T06:39:24.509Z","type":"InternalTektonResult"}]
      Exit Code:    1
      Started:      Mon, 19 Oct 2020 09:39:18 +0300
      Finished:     Mon, 19 Oct 2020 09:39:24 +0300
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  100Mi
    Requests:
      cpu:                0
      ephemeral-storage:  0
      memory:             0
    Environment:
      HOME:           /tekton/home
      DOCKER_CONFIG:  /PRIVATE
    Mounts:
      /PRIVATE from PRIVATE (ro)
      /tekton/creds from tekton-creds-init-home-k6lmx (rw)
      /tekton/home from tekton-internal-home (rw)
      /tekton/results from tekton-internal-results (rw)
      /tekton/tools from tekton-internal-tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ng6s8 (ro)
      /workspace from tekton-internal-workspace (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  tekton-internal-workspace:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tekton-internal-home:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tekton-internal-results:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tekton-internal-tools:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tekton-internal-downward:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations['tekton.dev/ready'] -> ready
  tekton-creds-init-home-6pgfl:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  tekton-creds-init-home-2rblp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  tekton-creds-init-home-k6lmx:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  PRIVATE:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  PRIVATE
    Optional:    false
  default-token-ng6s8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-ng6s8
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

Describe without private registry:

Name:         python-test-jvp4p-pod-nff9f
Namespace:    lambda
Priority:     0
Node:         nodes-z1-1-update-test-k8s-local/10.1.32.9
Start Time:   Mon, 19 Oct 2020 09:37:45 +0300
Labels:       app.kubernetes.io/managed-by=tekton-pipelines
              tekton.dev/task=knative-python37-runtime
              tekton.dev/taskRun=python-test-jvp4p
Annotations:  cni.projectcalico.org/podIP: 
              cni.projectcalico.org/podIPs: 
              kubernetes.io/limit-ranger:
                LimitRanger plugin set: cpu, memory limit for container step-git-source-sources-jl66z; cpu, memory limit for container step-dockerfile; cp...
              kubernetes.io/psp: 00-privileged
              pipeline.tekton.dev/release: v0.17.1
              tekton.dev/ready: READY
Status:       Failed
IP:           100.110.73.177
IPs:
  IP:           100.110.73.177
Controlled By:  TaskRun/python-test-jvp4p
Init Containers:
  place-tools:
    Container ID:  docker://3e079ddfdb70f38f8b6819987de6ce9e715f7a0bbbbd40cfcf112d91313a4153
    Image:         gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/entrypoint:v0.17.1@sha256:ec3db704fd4fd6007f41cb8949f14f1ab0d9d76d472f1942677a9e245a1c35af
    Image ID:      docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/entrypoint@sha256:ec3db704fd4fd6007f41cb8949f14f1ab0d9d76d472f1942677a9e245a1c35af
    Port:          <none>
    Host Port:     <none>
    Command:
      /ko-app/entrypoint
      cp
      /ko-app/entrypoint
      /tekton/tools/entrypoint
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 19 Oct 2020 09:37:47 +0300
      Finished:     Mon, 19 Oct 2020 09:37:47 +0300
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  100Mi
    Requests:
      cpu:        25m
      memory:     100Mi
    Environment:  <none>
    Mounts:
      /tekton/tools from tekton-internal-tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ng6s8 (ro)
Containers:
  step-git-source-sources-jl66z:
    Container ID:  docker://f4c8a7ab144c2b4da08db807d17f8a731894457a2c8421e461aef6a1e7b4a48f
    Image:         gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init:v0.17.1@sha256:a39b8c647d6eaf609dc04bc7b0fc0f787b3b25fd6b1c9b182066a5c29a64a18c
    Image ID:      docker-pullable://gcr.io/tekton-releases/github.com/tektoncd/pipeline/cmd/git-init@sha256:a39b8c647d6eaf609dc04bc7b0fc0f787b3b25fd6b1c9b182066a5c29a64a18c
    Port:          <none>
    Host Port:     <none>
    Command:
      /tekton/tools/entrypoint
    Args:
      -wait_file
      /tekton/downward/ready
      -wait_file_content
      -post_file
      /tekton/tools/0
      -termination_path
      /tekton/termination
      -entrypoint
      /ko-app/git-init
      --
      -url
      https://github.com/serverless/examples
      -path
      /workspace/workspace
      -revision
      master
    State:          Terminated
      Reason:       Completed
      Message:      [{"key":"commit","value":"9eef07b09ee67c33e99c89c73b830a45b7da6ddb","resourceName":"sources","resourceRef":{"name":"sources"}},{"key":"url","value":"https://github.com/serverless/examples","resourceName":"sources","resourceRef":{"name":"sources"}},{"key":"StartedAt","value":"2020-10-19T06:37:49.062Z","type":"InternalTektonResult"}]
      Exit Code:    0
      Started:      Mon, 19 Oct 2020 09:37:48 +0300
      Finished:     Mon, 19 Oct 2020 09:37:52 +0300
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  100Mi
    Requests:
      cpu:                0
      ephemeral-storage:  0
      memory:             0
    Environment:
      HOME:                  /tekton/home
      TEKTON_RESOURCE_NAME:  sources
      HOME:                  /tekton/home
    Mounts:
      /tekton/creds from tekton-creds-init-home-gv65m (rw)
      /tekton/downward from tekton-internal-downward (rw)
      /tekton/home from tekton-internal-home (rw)
      /tekton/results from tekton-internal-results (rw)
      /tekton/tools from tekton-internal-tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ng6s8 (ro)
      /workspace from tekton-internal-workspace (rw)
  step-dockerfile:
    Container ID:  docker://5a9316200e3cbfee6943f5a37115667fc646a22d944caddd0b73581c99414bc3
    Image:         gcr.io/kaniko-project/executor:debug-v0.8.0
    Image ID:      docker-pullable://gcr.io/kaniko-project/executor@sha256:a49506bad46fa4a8ce412239b25ebc2577290cc21ca04a0618d6157e379f811e
    Port:          <none>
    Host Port:     <none>
    Command:
      /tekton/tools/entrypoint
    Args:
      -wait_file
      /tekton/tools/0
      -post_file
      /tekton/tools/1
      -termination_path
      /tekton/termination
      -entrypoint
      /busybox/sh
      --
      -c
      cd /workspace/workspace/aws-python-simple-http-endpoint
      cat <<EOF > Dockerfile
        FROM gcr.io/triggermesh/knative-lambda-python37
      
        ENV _HANDLER "handler.endpoint"
      
        COPY . .
        RUN if [ -f requirements.txt ]; then pip3.7 install -r requirements.txt ;fi
      
        ENTRYPOINT ["/opt/aws-custom-runtime"]
      EOF
      
    State:          Terminated
      Reason:       Completed
      Message:      [{"key":"StartedAt","value":"2020-10-19T06:37:53.336Z","type":"InternalTektonResult"}]
      Exit Code:    0
      Started:      Mon, 19 Oct 2020 09:37:48 +0300
      Finished:     Mon, 19 Oct 2020 09:37:53 +0300
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  100Mi
    Requests:
      cpu:                0
      ephemeral-storage:  0
      memory:             0
    Environment:
      HOME:  /tekton/home
    Mounts:
      /tekton/creds from tekton-creds-init-home-gd7gt (rw)
      /tekton/home from tekton-internal-home (rw)
      /tekton/results from tekton-internal-results (rw)
      /tekton/tools from tekton-internal-tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ng6s8 (ro)
      /workspace from tekton-internal-workspace (rw)
  step-export:
    Container ID:  docker://a3ff868b8c7bd20e941d29ef3068f82457a1db319399ffb6f0891650a7d80344
    Image:         gcr.io/kaniko-project/executor@sha256:a49506bad46fa4a8ce412239b25ebc2577290cc21ca04a0618d6157e379f811e
    Image ID:      docker-pullable://gcr.io/kaniko-project/executor@sha256:a49506bad46fa4a8ce412239b25ebc2577290cc21ca04a0618d6157e379f811e
    Port:          <none>
    Host Port:     <none>
    Command:
      /tekton/tools/entrypoint
    Args:
      -wait_file
      /tekton/tools/1
      -post_file
      /tekton/tools/2
      -termination_path
      /tekton/termination
      -entrypoint
      /kaniko/executor
      --
      --context=/workspace/workspace/aws-python-simple-http-endpoint
      --dockerfile=Dockerfile
      --destination=knative.registry.svc.cluster.local/lambda/python-test:Mwn7H6
    State:          Terminated
      Reason:       Error
      Message:      [{"key":"StartedAt","value":"2020-10-19T06:37:53.614Z","type":"InternalTektonResult"}]
      Exit Code:    1
      Started:      Mon, 19 Oct 2020 09:37:48 +0300
      Finished:     Mon, 19 Oct 2020 09:38:58 +0300
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     200m
      memory:  100Mi
    Requests:
      cpu:                0
      ephemeral-storage:  0
      memory:             0
    Environment:
      HOME:           /tekton/home
      DOCKER_CONFIG:  /
    Mounts:
      /tekton/creds from tekton-creds-init-home-cdvj4 (rw)
      /tekton/home from tekton-internal-home (rw)
      /tekton/results from tekton-internal-results (rw)
      /tekton/tools from tekton-internal-tools (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ng6s8 (ro)
      /workspace from tekton-internal-workspace (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  tekton-internal-workspace:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tekton-internal-home:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tekton-internal-results:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tekton-internal-tools:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  tekton-internal-downward:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations['tekton.dev/ready'] -> ready
  tekton-creds-init-home-gv65m:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  tekton-creds-init-home-gd7gt:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  tekton-creds-init-home-cdvj4:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  default-token-ng6s8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-ng6s8
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>
Hi-Fi commented

Tested this a bit more, and it got interesting:
When auth is created with single word in project, it works:
tm set registry-auth --registry private.interlnal.registry --project demo test

But when project has slashes, that error with Dockerfile comes (and things only work when Dockerfile if given with full path):
tm set registry-auth --registry private.interlnal.registry --project demo/with/paths test2

Edit:
When I generate secret with backslashes
tm set registry-auth --registry private.interlnal.registry --project demo\/with\/paths test3 things seem to be working fine, even the secrets are equal.

Probably this is Kaniko issue, as that command just generates the secret?

tzununbekov commented

set registry-auth command creates dockerconfigjson type of secret, mounts it into the taskrun build pod, sets DOCKER_CONFIG env variable, and updates SA's image pull secret. Also, the registry and the project is used in kaniko's destination parameter, and according to the describe pod output slashes are all fine:

--destination=PRIVATE.SERVER.URL/klr-poc/images/python-test:furPRD

I still cannot reproduce the issue even if I use project value with the slashes but I noticed that my test environment is few versions behind (tekton, k8s), so maybe there were some changes in related components. I'll update the cluster, run some more tests and get back to you.

Hi-Fi commented

Just figured this out, and it was my own fault. We have runasuser-admission-controller in use, and it puts securityContext to pod level, which prevents Kaniko to work. Maybe backslashed version worked because that controller pod was down or something.

But disabling the controller for namespace made things work. Sorry for extra work caused.

tzununbekov commented

I'm glad we figured it out, because updating the cluster components didn't change anything, and I had no other ideas.