jenkins-log.txt: Directory nonexistent

Question

jenkins-log.txt: Directory nonexistent

Closed this issue 7 years ago · 28 comments

minotaurrr commented 7 years ago

Issues and Contributing

Please note that only issues related to this Docker image will be addressed here.

If you have Docker related issues, please ask in the Docker user mailing list.
If you have Jenkins related issues, please ask in the Jenkins mailing lists.
If you are not sure, then this is probably not the place to create an issue and you should use any of the previously mentioned mailing lists.

If after going through the previous checklist you still think you should create an issue here please provide:

Docker commands that you execute

Actual result

[testing] Running shell script
sh: 1: cannot create /var/jenkins_home/workspace/testing@tmp/durable-518faec8/jenkins-log.txt: Directory nonexistent
sh: 1: cannot create /var/jenkins_home/workspace/testing@tmp/durable-518faec8/jenkins-result.txt: Directory nonexistent

Expected outcome

Have you tried a non-dockerized Jenkins and get the expected outcome?

Output of `docker version`

Docker version 17.12.0-ce, build c97c6d6

Other relevant information

Hi guys,

I'm trying to run this Jenkinsfile but it keeps giving me the above error in the test stage.

node {
    def app
    stage ('checkout') {
        checkout scm
    }

    stage ('build') {
        app = docker.build('testing')
    }

    stage ('test') {
        app.inside {
            sh 'npm test'
        } 
    }
}

Ran docker container like this

docker run -d -u root -p 8080:8080 -v /usr/local/bin:/user/bin -v /var/run/docker.sock:/var/run/docker.sock jenkins

Couldn't find any answer on Google or here. Can anyone help please?

Answer 1 · 2018-01-14T10:02:02.000Z

you are executing sh in a docker container that is actually in the host, I don't think that will work with docker pipeline from a docker container, only from host (unless you share the /var/jenkins_home/workspace between both containers).
You should ask in jenkins-user mailing list

Answer 2 · 2018-01-17T08:15:47.000Z

It was working until recent updates either on Docker or Jenkins plugins.

Now it looks like if you run Jenkins as a Container and inside Jenkins you run Docker.inside command from pipeline it will use host filesystem, while previously it used Jenkins container volume.

Answer 3 · 2018-01-17T10:37:44.000Z

@midN Same here.

| Pipeline: DeclarativeAn opinionated, declarative Pipeline. | 1.2.6 | |
| Pipeline: Declarative Extension Points APIAPIs for extension points used in Declarative Pipelines. | 1.2.6 | |
| Pipeline: Model APIModel API for Declarative Pipeline. | 1.2.6 | |
| Pipeline: Stage Tags MetadataLibrary plugin for Pipeline stage tag metadata. | 1.2.6

These were updated from 1.2.5 and it stopped working.

Answer 4 · 2018-01-17T10:45:05.000Z

My workaround to not do changes in every project:
I've had a separate folder on HOST shared to Jenkins container to store workspaces such as "/imafolder" and on Jenkins container it was booted under "/var/imafolder".

Since the folder is shared, i just changed HOST folder to match exactly same as Jenkins used "/var/imafolder".

With these changes once you run a new container from Jenkins container, it will use given folder on HOST but since it's shared to Jenkins on same location, there will be no problems whatsoever.

Answer 5 · 2018-01-17T11:44:27.000Z

@midN do you mind sharing the Docker command you ran?

Answer 6 · 2018-01-17T15:02:07.000Z

@minotaurrr
Here are steps i use:

On Jenkins HOST i created directory to hold workspaces for projects built with Jenkinsfile

# Jenkins workspaces
mkdir -p /var/jenkins_workspaces
chmod -R 777 /var/jenkins_workspaces

Mount given dir as volume to Jenkins container ( i run ECS, so just -v pretty much. )

- Name: workspaces_volume
  Host:
    SourcePath: /var/jenkins_workspaces

Mount volume to EXACTLY same path on Container

- ContainerPath: /var/jenkins_workspaces
  SourceVolume: workspaces_volume

Wrap commands ran in Jenkinsfile with your custom workspace

ws("/var/jenkins_workspaces/helloworld"){
  YOUR STAGES

  docker.image("x").inside("-u 0:0") {
  }
}

So basically you are telling your Jenkins container to use "/var/jenkins_workspaces" dir on Container which is really a volume on Host with same path "/var/jenkins_workspaces".
Now when i run a "docker.image.inside", it will use Hosts "/var/jenkins_workspaces" path, which is mounted to Jenkins.

Answer 7 · 2018-01-17T19:58:58.000Z

@midN thanks for sharing! I’ll try it and let you know later :)

Answer 8 · 2018-01-18T18:43:02.000Z

Having the same issue here.

Answer 9 · 2018-01-25T15:35:48.000Z

Same issue here. And tried @midN workaround, it works great!

Answer 10 · 2018-01-25T15:38:13.000Z

As we run Jenkins jnlp docker in ECS, it's really annoying that if we scale cluster instance. You have to create jenkins workspaces manually on each cluster instances.
Does anyone know with which (Docker or jenkins) version it still works with container volume instead of host volume? Maybe I can downgrade it.

Answer 11 · 2018-01-26T08:39:29.000Z

@tao-zhang I do it in user-data, so you don't need to create it manually everytime :)
Or if you don't use autoscaling, create custom AMI with folder created.

Or, you can use EFS to have shared workspace for all ec2 machines, although creating docker images on EFS was quite slow compared to EBS, that's why i moved my workspaces out to EBS

Answer 12 · 2018-01-30T02:32:04.000Z

I have the same issue. Docker on OSX.
Docker 17.12.0-ce
Docker plugin : 1.1.2
Jenkins : 2.104

Hi guys - is there any other workaround for this issue? @midN's didn't go well in my context.

Answer 13 · 2018-01-30T09:53:40.000Z

In order to avoid more me too comments this is an issue with docker-workflow-plugin see https://issues.jenkins-ci.org/browse/JENKINS-49063 which is already merged in docker-workflow-plugin

See @midN comment #626 (comment) for workaround on ECS. Similar in other environments

Answer 14 · 2018-09-19T20:41:38.000Z

Is this fixed? Without workarounds?
I'm still seeing issues, and I'm just using a sample Jenkins file from Jenkins website:

    agent none 
    stages {
        stage('Example Build') {
            agent { docker 'maven:3-alpine' } 
            steps {
                echo 'Hello, Maven'
                sh 'mvn --version'
            }
        }
        stage('Example Test') {
            agent { docker 'openjdk:8-jre' } 
            steps {
                echo 'Hello, JDK'
                sh 'java -version'
            }
        }
    }
}

Running shell script
sh: can't create /home/jenkins/workspace/test@tmp/durable-1878c51f/jenkins-log.txt: nonexistent directory
sh: can't create /home/jenkins/workspace/test@tmp/durable-1878c51f/jenkins-result.txt.tmp: nonexistent directory
mv: can't rename '/home/jenkins/workspace/test@tmp/durable-1878c51f/jenkins-result.txt.tmp': No such file or directory

Answer 15 · 2019-01-14T20:35:05.000Z

This is still not fixed, so why is it closed?

Answer 16 · 2019-01-14T21:42:19.000Z

@carlossg @ndeloof is this supposed to be fixed by jenkinsci/docker-workflow-plugin@f342888 ? Looks like https://issues.jenkins-ci.org/browse/JENKINS-49063 points to jenkinsci/docker-workflow-plugin#128 but on both this and the issue on jenkins-ci.org people are complaining that it's not fixed (that's what I'm seeing too).

Answer 17 · 2019-03-26T22:57:17.000Z

Your image may be running as root and not jenkins this means that any job that runs a jenkins may not be able to create a new file in the necessary directories.

Answer 18 · 2019-04-11T09:26:19.000Z

Same issue, after few hours research, I got a magic fixed.
have you try to add label is slave machine running?

node (label: 'master') {
    ...
}

Answer 19 · 2019-05-29T13:51:44.000Z

We're having the issue too, with a descriptive pipeline and using the docker agent. The workaround proposed wouldn't work in our case though so it would be great if this was fixed.

Answer 20 · 2019-07-04T11:13:10.000Z

Same issue here. As you can see in this output, Jenkins tries to write in "$WORKSPACE@tmp/durable-70e158c6" inside the alpine container while this directory don't exists, it was "$WORKSPACE@tmp/durable-8eefd86a" in the parent agent.

Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node

Running on slave-000kjwxb51gdi on LocalDockerHost in /home/jenkins/workspace/prueba-pipeline
[Pipeline] {
[Pipeline] sh

+ id
uid=1000(jenkins) gid=1000(jenkins) groups=1000(jenkins)
[Pipeline] sh
+ ls -l /home/jenkins/workspace/prueba-pipeline@tmp
total 4
drwxr-sr-x    2 jenkins  jenkins       4096 Jul  4 11:04 durable-8eefd86a
[Pipeline] sh
+ docker inspect -f . alpine:3.9
.
[Pipeline] withDockerContainer
slave-000kjwxb51gdi on LocalDockerHost seems to be running inside container 862f601a536f9678f43a29d0dec0d3c3b89bb439cde8f174bcd61ecff3512cd3
but /home/jenkins/workspace/prueba-pipeline could not be found among [/git]
but /home/jenkins/workspace/prueba-pipeline@tmp could not be found among [/git]
$ docker run -t -d -u 1000:1000 -w /home/jenkins/workspace/prueba-pipeline -v /home/jenkins/workspace/prueba-pipeline:/home/jenkins/workspace/prueba-pipeline:rw,z -v /home/jenkins/workspace/prueba-pipeline@tmp:/home/jenkins/workspace/prueba-pipeline@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** alpine:3.9 cat

$ docker top 74db144b89cbd0da0c9245c713d2fdf67d37e0c7eca5d9fcd2693d1e754c10e6 -eo pid,comm
[Pipeline] {
[Pipeline] echo
hola mundo
[Pipeline] sh
sh: can't create /home/jenkins/workspace/prueba-pipeline@tmp/durable-70e158c6/jenkins-log.txt: nonexistent directory
sh: can't create /home/jenkins/workspace/prueba-pipeline@tmp/durable-70e158c6/jenkins-result.txt.tmp: nonexistent directory
mv: can't rename '/home/jenkins/workspace/prueba-pipeline@tmp/durable-70e158c6/jenkins-result.txt.tmp': No such file or directory

Answer 21 · 2019-09-20T20:28:02.000Z

I hit this issue as well after upgrading Jenkins and the Docker plugin. My workaround was to make similar changes as those suggested by @midN , but within the "Configure Jenkins" UI.
There, I changed the Container Template Working Directory from /home/jenkins/agent to just /home/jenkins so that the directory /home/jenkins/workspace is at the exact same location in the child and parent containers.

Answer 22 · 2021-11-12T20:29:42.000Z

This took me way to long to figure out. I also think I still have an error with my setup because it used to work. Anyway here is the workaround that works for me.

Consider the following logs (stolen form @javiersanp):

$ docker run -t -d -u 1000:1000 -w /home/jenkins/workspace/prueba-pipeline -v /home/jenkins/workspace/prueba-pipeline:/home/jenkins/workspace/prueba-pipeline:rw,z -v /home/jenkins/workspace/prueba-pipeline@tmp:/home/jenkins/workspace/prueba-pipeline@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** alpine:3.9 cat

$ docker top 74db144b89cbd0da0c9245c713d2fdf67d37e0c7eca5d9fcd2693d1e754c10e6 -eo pid,comm
[Pipeline] {
[Pipeline] echo
hola mundo
[Pipeline] sh
sh: can't create /home/jenkins/workspace/prueba-pipeline@tmp/durable-70e158c6/jenkins-log.txt: nonexistent directory
sh: can't create /home/jenkins/workspace/prueba-pipeline@tmp/durable-70e158c6/jenkins-result.txt.tmp: nonexistent directory
mv: can't rename '/home/jenkins/workspace/prueba-pipeline@tmp/durable-70e158c6/jenkins-result.txt.tmp': No such file or directory

This tells you that jenkins has its workspaces under /home/jenkins/workspace. Your docker image started by jenkins now tries to mount the folder as a docker volume. Since your jenkins runs in docker as a dood this can NOT work, because your host does not have the directory only your dockerized jenkins has.

As a quick and dirty workaround:

mkdir -p /home
cd /home
ln -s /path/to/your/docker/jenkins/home/volume/folder .

Tada, eveything works again. Ofc you have to adjust this workaround to your path. In my case it was /var/jenkins_home.

Answer 23 · 2022-01-03T19:22:44.000Z

This took me way to long to figure out. I also think I still have an error with my setup because it used to work. Anyway here is the workaround that works for me.

Consider the following logs (stolen form @javiersanp):
$ docker run -t -d -u 1000:1000 -w /home/jenkins/workspace/prueba-pipeline -v /home/jenkins/workspace/prueba-pipeline:/home/jenkins/workspace/prueba-pipeline:rw,z -v /home/jenkins/workspace/prueba-pipeline@tmp:/home/jenkins/workspace/prueba-pipeline@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** alpine:3.9 cat

$ docker top 74db144b89cbd0da0c9245c713d2fdf67d37e0c7eca5d9fcd2693d1e754c10e6 -eo pid,comm
[Pipeline] {
[Pipeline] echo
hola mundo
[Pipeline] sh
sh: can't create /home/jenkins/workspace/prueba-pipeline@tmp/durable-70e158c6/jenkins-log.txt: nonexistent directory
sh: can't create /home/jenkins/workspace/prueba-pipeline@tmp/durable-70e158c6/jenkins-result.txt.tmp: nonexistent directory
mv: can't rename '/home/jenkins/workspace/prueba-pipeline@tmp/durable-70e158c6/jenkins-result.txt.tmp': No such file or directory
This tells you that jenkins has its workspaces under /home/jenkins/workspace. Your docker image started by jenkins now tries to mount the folder as a docker volume. Since your jenkins runs in docker as a dood this can NOT work, because your host does not have the directory only your dockerized jenkins has.

As a quick and dirty workaround:
mkdir -P /home
cd /home
ln -s /path/to/your/docker/jenkins/home/volume/folder .
Tada, eveything works again. Ofc you have to adjust this workaround to your path. In my case it was /var/jenkins_home.

Just a heads up, using -P will error out for some. its supposed to be lowercase :)

Answer 24 · 2022-01-04T14:55:24.000Z

Thank you. Updated!

Answer 25 · 2022-01-14T07:44:14.000Z

workaround

Hi,

I don't really understand where you put that.
Is it in your jenkins pipeline or in your jenkins docker?

I have the same problem as you. I think that the volume does not go up in the docker :

Jenkins does not seem to be running inside a container
$ docker run -t -d -u 1000:1000 -u 0 -w /var/jenkins_home/workspace/LEVEL-UP/NODE-BACK@2 -v /var/jenkins_home/workspace/LEVEL-UP/NODE-BACK@2:/var/jenkins_home/workspace/LEVEL-UP/NODE-BACK@2:rw,z -v /var/jenkins_home/workspace/LEVEL-UP/NODE-BACK@2@tmp:/var/jenkins_home/workspace/LEVEL-UP/NODE-BACK@2@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:14.16 cat

As you can see there is Jenkins does not seem to be running inside a container.

thanks

Answer 26 · 2022-05-27T14:07:54.000Z

workaround

Hi,

I don't really understand where you put that. Is it in your jenkins pipeline or in your jenkins docker?