erda-project/erda

java编译，maven仓库的缓存问题。

Closed this issue · 15 comments

luoluo2001108 commented

编译java项目的时候，需要从maven仓库里下载jar包和pom文件，但是每次编译jar包都会去maven仓库拉取，下载都会耗时，影响效率，有什么方式可以缓存maven仓库么？

日志如下，每次编译都有大量的download 信息：

2022-12-22 17:49:42successfully loaded action config
2022-12-22 17:49:42successfully replaced action placeholder
2022-12-22 17:49:42change workding directory to: /.pipeline/container/context/git-checkout
2022-12-22 17:49:42build maven app
2022-12-22 17:49:42Run: /bin/bash, [-c MAVEN_OPTS=-Xmx2016m mvn clean package -e -B -U -Dmaven.test.skip -s /opt/action/comp/maven/settings.xml ]
2022-12-22 17:49:43[INFO] Error stacktraces are turned on.
2022-12-22 17:49:43[INFO] Scanning for projects...

2022-12-22 17:49:44[INFO] Downloading from terminus: http://192.168.14.12:8081/nexus/repository/public/com/dhc/leapower/business/leapower-business/4.1-SNAPSHOT/maven-metadata.xml

2022-12-22 17:49:44[INFO] Downloaded from terminus: http://192.168.14.12:8081/nexus/repository/public/com/dhc/leapower/business/leapower-business/4.1-SNAPSHOT/maven-metadata.xml (617 B at 6.3 kB/s)
2022-12-22 17:49:44[INFO] Downloading from terminus: http://192.168.14.12:8081/nexus/repository/public/com/dhc/leapower/business/leapower-business/4.1-SNAPSHOT/leapower-business-4.1-20221219.030915-44.pom
2022-12-22 17:49:44[INFO] Downloaded from terminus: http://192.168.14.12:8081/nexus/repository/public/com/dhc/leapower/business/leapower-business/4.1-SNAPSHOT/leapower-business-4.1-20221219.030915-44.pom (8.6 kB at 720 kB/s)

chengjoey commented

java编译打包的过程可以通过以下两种方法进行加速：

  1. snapshot 用 -U 强制更新保证正确性，切到release版本可以一定程度上加快
  2. 利用流水线的缓存功能，将jar进行缓存加快构建，详细办法可以参考文档
luoluo2001108 commented

我们之前已经加上了caches选项，没错误吧？

  • stage:
    - java:
    alias: hse
    description: 针对 java 工程的编译打包任务，产出可运行镜像
    params:
    build_type: maven
    container_type: spring-boot
    target: ./target/imom-business-hse-4.1-SNAPSHOT.jar
    workdir: ${git-checkout}
    resources:
    cpu: 2
    caches:
    - path: /root/.m2/repository

但还是编译的时候，每次都从maven镜像仓库拉取资源 -
我们想确认的是，每次都从maven镜像仓库拉取资源，这个有方法可以解决么？

chengjoey commented

我们之前已经加上了caches选项，没错误吧？

  • stage:
    • java:
      alias: hse
      description: 针对 java 工程的编译打包任务，产出可运行镜像
      params:
      build_type: maven
      container_type: spring-boot
      target: ./target/imom-business-hse-4.1-SNAPSHOT.jar
      workdir: ${git-checkout}
      resources:
      cpu: 2
      caches:
    • path: /root/.m2/repository

但还是编译的时候，每次都从maven镜像仓库拉取资源 - 我们想确认的是，每次都从maven镜像仓库拉取资源，这个有方法可以解决么？

可以看下任务的日志确定一下缓存是否成功生效，如果有看到get action cache: /root/.m2/repository success则说明已经成功缓存了，与你本地编译过应该是一样的效果

luoluo2001108 commented

任务日志中没有看到你说的字段，咋办

日志如下：
2022-12-26 08:49:26successfully loaded action config
2022-12-26 08:49:27successfully replaced action placeholder
2022-12-26 08:49:27change workding directory to: /.pipeline/container/context/git-checkout
2022-12-26 08:49:27build maven app
2022-12-26 08:49:27Run: /bin/bash, [-c MAVEN_OPTS=-Xmx2016m mvn clean package -e -B -U -Dmaven.test.skip -s /opt/action/comp/maven/settings.xml ]
2022-12-26 08:49:28[INFO] Error stacktraces are turned on.
2022-12-26 08:49:28[INFO] Scanning for projects...
2022-12-26 08:49:28[INFO] Downloading from terminus: http://192.168.14.12:8081/nexus/repository/public/com/dhc/leapower/business/leapower-business/4.
后面就是一大堆的从maven仓库下载的信息日志

chengjoey commented

任务日志中没有看到你说的字段，咋办

日志如下： 2022-12-26 08:49:26successfully loaded action config 2022-12-26 08:49:27successfully replaced action placeholder 2022-12-26 08:49:27change workding directory to: /.pipeline/container/context/git-checkout 2022-12-26 08:49:27build maven app 2022-12-26 08:49:27Run: /bin/bash, [-c MAVEN_OPTS=-Xmx2016m mvn clean package -e -B -U -Dmaven.test.skip -s /opt/action/comp/maven/settings.xml ] 2022-12-26 08:49:28[INFO] Error stacktraces are turned on. 2022-12-26 08:49:28[INFO] Scanning for projects... 2022-12-26 08:49:28[INFO] Downloading from terminus: http://192.168.14.12:8081/nexus/repository/public/com/dhc/leapower/business/leapower-business/4. 后面就是一大堆的从maven仓库下载的信息日志

可以分别点击标准日志和错误日志看下有没有，而且用的是java action，默认会在编译的时候加上-U强制从远端更新snapshot和release，所以可能没有办法不从maven仓库拉取

luoluo2001108 commented

Error日志下有，但是还是每次都强制远端更新了，确认没有办法不从maven仓库拉取了么？
日志如下
2022-12-23 12:20:23[Platform Log] INFO [2022-12-23T12:20:23.705863717+08:00] get action cache: /root/.m2/repository success
2022-12-23 12:20:23openjdk version "1.8.0_272"
2022-12-23 12:20:23OpenJDK Runtime Environment (build 1.8.0_272-b10)
2022-12-23 12:20:23OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)
2022-12-23 12:22:30#1 [internal] load build definition from Dockerfile
2022-12-23 12:22:30#1 transferring dockerfile: 967B done
2022-12-23 12:22:30#1 DONE 0.0s
2022-12-23 12:22:30

sfwn commented

参考 https://github.com/erda-project/erda-actions/blob/master/actions/java/1.0/spec.yml#L22
使用 build_cmd 指定编译命令，把 -U 去掉，就不会强制检查和拉取 SNAPSHOT 包。

luoluo2001108 commented

加了 build_cmd 去掉了-U，没看到效果。

  • stage:
    - java:
    alias: patrol
    description: 针对 java 工程的编译打包任务，产出可运行镜像
    params:
    build_cmd: mvn clean package -e -B -Dmaven.test.skip
    build_type: maven
    container_type: spring-boot
    target: ./target/imom-business-patrol-4.1-SNAPSHOT.jar
    workdir: ${git-checkout}
    resources:
    cpu: 2
    caches:
    - path: /root/.m2/repository
    日志如下：
    2022-12-27 14:52:56successfully loaded action config
    2022-12-27 14:52:56successfully replaced action placeholder
    2022-12-27 14:52:56change workding directory to: /.pipeline/container/context/git-checkout
    2022-12-27 14:52:56build maven app
    2022-12-27 14:52:56Run: /bin/bash, [-c MAVEN_OPTS=-Xmx2016m **_

mvn clean package -e -B -Dmaven.test.skip -s /opt/action/comp/maven/settings.xml ]

_**
2022-12-27 14:52:57[INFO] Error stacktraces are turned on.
2022-12-27 14:52:57[INFO] Scanning for projects...
2022-12-27 14:52:58[INFO] Downloading from terminus: http://192.168.14.12:8081/nexus/repository/public/com/dhc/leapower/business/leapower-business/4.1-SNAPSHOT/maven-metadata.xml
2022-12-27 14:52:58[INFO] Downloaded from terminus: http://192.168.14.12:808
下面还是大段的从maven仓库拉取的日志

  • 👍1
sfwn commented
  1. 确认 cache 是否加载成功：https://github.com/erda-project/erda/blob/master/internal/tools/pipeline/actionagent/step_restore.go#L84
  2. 修改 build_cmd：ls 一下看看 .m2 目录下有没有具体的包
luoluo2001108 commented
  1. 确认 cache 是否加载成功：https://github.com/erda-project/erda/blob/master/internal/tools/pipeline/actionagent/step_restore.go#L84
  2. 修改 build_cmd：ls 一下看看 .m2 目录下有没有具体的包

java这个stage好像没法用ls, build_cmd 不支持输入ls,不支持数组的方式输入命令，
我看了一下 命令后面会加上 -s /opt/action/comp/maven/settings.xml

我说的下载maven重复下载，可能跟这个settings.yml有关，这里面有个updatepolicy是always。

  • 👍1
sfwn commented

I think you are right!

可以通过流水线配置上传自定义 settings 文件，假设名字叫 settings_xml；
然后使用 params: options: -s {{ configs.settings_xml }} 指定使用。

另外还是要从日志里确认下 cache 是否加载成功了。

luoluo2001108 commented

build_cmd 里加了一个选项，已经发现使用了指定的settings。

stage:
- java:
alias: patrol
description: 针对 java 工程的编译打包任务，产出可运行镜像
params:
build_cmd: mvn clean package -e -B --settings ((MAVEN_SETTING_FILE)) -Dmaven.test.skip=true
build_type: maven
container_type: spring-boot
target: ./target/imom-business-patrol-4.1-SNAPSHOT.jar
workdir: ${git-checkout}
resources:
cpu: 2
caches:
- path: /root/.m2/repository

至于缓存有没有生效，这是日志：
2022-12-28 11:14:52[Platform Log] INFO [2022-12-28T11:14:52.35214582+08:00] get action cache: /root/.m2/repository success
2022-12-28 11:14:52openjdk version "1.8.0_272"
2022-12-28 11:14:52OpenJDK Runtime Environment (build 1.8.0_272-b10)
2022-12-28 11:14:52OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)
2022-12-28 11:15:23#1 [internal] load build definition from Dockerfile
2022-12-28 11:15:23#1 transferring dockerfile: 967B done
2022-12-28 11:15:23#1 DONE 0.0s

2#19 exporting to image
2022-12-28 11:15:32#19 exporting layers
2022-12-28 11:15:41#19 exporting layers 9.4s done
2022-12-28 11:15:41#19 exporting manifest sha256:e671914277529109b0f4792d1de10833d7009ffd69dfc5e074fdb4721a6b3463
2022-12-28 11:15:41#19 exporting manifest sha256:e671914277529109b0f4792d1de10833d7009ffd69dfc5e074fdb4721a6b3463 done
2022-12-28 11:15:41#19 exporting config sha256:d6c61e5a679a6f90b318c3161250d8facb0c64ab8a7d3cf8a02b9d019242b01a done
2022-12-28 11:15:41#19 pushing layers
2022-12-28 11:15:45[Platform Log] INFO [2022-12-28T11:15:45.07757201+08:00] upload action cache error: /root/.m2/repository is not dir
2022-12-28 11:15:45#19 pushing layers 3.3s done

chengjoey commented

build_cmd 里加了一个选项，已经发现使用了指定的settings。

stage: - java: alias: patrol description: 针对 java 工程的编译打包任务，产出可运行镜像 params: build_cmd: mvn clean package -e -B --settings ((MAVEN_SETTING_FILE)) -Dmaven.test.skip=true build_type: maven container_type: spring-boot target: ./target/imom-business-patrol-4.1-SNAPSHOT.jar workdir: ${git-checkout} resources: cpu: 2 caches: - path: /root/.m2/repository

至于缓存有没有生效，这是日志：

2022-12-28 11:14:52[Platform Log] INFO [2022-12-28T11:14:52.35214582+08:00] get action cache: /root/.m2/repository success
2022-12-28 11:14:52openjdk version "1.8.0_272"
2022-12-28 11:14:52OpenJDK Runtime Environment (build 1.8.0_272-b10)
2022-12-28 11:14:52OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)
2022-12-28 11:15:23#1 [internal] load build definition from Dockerfile
2022-12-28 11:15:23#1 transferring dockerfile: 967B done
2022-12-28 11:15:23#1 DONE 0.0s
2#19 exporting to image 2022-12-28 11:15:32#19 exporting layers 2022-12-28 11:15:41#19 exporting layers 9.4s done 2022-12-28 11:15:41#19 exporting manifest sha256:e671914277529109b0f4792d1de10833d7009ffd69dfc5e074fdb4721a6b3463 2022-12-28 11:15:41#19 exporting manifest sha256:e671914277529109b0f4792d1de10833d7009ffd69dfc5e074fdb4721a6b3463 done 2022-12-28 11:15:41#19 exporting config sha256:d6c61e5a679a6f90b318c3161250d8facb0c64ab8a7d3cf8a02b9d019242b01a done 2022-12-28 11:15:41#19 pushing layers 2022-12-28 11:15:45[Platform Log] INFO [2022-12-28T11:15:45.07757201+08:00] upload action cache error: /root/.m2/repository is not dir 2022-12-28 11:15:45#19 pushing layers 3.3s done

查看下面的日志似乎这会儿上传缓存失败了，日志里还有大量从maven仓库下载jar包的日志吗

luoluo2001108 commented

这个路径/root/.m2/repository是不是也是由settings里面的配置决定的?

我们自定义的settings文件 里，localRepository配置是网盘的路径/netdata/devops/storage/repository/，这个网盘路径下我看到了下载的maven文件。加了这个配置，maven才不下载