Missing Jar during Randomwalk

Question

Missing Jar during Randomwalk

Opened this issue 3 years ago · 9 comments

While running the MultiTable RW module, I saw it die due to this exception.

2022-02-17T15:19:19,515 [testing.randomwalk.Framework] INFO : Test finished
Exception in thread "Thread-3" java.lang.RuntimeException: java.nio.file.NoSuchFileException: /home/mike/workspace/accumulo-testing/target/dependency/hadoop-client-api.jar
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3037)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2984)
	at org.apache.hadoop.conf.Configuration.loadProps(Configuration.java:2864)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2846)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200)
	at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1812)
	at org.apache.hadoop.conf.Configuration.getTimeDuration(Configuration.java:1789)
	at org.apache.hadoop.util.ShutdownHookManager.getShutdownTimeout(ShutdownHookManager.java:183)
	at org.apache.hadoop.util.ShutdownHookManager.shutdownExecutor(ShutdownHookManager.java:145)
	at org.apache.hadoop.util.ShutdownHookManager.access$300(ShutdownHookManager.java:65)
	at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:102)
Caused by: java.nio.file.NoSuchFileException: /home/mike/workspace/accumulo-testing/target/dependency/hadoop-client-api.jar
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149)
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
	at java.base/java.nio.file.Files.readAttributes(Files.java:1764)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1259)
	at java.base/java.util.zip.ZipFile$CleanableResource.(ZipFile.java:831)
	at java.base/java.util.zip.ZipFile$CleanableResource$FinalizableResource.(ZipFile.java:857)
	at java.base/java.util.zip.ZipFile$CleanableResource.get(ZipFile.java:846)
	at java.base/java.util.zip.ZipFile.(ZipFile.java:248)
	at java.base/java.util.zip.ZipFile.(ZipFile.java:177)
	at java.base/java.util.jar.JarFile.(JarFile.java:350)
	at java.base/sun.net.www.protocol.jar.URLJarFile.(URLJarFile.java:103)
	at java.base/sun.net.www.protocol.jar.URLJarFile.getJarFile(URLJarFile.java:72)
	at java.base/sun.net.www.protocol.jar.JarFileFactory.get(JarFileFactory.java:99)
	at java.base/sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:125)
	at java.base/sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:155)
	at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2957)
	at org.apache.hadoop.conf.Configuration.getStreamReader(Configuration.java:3053)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3011)
	... 10 more

Answer 1 · 2022-02-27T15:29:13.000Z

Did you run the build script or did the rwalk script run the build script? That is what creates the jar and that dependency directory. I confirmed on my end, that the jar creation takes place on the latest commit.

Answer 2 · 2022-02-28T14:26:51.000Z

Did you run the build script or did the rwalk script run the build script?

I did not run the build script manually.

Answer 3 · 2022-03-17T18:11:58.000Z

I saw this again and after doing a build in testing, there is no jar:
accumulo-testing/target/dependency/hadoop-client-api.jar

The directory ../target/dependency does not exist. This is created by the shaded jar build:

[INFO] --- maven-dependency-plugin:3.1.2:copy-dependencies (default-cli) @ accumulo-testing ---
[INFO] Copying hadoop-client-api-3.2.2.jar to /home/mike/workspace/accumulo-testing/target/dependency/hadoop-client-api.jar
[INFO] Copying hadoop-client-runtime-3.2.2.jar to /home/mike/workspace/accumulo-testing/target/dependency/hadoop-client-runtime.jar

So this could be happening if a build occurred but the shaded jar didn't get built.

Answer 4 · 2022-03-17T18:17:50.000Z

So this could be happening if a build occurred but the shaded jar didn't get built.

For reference, the shaded jar doesn't get built by default, it requires -Pcreate-shade-jar to activate that profile. The bin/performance script and conf/env.sh both activate this profile.

Answer 5 · 2022-07-24T17:42:07.000Z

@milleruntime Is there anything actionable to be done on this issue?

Answer 6 · 2022-07-25T16:32:02.000Z

I think the actionable thing is that if randomwalk requires the shaded jar, it might need to build it itself, or have instructions to build it.

Answer 7 · 2022-07-25T16:44:07.000Z

I think the actionable thing is that if randomwalk requires the shaded jar, it might need to build it itself, or have instructions to build it.

The rwalk script already calls source "${bin_dir}/build" so I am not sure there is anything to be done.

Answer 8 · 2022-07-25T16:48:12.000Z

Is it possible that build script won't build the shaded jar if there's another jar that's already present? If so, that could be fixed. Otherwise, yeah, there might not be anything to do here.

Answer 9 · 2022-08-07T16:53:15.000Z

Is it possible that build script won't build the shaded jar if there's another jar that's already present? If so, that could be fixed. Otherwise, yeah, there might not be anything to do here.

That is correct. The build script will not re-build the jars if they are currently present. However, I haven't really run into the original issue in my testing. I can see there being a possible benefit to echo to the shell if the jar is present so a user could be more aware that they may have old jars present.