Bloop fails to compile project with weird error

Question

Bloop fails to compile project with weird error

ktonga opened this issue 2 years ago · 11 comments

It happens for only one of the modules (maybe some other after passing that one too) in a multi-module project, there are many modules before that one is attempted which finish compiling without issue. Also if I compile using sbt it doesn't happen.

I wasn't able to isolate the issue in a smaller self-contained project. I can help with any debugging if I'm guided, TBH I don't know where I could start from. To see that error I had to add the --verbose flag.

[E] Unexpected error when compiling merchant-model: 'null'
[T] java.nio.file.ClosedFileSystemException
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.ensureOpen(ZipFileSystem.java:1628)
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipFileSystem.getFileAttributes(ZipFileSystem.java:536)
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:767)
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipPath.readAttributes(ZipPath.java:777)
[T] 	jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:276)
[T] 	java.base/java.nio.file.Files.readAttributes(Files.java:1851)
[T] 	java.base/java.nio.file.Files.size(Files.java:2468)
[T] 	sbt.internal.inc.HashUtil$.farmHash(HashUtil.scala:31)
[T] 	sbt.internal.inc.FarmHash$.ofPath(Stamp.scala:89)
[T] 	sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$3(Stamp.scala:233)
[T] 	sbt.internal.inc.Stamper$.tryStamp(Stamp.scala:188)
[T] 	sbt.internal.inc.Stamper$.$anonfun$forHashInRootPaths$1(Stamp.scala:233)
[T] 	sbt.internal.inc.UncachedStamps.library(Stamp.scala:468)
[T] 	sbt.internal.inc.InitialStamps.$anonfun$library$1(Stamp.scala:422)
[T] 	scala.collection.mutable.HashMap.getOrElseUpdate(HashMap.scala:86)
[T] 	sbt.internal.inc.InitialStamps.library(Stamp.scala:422)
[T] 	sbt.internal.inc.TimeWrapBinaryStamps.$anonfun$library0$1(Stamp.scala:446)
[T] 	sbt.internal.inc.Stamper$.$anonfun$timeWrap$1(Stamp.scala:252)
[T] 	sbt.internal.inc.TimeWrapBinaryStamps.library(Stamp.scala:447)
[T] 	sbt.internal.inc.bloop.internal.BloopAnalysisCallback.$anonfun$addProductsAndDeps$16(BloopAnalysisCallback.scala:400)
[T] 	scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
[T] 	scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
[T] 	scala.collection.TraversableLike.map(TraversableLike.scala:286)
[T] 	scala.collection.TraversableLike.map$(TraversableLike.scala:279)
[T] 	scala.collection.mutable.AbstractSet.scala$collection$SetLike$$super$map(Set.scala:50)
[T] 	scala.collection.SetLike.map(SetLike.scala:105)
[T] 	scala.collection.SetLike.map$(SetLike.scala:105)
[T] 	scala.collection.mutable.AbstractSet.map(Set.scala:50)
[T] 	sbt.internal.inc.bloop.internal.BloopAnalysisCallback.$anonfun$addProductsAndDeps$1(BloopAnalysisCallback.scala:398)
[T] 	scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:196)
[T] 	scala.collection.TraversableOnce$folder$1.apply(TraversableOnce.scala:194)
[T] 	scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
[T] 	scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:199)
[T] 	scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:192)
[T] 	scala.collection.AbstractTraversable.foldLeft(Traversable.scala:108)
[T] 	sbt.internal.inc.bloop.internal.BloopAnalysisCallback.addProductsAndDeps(BloopAnalysisCallback.scala:356)
[T] 	sbt.internal.inc.bloop.internal.BloopAnalysisCallback.get(BloopAnalysisCallback.scala:298)
[T] 	sbt.internal.inc.bloop.internal.BloopIncremental$.$anonfun$compileIncremental$5(BloopIncremental.scala:119)
[T] 	bloop.task.Task.$anonfun$runAsync$7(Task.scala:265)
[T] 	monix.eval.Task$Map.apply(Task.scala:4604)
[T] 	monix.eval.Task$Map.apply(Task.scala:4600)
[T] 	monix.eval.internal.TaskRunLoop$.startFull(TaskRunLoop.scala:170)
[T] 	monix.eval.internal.TaskRestartCallback.syncOnSuccess(TaskRestartCallback.scala:101)
[T] 	monix.eval.internal.TaskRestartCallback.onSuccess(TaskRestartCallback.scala:74)
[T] 	monix.eval.internal.TaskExecuteOn$AsyncRegister$$anon$1.run(TaskExecuteOn.scala:71)
[T] 	java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
[T] 	java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
[T] 	java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
[T] 	java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
[T] 	java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
[T] 	java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
[E] Failed to compile 'merchant-model'

Answer 1 · 2023-05-18T10:34:27.000Z

Thanks for reporting! This looks like an issue with Zinc it or the way we are using it 🤔

Answer 2 · 2023-05-27T12:41:17.000Z

Hey @tgodzik, after testing with the snapshot as you suggested on Twitter, I got the following error logs:

2023.05.27 22:34:42 ERROR Could not calculate hash for /BC/java.base/java/lang/CharSequence.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BC/java.xml/javax/xml/parsers/DocumentBuilderFactory.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.xml/org/w3c/dom/Node.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /B/java.base/java/io/InputStream.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.xml/javax/xml/xpath/XPathConstants.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.xml/org/w3c/dom/NodeList.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.base/java/io/ByteArrayInputStream.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/xpath/XPathExpression.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/xpath/XPath.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/namespace/QName.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/parsers/DocumentBuilder.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.xml/javax/xml/xpath/XPathFactory.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.xml/org/w3c/dom/Document.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEFG/java.base/java/io/File.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEFG/java.base/java/lang/ClassLoader.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BC/java.base/java/nio/file/Path.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.base/java/nio/file/Paths.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/OffsetDateTime.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /89ABCDEFG/java.base/java/util/HashMap.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEF/java.base/java/time/ZoneOffset.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/LocalDate.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/ZoneId.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /879ABCDEFG/java.base/java/nio/charset/Charset.sig because of null
2023.05.27 22:34:42 ERROR Could not calculate hash for /BCDEFG/java.base/java/nio/charset/StandardCharsets.sig because of null
2023.05.27 22:34:43 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/atomic/AtomicLong.sig because of null
2023.05.27 22:34:43 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/Iterator.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEF/java.base/java/util/stream/Stream.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/stream/BaseStream.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /9AB/java.base/java/lang/Integer.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCD/java.base/java/lang/Thread.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/Future.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /B/java.base/java/util/concurrent/CompletableFuture.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/Callable.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /89ABCDEFG/java.base/java/util/Collections.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /9ABCDEFG/java.base/java/util/concurrent/Executors.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/ExecutorService.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEFG/java.base/java/lang/Iterable.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /BCDEF/java.base/java/util/function/Consumer.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/ZonedDateTime.sig because of null
2023.05.27 22:34:44 ERROR Could not calculate hash for /9ABCDEF/java.base/java/time/LocalTime.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /9ABCDEFG/java.base/java/util/concurrent/ArrayBlockingQueue.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /BCDEF/java.base/java/util/function/IntUnaryOperator.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/concurrent/atomic/AtomicInteger.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /BCDEFG/java.base/java/util/Collection.sig because of null
2023.05.27 22:34:46 ERROR Could not calculate hash for /BCDEF/java.base/java/util/concurrent/atomic/AtomicReference.sig because of null

Hope it helps, If you need any more help just let me know.
Cheers.

Answer 3 · 2023-05-27T18:11:13.000Z

Och, that's curious. It shouldn't reach into the jar, just calculate the hash of the entire jar.

Answer 4 · 2023-05-27T18:36:17.000Z

Also, any idea where those .sig files are coming from? Should they be part of the JDK itself? I wonder if we should just filter them out and only calculate hashes for .class files.

Answer 5 · 2023-05-29T07:51:04.000Z

Hey @tgodzik

It doesn't look like those files are in the JDK runtime FS.
I played around a bit with it and it seems it only contains .class files:

val fs = java.nio.file.FileSystems.getFileSystem(java.net.URI.create("jrt:/"))
java.nio.file.Files.list(fs.getPath("modules", "java.base", "java/lang")).forEach(println(_))

modules/java.base/java/lang/AbstractMethodError.class
modules/java.base/java/lang/AbstractStringBuilder.class
modules/java.base/java/lang/Appendable.class
modules/java.base/java/lang/ApplicationShutdownHooks$1.class
modules/java.base/java/lang/ApplicationShutdownHooks.class
modules/java.base/java/lang/ArithmeticException.class
modules/java.base/java/lang/ArrayIndexOutOfBoundsException.class
modules/java.base/java/lang/ArrayStoreException.class
modules/java.base/java/lang/AssertionError.class
modules/java.base/java/lang/AssertionStatusDirectives.class
modules/java.base/java/lang/AutoCloseable.class
modules/java.base/java/lang/Boolean.class
modules/java.base/java/lang/BootstrapMethodError.class
modules/java.base/java/lang/Byte$ByteCache.class
modules/java.base/java/lang/Byte.class
modules/java.base/java/lang/CharSequence$1CharIterator.class
modules/java.base/java/lang/CharSequence$1CodePointIterator.class
modules/java.base/java/lang/CharSequence.class
modules/java.base/java/lang/Character$CharacterCache.class
....

 java.nio.file.Files.lines(fs.getPath("modules", "java.base", "java/lang/CharSequence.sig")).forEach(println(_))

java.nio.file.NoSuchFileException: /modules/java.base/java/lang/CharSequence.sig
  at java.base/jdk.internal.jrtfs.JrtFileSystem.checkNode(JrtFileSystem.java:486)
  at java.base/jdk.internal.jrtfs.JrtFileSystem.getFileContent(JrtFileSystem.java:245)
  at java.base/jdk.internal.jrtfs.JrtFileSystem.newInputStream(JrtFileSystem.java:334)
  at java.base/jdk.internal.jrtfs.JrtPath.newInputStream(JrtPath.java:637)
  at java.base/jdk.internal.jrtfs.JrtFileSystemProvider.newInputStream(JrtFileSystemProvider.java:325)
  at java.base/java.nio.file.Files.newInputStream(Files.java:160)
  at java.base/java.nio.file.Files.newBufferedReader(Files.java:2922)
  at java.base/java.nio.file.Files.lines(Files.java:4141)
  at java.base/java.nio.file.Files.lines(Files.java:4227)
  ... 59 elided

Answer 6 · 2023-05-30T18:01:22.000Z

the .sig files are supposed to be pickles needed for pipelining (not only probably) but we don't have any pickles for Java files, especially not inside a jar.

I will dig in a bit more.

Answer 7 · 2023-05-30T18:10:06.000Z

Are you using any JDK specific classes or options? How is the modules defined, would you be able to share that?

Looks like the sig files should not pop up there and they are somehow 🤔

Answer 8 · 2023-06-02T00:23:25.000Z

Are you using any JDK specific classes or options? How is the modules defined, would you be able to share that?

Looks like the sig files should not pop up there and they are somehow 🤔

So this is the JDK im using locally:

$ java -version
openjdk version "17.0.7" 2023-04-18
OpenJDK Runtime Environment Homebrew (build 17.0.7+0)
OpenJDK 64-Bit Server VM Homebrew (build 17.0.7+0, mixed mode, sharing)

And this, I think, are all the relevant build definition bits:

val baseSettings =
  Seq(
    scalaVersion := Dependencies.Version.scala,
    organization := "...",
    logLevel     := Level.Info,
    scalacOptions ++= Seq("-release", "11"),
    Compile / scalacOptions ++= Seq(
      "-Wconf:any:warning-verbose",
      "-Wunused:nowarn",
      "-Ymacro-annotations",
      "-Xsource:3"
    ),
    PB.protocVersion := Version.protobufJava, // generate objects at same version as lib
    Compile / scalacOptions ~= {
      options: Seq[String] =>
        options.filterNot(
          Set(
            "-Wunused:imports",
            "-Wunused:params",
            "-Wunused:privates",
            "-Xlint:infer-any"
          )
        )
    },
    javacOptions ++= Seq("-source", "16", "-target", "16"),
    //      scalafmtOnCompile := true,
    testFrameworks ++= Seq(
      TestFramework("zio.test.sbt.ZTestFramework"),
      TestFramework("hedgehog.sbt.Framework")
    ),
    Global / cancelable           := true,
    Global / turbo                := true, // 🤞
    Global / fork                 := true, // https://github.com/sbt/sbt/issues/2274
    Global / onChangedBuildSource := ReloadOnSourceChanges,
    resolvers ++= Seq(
      "Sonatype OSS Snapshots" at "https://oss.sonatype.org/content/repositories/snapshots",
      "Confluent" at "https://packages.confluent.io/maven",
      "Ovotech" at "https://kaluza.jfrog.io/artifactory/maven/"
      //        Resolver.bintrayRepo("ovotech", "maven")
    ),
    // Don't make or publish scaladoc
    Compile / doc / sources                := Nil,
    Compile / packageDoc / publishArtifact := false,
    // parallel and forked tests
    Test / fork               := true,
    Test / testForkedParallel := true,
    Test / parallelExecution  := true,
    // [for the moment] Test / turbo := true,
    Test / concurrentRestrictions := Seq(
      Tags.limit(Tags.CPU, java.lang.Runtime.getRuntime.availableProcessors()),
      Tags.limit(Tags.Test, java.lang.Runtime.getRuntime.availableProcessors())
    ),
    // incremental compilation cache
    pushRemoteCacheTo := Some(MavenCache("sbt-build-cache", file(".sbt-build-cache")))
  )

val compilerPlugins =
  List(
    kindProjector,
    betterMonadicFor
  )

val commonSettings =
  baseSettings ++
    Seq(
      Compile / compile / wartremoverErrors ++= warts,
      libraryDependencies ++= compilerPlugins,
      dependencyOverrides ++= List(
        ...
      )
    )

lazy val `merchant-model` = (project in file("modules/merchant-model"))
  .settings(
    commonSettings,
    libraryDependencies ++=
      List(
        ...
      )
  )
  .dependsOn(
    ...
  )

Answer 9 · 2023-06-30T10:32:35.000Z

I finally got back to the PR with a potential fix for your issue @ktonga <- it should be in the newest snapshot version

Answer 10 · 2023-06-30T10:37:10.000Z

I finally got back to the PR with a potential fix for your issue @ktonga <- it should be in the newest snapshot version

Thank you so much, I'll give it a go. 🙏

Answer 11 · 2023-07-05T01:53:34.000Z

Hey! I was able to update to 1.5.7 and I'm not seeing the error any more, so I think we could close this one as fixed.