fzakaria/mvn2nix

mvn requires maven-metadata-central.xml files to resolve dependencies with version ranges

Closed this issue · 8 comments

datakurre commented

I have a project that got a few build errors like

[ERROR] Failed to execute goal on project app: Could not resolve dependencies for project ...: Failed to collect dependencies at org.springframework.boot:spring-boot-starter-oauth2-client:jar:2.2.1.RELEASE -> org.springframework.security:spring-security-oauth2-client:jar:5.2.1.RELEASE -> com.nimbusds:oauth2-oidc-sdk:jar:6.14 -> com.nimbusds:nimbus-jose-jwt:jar:[6.0.1,): No versions available for com.nimbusds:nimbus-jose-jwt:jar:[6.0.1,) within specified range -> [Help 1]

that were resolved by manually adding maven-metadata.xml files into mvn2nix-lock.json as follows:

+    "com.nimbusds:nimbus-jose-jwt:metadata": {
+      "layout": "com/nimbusds/nimbus-jose-jwt/maven-metadata-central.xml",
+      "sha256": "16jvjrr0w61c24zmc5pm9i2iwrx7i77zliivf3wc64qj7fwicxsc",
+      "url": "https://repo.maven.apache.org/maven2/com/nimbusds/nimbus-jose-jwt/maven-metadata.xml"
+    },

So, the required files were actually present in the built repository, but maven were unable to resolve them without those metadata files. Curious.

fzakaria commented

I see.
I think it's because the pom.xml uses a version range.

Maven then needs this additional metadata file to list all the versions to correctly solve for one.

I explicitly filter out that file while we walk the tree

            return Files.walk(localRepository.toPath())
                 .filter(Files::isRegularFile)
                 .filter(f -> !f.endsWith("maven-metadata-local.xml"))

Seems unecessary to have this file always present though.

I see a few solutions:

  1. Try to guide maven developers to the correct approach for packaging in Nix. In this circumstance a patch file or even changing the pom.xml to not use version ranges would be the way to go.
  2. Allow a switch on the command line such that maven-metadata-central.xml are not excluded.

I'm in favor of (1) since using version ranges in Maven is a bit un-idiomatic already.
Using version ranges defeats the ability to make the software reproducible in any meaningful way.
(Maven projects have no lock-file counterpart)

datakurre commented

I agree that it is unnecessary to have this file always.

Also, I guess, that those files will change once new versions are available, breaking the build again due to changes hash.

A working option could be to generate those files with content matching the versions included in the built local repository. But that would be a big change for the current repository builder...

datakurre commented

My workaround was to inline the complete buildMavenRepositoryFromLockFile and inject minimal required metadata files into repository as follows:

let

  linkFarm = name: entries: pkgs.runCommand name { preferLocalBuild = true; allowSubstitutes = false; }
  ''mkdir -p $out
    cd $out
    ${pkgs.lib.concatMapStrings (x: ''
        mkdir -p "$(dirname ${pkgs.lib.escapeShellArg x.name})"
        ln -s ${pkgs.lib.escapeShellArg x.path} ${pkgs.lib.escapeShellArg x.name}
    '') entries}
    cat > $out/net/minidev/json-smart/maven-metadata-central.xml << EOF
<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>net.minidev</groupId>
  <artifactId>json-smart</artifactId>
  <versioning>
    <versions>
      <version>2.3</version>
    </versions>
  </versioning>
</metadata>
EOF
    cat > $out/com/nimbusds/lang-tag/maven-metadata-central.xml << EOF
<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>com.nimbusds</groupId>
  <artifactId>lang-tag</artifactId>
  <versioning>
    <versions>
      <version>1.5</version>
    </versions>
  </versioning>
</metadata>
EOF
    cat > $out/com/nimbusds/nimbus-jose-jwt/maven-metadata-central.xml << EOF
<?xml version="1.0" encoding="UTF-8"?>
<metadata>
  <groupId>com.nimbusds</groupId>
  <artifactId>nimbus-jose-jwt</artifactId>
  <versioning>
    <versions>
      <version>7.8</version>
    </versions>
  </versioning>
</metadata>
EOF
  '';

  buildMavenRepository = { dependencies }:
    let
      dependenciesAsDrv = (pkgs.lib.forEach (builtins.attrValues dependencies) (dependency: {
        drv = builtins.fetchurl {
          url = dependency.url;
          sha256 = dependency.sha256;
        };
        layout = dependency.layout;
      }));
    in linkFarm "mvn2nix-repository" (pkgs.lib.forEach dependenciesAsDrv (dependency: {
      name = dependency.layout;
      path = dependency.drv;
    }));

  buildMavenRepositoryFromLockFile = { file }:
    let
      dependencies = (builtins.fromJSON (builtins.readFile file)).dependencies;
    in buildMavenRepository { inherit dependencies; };

  mavenRepository = buildMavenRepositoryFromLockFile {
    file = ./mvn2nix-lock.json;
  };

in
fzakaria commented

Why not have a patch phase and fixup the pom file to pin to a dependency version >?
I think that would be more "nix" appropriate as well.

datakurre commented

Thanks. I somehow didn't realize that minimal builders also have patchPhase. And didn't know that Maven will look the dependencies from the downloaded pom files. Let's keep this open, and I'll add a new workaround example, once I have managed to try that out (next week).

datakurre commented

It seems that I have not had a single issue with mvn2nix for a year, so it bow deep, close this and re-open if I face similar issues later (and am unable to fix them with patchPhase, which I did not yet try).

oneingan commented

Im agree with solution 1, but i have version ranges coming from deep dependency tree. What do you recommend then? Just fix the third-party dependency in my pom.xml in patchPhase?

edit: I tried to patch pom.xml with a fixed version but dependency tree still search for range:

patch file:

--- a/afirma-simple/pom.xml	2022-01-17 17:39:45.868630119 +0100
+++ b/afirma-simple/pom.xml	2022-01-18 11:35:26.441881769 +0100
@@ -24,6 +24,12 @@
 	<dependencies>
 	
 	    <dependency>
+			<groupId>com.google.guava</groupId>
+  			<artifactId>guava</artifactId>
+  			<version>14.0.1</version>
+		</dependency>
+
+	    <dependency>
 			<groupId>com.dmurph</groupId>
   			<artifactId>JGoogleAnalyticsTracker</artifactId>
   			<version>1.2.0</version>

Still same error:

[ERROR] Failed to execute goal on project afirma-ui-simpleafirma: Could not resolve dependencies for project es.gob.afirma:afirma-ui-simpleafirma:jar:1.6.5: Failed to collect dependencies at com.github.markusbernhardt:proxy-vole:jar:1.0.4 -> org.javadelight:delight-rhino-sandbox:jar:0.0.6 -> org.eclipse.xtend:org.eclipse.xtend.lib:jar:2.8.3 -> org.eclipse.xtext:org.eclipse.xtext.xbase.lib:jar:2.8.3 -> com.google.guava:guava:jar:[10.0.1,14.0.1]: No versions available for com.google.guava:guava:jar:[10.0.1,14.0.1] within specified range -> [Help 1]

chriswarbo commented

I hit this problem with aws-java-sdk-* packages. There are lots of such packages (one per AWS service, e.g. -s3, -dynamodb, -xray, and so on), they have lots of inter-dependencies (using version ranges), and they have very frequent updates, so chasing all the dependencies and patching all the poms manually seems futile.
Instead, I've written a Python script which does the following:

  1. Run mvn2nix to get a package-info.json
  2. Loop though its dependencies array to look for range versions
  3. Replace them with their lower bound (split at commas, take the first element, and remove any brackets/parens)
  4. Run mvn org.apache.maven.plugins:maven-dependency-plugin:2.8:get to fetch that particular version
  5. If it's a POM, and we've not already processed it, run steps 1 to 5 on it.
  6. Collect up the dependencies entries of all the processed POMs and append them to the original package-info.json

I imagine this algorithm will have edge-cases, especially when turning a range into a version, but it's deterministic, and seems to work for my use-cases.

Note that I'm picking lower bounds, under the assumption that upper bounds might not exist (e.g. if someone's specified a range like [1,9999])