Unable to add kafka-connect-http as a Gradle dependency

[mattj65817]

I am unable to add kafka-connect-http as a Gradle dependency as follows:

dependencies {
    implementation(
            [group: "com.github.castorm", name: "kafka-connect-http", version: kafka_connect_http_version]
    )
}

The problem appears to be that kafka-connect-http-0.8.9.pom refers to its parent project using a relative path (../pom.xml) which is not available in the Maven Central repository:

<parent>
    <artifactId>kafka-connect-http-parent</artifactId>
    <groupId>com.github.castorm</groupId>
    <version>0.8.9</version>
    <relativePath>../pom.xml</relativePath>
</parent>

The solution is (probably?) to publish the parent POM along with the rest of the artifacts, assuming there's nothing "secret" in it.
Until/unless this is done, the only way to express this dependency is to include the kafka-connect-http.jar in the project and depend on it as a file, which is not ideal for obvious reasons. It does not appear to be possible to avoid this by excluding transitive dependencies.
The raw build output follows.

$ ./gradlew clean build
> Task :modules:lib-kafka-plugins:compileTestJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':modules:lib-kafka-plugins:compileTestJava'.
> Could not resolve all files for configuration ':modules:lib-kafka-plugins:testCompileClasspath'.
   > Could not resolve com.github.castorm:kafka-connect-http:0.8.9.
     Required by:
         project :modules:lib-kafka-plugins
      > Could not resolve com.github.castorm:kafka-connect-http:0.8.9.
         > Could not parse POM https://repo.maven.apache.org/maven2/com/github/castorm/kafka-connect-http/0.8.9/kafka-connect-http-0.8.9.pom
            > Could not find com.github.castorm:kafka-connect-http-parent:0.8.9.
              Searched in the following locations:
                - https://repo.maven.apache.org/maven2/com/github/castorm/kafka-connect-http-parent/0.8.9/kafka-connect-http-parent-0.8.9.pom
                - https://pkgs.dev.azure.com/mjensen65816/dsds/_packaging/dsds/maven/v1/com/github/castorm/kafka-connect-http-parent/0.8.9/kafka-connect-http-parent-0.8.9.pom

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s
4 actionable tasks: 3 executed, 1 up-to-date

[castorm]

Hi Matt,

Thank you for reporting this.

At some point I introduced maven multi-modules in this project, but only the main artifact is accepted as of today in maven central, and I didn't realize.

I would need to check with sonatype whether they will accept an artifact that is not named the same as the github repo, so the parent pom can be published.

The alternative is kind of unclear to me, because unfortunately there are dependencies across modules that require some build order that I believe wouldn't be possible in a single module.

Best regards.

[castorm]

In the meantime you might want to add github packages repository, all artifacts are published there, so you wouldn't have that issue.

https://docs.github.com/en/packages/guides/configuring-apache-maven-for-use-with-github-packages#authenticating-to-github-packages

[mattj65817]

@castorm thanks, I'll look into that. I'm so far removed from Maven at this point (been working in Gradle for years) that I know nothing about parent POM, Maven best practices, etc. :(

[lobbin]

Tried to get this working, but I think it fails on that kafka-connect-http 0.8.11 depends on kafka-connect-http-parent 0.8.11 where as only kafka-connect-http-parent 0.8.1 seems to be published.

[zpearce]

What would it take to get this fixed? I'm seeing the same thing as @lobbin - the latest kafka-connect-http-parent that has been published to github packages is 0.8.1. So, even after adding gh packages as a maven repository, you still get this error:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileKotlin'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not resolve com.github.castorm:kafka-connect-http:0.8.11.
     Required by:
         project :
      > Could not resolve com.github.castorm:kafka-connect-http:0.8.11.
         > Could not parse POM https://maven.pkg.github.com/castorm/kafka-connect-http/com/github/castorm/kafka-connect-http/0.8.11/kafka-connect-http-0.8.11.pom
            > Could not find com.github.castorm:kafka-connect-http-parent:0.8.11.
              Searched in the following locations:
                - https://maven.pkg.github.com/castorm/kafka-connect-http/com/github/castorm/kafka-connect-http-parent/0.8.11/kafka-connect-http-parent-0.8.11.pom
              If the artifact you are trying to retrieve can be found in the repository but without metadata in 'Maven POM' format, you need to adjust the 'metadataSources { ... }' of the repository declaration.