local_repository: external dependencies not resolved

Question

local_repository: external dependencies not resolved

davido opened this issue 8 years ago · 10 comments

It's related to #2224.

I'm trying a different approach. Instead of linking the Gerrit plugin into Gerrit tree, I'm trying to consume the Gerrit Plugin API through workspace rule local_repository from withing plugin. The difference between standalone build is to be able to consume unpublished plugin API (bleeding edge) from plugin. Without needing to build gerrit locally, and publish Gerrit plugin API to local Maven repository and consume the API over Maven.

So the situation is, I have two independent projects directories, without any symbolic linking between them:

/home/davido/projects/gerrit
/home/davido/projects/gerrit-oauth-provider

For this to work I extended the bazlets with this new gerrit_api_gerrit_local.bzl content to support this scenario: ``:

def gerrit_api_gerrit_local(path):
  native.local_repository(
    name = "gerrit",
    path = path,
  )
  native.bind(
    name = 'gerrit-plugin-api',
    actual = '@gerrit//gerrit-plugin-api:lib')
  native.bind(
    name = 'gerrit-plugin-gwtui',
    actual = '@gerrit//gerrit-plugin-gwtui:gwtui-api-lib')
  native.bind(
    name = 'gerrit-acceptance-framework',
    actual = '@gerrit//gerrit-acceptance-framework:lib')
  native.bind(
    name = 'gerrit-plugin-api-neverlink',
    actual = '@gerrit//gerrit-plugin-api:lib-neverlink')
  native.bind(
    name = 'gerrit-plugin-gwtui-neverlink',
    actual = '@gerrit//gerrit-plugin-gwtui:gwtui-api-lib-neverlink')
  # Neverlink is missing here, use regular library
  native.bind(
    name = 'gerrit-acceptance-framework-neverlink',
    actual = '@gerrit//gerrit-acceptance-framework:lib')

And invoke it from gerrit-oauth-provider plugin's, WORKSPACE file:

[...]
# In Gerrit tree build
load(
    "@com_googlesource_gerrit_bazlets//:gerrit_api_gerrit_local.bzl",
    "gerrit_api_gerrit_local",
)

gerrit_api_gerrit_local("/home/davido/projects/gerrit")

Now, trying to build the plugin and consume the plugin API from local_repository rule is failing to resolve external dependencies, defined in Gerrit plugin API definition:

davido@wizball:~/projects/gerrit-oauth-provider (master *%=)$ bazel build gerrit-oauth-provider --verbose_failures 
ERROR: /home/davido/.cache/bazel/_bazel_davido/f2ec7cf5b8dc484010bd5d650d1db155/external/gerrit/lib/guice/BUILD:12:1: no such package '@guice_library//jar': error loading package 'external': The repository named 'guice_library' could not be resolved and referenced by '@gerrit//lib/guice:guice_library'.
ERROR: Analysis of target '//:gerrit-oauth-provider' failed; build aborted.

Where guice_library//jar is defined in @gerrit//lib/guice as following in gerrit repository:

java_library(
    name = "guice_library",
    data = ["//lib:LICENSE-Apache2.0"],
    visibility = ["//visibility:public"],
    exports = ["@guice_library//jar"],
    runtime_deps = ["aopalliance"],
)

As workaround, I could paste the whole WORKSPACE content from gerrit repository to my plugin's WORKSPACE file, then it works:

$ bazel build gerrit-oauth-provider
INFO: Found 1 target...
INFO: From Building external/gerrit/gerrit-server/libserver.jar (1294 source files, 41 resources):
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
Target //:gerrit-oauth-provider up-to-date:
  bazel-genfiles/gerrit-oauth-provider.jar
INFO: Elapsed time: 25.499s, Critical Path: 20.16s

But why is this needed? I'm consuming gerrit-plugin-api:lib artifact from local_repository rule from gerrit directory and @gerrit//gerrit-plugin-api:lib transitively depends on dozens of external packages in gerrit (!) repository, as can be seen with this query command

$ bazel query --noimplicit_deps "deps(@gerrit//gerrit-plugin-api:lib)" | grep -v "@gerrit" | sed 's#//.*##' | sed 's/@//' | sort | uniq
antlr27
aopalliance
args4j
automaton
auto_value
backward_codecs
bazel_tools
bcprov
blame_cache
commons_codec
commons_collections
commons_compress
commons_dbcp
commons_lang
commons_net
commons_oro
commons_pool
commons_validator
dropwizard_core
grappa
gson
guava
guava_retrying
guice_assistedinject
guice_library
guice_servlet
gwtjsonrpc
gwtorm_client
h2
httpclient
httpcore
icu4j
javaewah
java_runtime
javax_inject
jcl_over_slf4j
jgit
jgit_archive
jgit_servlet
jitescript
joda_convert
joda_time
jsch
jsonevent_layout
json_smart
jsoup
jsr305
juniversalchardet
log4j
log_api
lucene_analyzers_common
lucene_core
lucene_misc
lucene_queryparser
mime4j_core
mime4j_dom
mime_util
mina_core
multibindings
org_antlr
ow2_asm
ow2_asm_analysis
ow2_asm_commons
ow2_asm_tree
ow2_asm_util
pegdown
prolog_runtime
protobuf
servlet_api_3_1
soy
sshd
stringtemplate
tukaani_xz
velocity

Shouldn't Bazel realize, that those are comming from gerrit and be rellocated to it, instead of trying to locate them in my plugin's own WORKSPACE?

Answer 1 · 2017-01-23T01:29:35.000Z

FWIW I also encountered this issue when trying to separate out dependencies auto-generated by generate_workspace from hand-authored code in WORKSPACE by creating a sub-workspace under //third_party/... using local_repository. I ended up just concatenating the generated WORKSPACE rules to the hand-authored code in //WORKSPACE, but that's going to be a pain next time someone needs to add a Maven dependency.

Answer 2 · 2017-01-25T10:19:19.000Z

Is this actually the same as #2224 or is it different? Is it caused by the lack of support for recursive workspaces?

Adding @kchodorow .

Answer 3 · 2017-01-25T10:36:44.000Z

Is this actually the same as #2224 or is it different?

I think it's a different problem. I must admit, I emphasized the problem using Gerrit Code Review project and one of its plugins. But this example is very complex and probably hard to follow. I haven't isolated it to a trivial one, but I could do, if you prefer and upload a trivial reproducer to GH.

So the issue is, let projectA be:

projectA
---> WORKSPACE defines maven_jar(name = "guava", ...)
---> BUILD defines java_library(name= "utilFromProjectA", deps = ["guava"], ...)

and from the projectB this @projectA//utilFromProjectA rule is consumed, by referencing the projectA with local_repositoryrule:

projectB
---> WORKSPACE defines local_repository(name = "projectA", path = <path/to/projectA>)
# Note, that WORKSPACE in projectB doesn't expose @guava//... rule!
---> BUILD defines java_library(name = "foo", deps = ["@projectA//:utilFromProjectA"]

Now building in projectB target foo would fail with something like:

no such package '@guava//jar': error loading package 'external':
The repository named 'guava' could not be resolved and referenced by '@projectB//:foo'.

So the issue is: why Bazel is trying to locate guava package in external of projectB (where the utilFromProjectA target is consumed) and not from the projectA where the @projectA//:utilFromProjectA is actually defined?

This problem makes the consumption of cross repository dependencies feature totally useless, as it actually means, that I need to copy/paste the whole content of WORKSPACE from projectA to projectB. So, in the example above the work around is to define guava in projectB and then it will just work.

Answer 4 · 2017-08-31T09:39:29.000Z

I can confirm the problem on the latest bazel 0.5.4

I have following structure:

<PROJECT_NAME>/WORKSPACE
<PROJECT_NAME>/BUILD
<PROJECT_NAME>/<LIBRARY_A>/BUILD
<PROJECT_NAME>/<SUB_PROJECT_NAME>/WORKSPACE
<PROJECT_NAME>/<SUB_PROJECT_NAME>/BUILD

File <PROJECT_NAME>/<LIBRARY_A>/BUILD is following:

load("@org_pubref_rules_protobuf//go:rules.bzl", "go_proto_library", "GRPC_COMPILE_DEPS")

# https://github.com/pubref/rules_protobuf/tree/master/examples/wkt/go
go_proto_library(
    name = "go_default_library",
    protos = ["<SOME_NAME>.proto"],
    imports = ["external/com_github_google_protobuf/src/"],
    inputs = ["@com_github_google_protobuf//:well_known_protos"],
    deps = [
        "@com_github_golang_protobuf//ptypes/empty:go_default_library",
        "@com_github_golang_protobuf//ptypes/duration:go_default_library",
        "@com_github_golang_protobuf//ptypes/timestamp:go_default_library",
    ] + GRPC_COMPILE_DEPS,
    verbose = 3,
    visibility = ["//visibility:public"],
    with_grpc = True,
)

I able to build <LIBRARY_A> by

cd <PROJECT_NAME>
bazel build //<LIBRARY_A>:go_default_library

FIle <PROJECT_NAME>/<SUB_PROJECT_NAME>/WORKSPACE contains following:

local_repository(
    name = "com_github_<USER_NAME>_<PROJECT_NAME>",
    path = "../",
)

FIle <PROJECT_NAME>/<SUB_PROJECT_NAME>/BUILD is following:


go_library(
    name = "go_default_library",
    srcs = glob(["*.go"], exclude=["*_test.go"]),
    deps = [
        "@com_github_<USER_NAME>_<PROJECT_NAME>//<LIBRARY_A>:go_default_library",
    ],
)

I am NOT able to build <PROJECT_NAME>/<SUB_PROJECT_NAME> with the following error

cd <PROJECT_NAME>/<SUB_PROJECT_NAME>
bazel build //:go_default_library
ERROR: /Users/oleg/go/src/github.com/<USER_NAME>/<PROJECT_NAME>/<SUB_PROJECT_NAME>/BUILD:3:1: error loading package '@com_github_<USER_NAME>_<PROJECT_NAME>//<LIBRARY_A>': Extension file not found. Unable to load package for '@org_pubref_rules_protobuf//go:rules.bzl': The repository could not be resolved and referenced by '//:go_default_library'.
ERROR: Analysis of target '//:go_default_library' failed; build aborted.
INFO: Elapsed time: 0.075s

Seems like dependency to external Bazel-based project ignores the external dependencies from WORKSPACE of that project

Answer 5 · 2019-02-03T09:38:03.000Z

Some news about this issue?

Answer 6 · 2019-03-20T00:49:30.000Z

I think I have another example of this going wrong in a Google project (protobuf):
protocolbuffers/protobuf#5905

Answer 7 · 2019-03-26T04:54:37.000Z

Any updates on this?

Answer 8 · 2020-05-27T11:26:11.000Z

@davido did it get fixed?

Answer 9 · 2020-05-27T12:25:09.000Z

I am not sure, I was just closing my outdated issues. Feel free to write a new issue with exact reproducer.

Answer 10 · 2023-02-01T23:40:39.000Z

Sorry for necro-posting on this; I ran into the same issue, and is wondering if there has been a new issue that references this issue?