File is not in the current source path

Question

File is not in the current source path

lukehutch opened this issue 9 years ago · 30 comments

I am trying to get Ceylon working in Google's internal Blaze/Forge build system. I specify the "--src=" commandline parameter to point the Ceylon compiler at the source root, but get the following on launch:

An exception has occurred in the compiler (ceylonc 1.2.0 (A Series Of Unlikely Explanations)).
java.lang.RuntimeException: java/com/google/thirdparty/ceylon/helloworld/run.ceylon is not in the current source path
    at com.redhat.ceylon.compiler.java.tools.LanguageCompiler.getSrcDir(LanguageCompiler.java:732)
    at com.redhat.ceylon.compiler.java.tools.LanguageCompiler.ceylonParse(LanguageCompiler.java:415)
    at com.redhat.ceylon.compiler.java.tools.LanguageCompiler.parse(LanguageCompiler.java:366)
    at com.sun.tools.javac.main.JavaCompiler.parseFiles(JavaCompiler.java:910)
    at com.redhat.ceylon.compiler.java.tools.LanguageCompiler.parseFiles(LanguageCompiler.java:511)
    at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:824)
    at com.redhat.ceylon.compiler.java.tools.LanguageCompiler.compile(LanguageCompiler.java:264)
    at com.redhat.ceylon.compiler.java.launcher.Main.compile(Main.java:650)
    at com.redhat.ceylon.compiler.java.launcher.Main.compile(Main.java:566)
    at com.redhat.ceylon.compiler.java.launcher.Main.compile(Main.java:556)
    at com.redhat.ceylon.compiler.CeylonCompileTool.run(CeylonCompileTool.java:538)
    at com.redhat.ceylon.common.tools.CeylonTool.run(CeylonTool.java:491)
    at com.redhat.ceylon.common.tools.CeylonTool.execute(CeylonTool.java:380)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.redhat.ceylon.launcher.Launcher.runInJava7Checked(Launcher.java:114)
    at com.redhat.ceylon.launcher.Launcher.run(Launcher.java:41)
    at com.redhat.ceylon.launcher.Launcher.run(Launcher.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.redhat.ceylon.launcher.Bootstrap.run(Bootstrap.java:55)
    at com.redhat.ceylon.launcher.Bootstrap.main(Bootstrap.java:32)
ceylon compile: Fatal error: The compiler exited abnormally (4) due to a bug in
the compiler.
Please report it:
[...]

The code in the stacktrace shows:

LanguageCompiler:724-734:

    // FIXME: this function is terrible, possibly refactor it with getPackage?
    private File getSrcDir(File sourceFile) throws IOException {
        Iterable<? extends File> prefixes = ((JavacFileManager)fileManager).getLocation(StandardLocation.SOURCE_PATH);
        File srcDirFile = FileUtil.selectPath(prefixes, sourceFile.getPath());
        if (srcDirFile != null) {
            return srcDirFile;
        } else {
            // This error should have been caught by the tool chain
            throw new RuntimeException(sourceFile.getPath() + " is not in the current source path");
        }
    }

LanguageCompiler:413-415:

            // FIXME: temporary solution
            VirtualFile file = vfs.getFromFile(sourceFile);
            VirtualFile srcDir = vfs.getFromFile(getSrcDir(sourceFile));

I don't know what is triggering the issue (I can build the code on the commandline fine, but when I try building in the Blaze/Forge sandbox, I get this error). However, it seems to indicate some sort of compiler inconsistency, and the code looks like it is going to be rewritten at some point. Any ideas as to what might be triggering this, and/or how to fix it?

Answer 1 · 2015-11-04T00:32:17.000Z

@lukehutch what is the whole command line you're using to invoke ceylon compile? Thanks.

Answer 2 · 2015-11-04T00:48:19.000Z

The whole command is:

${CEYLON_BIN} compile --cacherep=${CACHE_DIR} --src=java com.google.thirdparty.ceylon.helloworld

If I run the same command directly from the commandline, it succeeds. However, when running inside the Google build system (which is sandboxed), it fails.

The files copied into the sandbox by my build rules include:

./java
./java/com
./java/com/google
./java/com/google/thirdparty
./java/com/google/thirdparty/ceylon
./java/com/google/thirdparty/ceylon/helloworld
./java/com/google/thirdparty/ceylon/helloworld/run.ceylon
./java/com/google/thirdparty/ceylon/helloworld/package.ceylon
./java/com/google/thirdparty/ceylon/helloworld/module.ceylon
./blaze-out
./blaze-out/gcc-4.X.Y-crosstool-v18-hybrid-grtev4-k8-opt
./blaze-out/gcc-4.X.Y-crosstool-v18-hybrid-grtev4-k8-opt/genfiles
./blaze-out/gcc-4.X.Y-crosstool-v18-hybrid-grtev4-k8-opt/genfiles/java
./blaze-out/gcc-4.X.Y-crosstool-v18-hybrid-grtev4-k8-opt/genfiles/java/com
./blaze-out/gcc-4.X.Y-crosstool-v18-hybrid-grtev4-k8-opt/genfiles/java/com/google
./blaze-out/gcc-4.X.Y-crosstool-v18-hybrid-grtev4-k8-opt/genfiles/java/com/google/thirdparty
./blaze-out/gcc-4.X.Y-crosstool-v18-hybrid-grtev4-k8-opt/genfiles/java/com/google/thirdparty/ceylon
./blaze-out/gcc-4.X.Y-crosstool-v18-hybrid-grtev4-k8-opt/genfiles/java/com/google/thirdparty/ceylon/helloworld
./blaze-out/host
./blaze-out/host/bin
./blaze-out/host/bin/third_party
./blaze-out/host/bin/third_party/java
./blaze-out/host/bin/third_party/java/ceylon
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/jdk
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/jdk/jdk-64
[...]
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/ceylon
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/ceylon/v1_2_0
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/ceylon/v1_2_0/templates
[...]
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/ceylon/v1_2_0/repo
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/ceylon/v1_2_0/repo/org
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/ceylon/v1_2_0/repo/org/tautua
[...]
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/ceylon/v1_2_0/lib
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/ceylon/v1_2_0/lib/ceylon-bootstrap.jar
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/ceylon/ceylon.jar
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/ceylon/ceylon
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/_cpp_runtimes
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/_cpp_runtimes/libstdc++.so.6
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/_cpp_runtimes/libssp.so.0
./blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/_cpp_runtimes/libgcc_s.so.1
./blaze-out/host/bin/third_party/java/ceylon/ceylon.jar
./blaze-out/host/bin/third_party/java/ceylon/ceylon

The current directory ("./" at the beginning of all these paths) is the "google3" base directory for the build.

The "ceylon.runfiles" directory is the set of dependencies of the build rule (i.e. the Ceylon compiler and the JDK are included in this directory, because they are dependencies of "helloworld").

Answer 3 · 2015-11-04T00:55:04.000Z

This doesn't look right:

--src=java com.google.thirdparty.ceylon.helloworld

--src is supposed to specify the source folder. In this case it should be --src=java I think.

(That's no excuse for the terrible error the compiler is throwing, of course.)

Answer 4 · 2015-11-04T00:58:38.000Z

I don't understand your previous comment -- I do have --src=java, followed by the package name to compile, com.google.thirdparty.ceylon.helloworld.

I had to specify the source directory and the base package to compile, rather than the source directory and the full path of files to compile, because the Google build tree requires java-like things to be in the subdirectory "java", but doesn't set that subdirectory to be the current directory before building. (Ugh.)

Answer 5 · 2015-11-04T01:00:51.000Z

Sorry, one other thing that may be a confounding factor: ${CEYLON_BIN} is an autogenerated wrapper created by the Google build system that launches ceylon-bootstrap.jar's main class. The working compile command (directly from the commandline) ends up launching the following from the script:

exec /google/src/cloud/lukehutch/ceylon/google3/blaze-bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/jdk/jdk-64/bin/java -classpath blaze-bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/ceylon/v1_2_0/lib/ceylon-bootstrap.jar com.redhat.ceylon.launcher.Bootstrap compile --src=java com.google.thirdparty.ceylon.helloworld --out=/tmp/modules

The compile command that fails (running on Forge, the remote build sandbox) is:

exec /build/work/6cc406ca63bf0fc3e72cdabb94e43d17/google3/blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/jdk/jdk-64/bin/java -classpath blaze-out/host/bin/third_party/java/ceylon/ceylon.runfiles/google3/third_party/java/ceylon/v1_2_0/lib/ceylon-bootstrap.jar -client -XX:ErrorFile=/dev/stderr com.redhat.ceylon.launcher.Bootstrap compile --cacherep=blaze-out/gcc-4.X.Y-crosstool-v18-hybrid-grtev4-k8-opt/genfiles/java/com/google/thirdparty/ceylon/helloworld/cache --src=java com.google.thirdparty.ceylon.helloworld

The main difference I see is that on Forge, the additional switches -client -XX:ErrorFile=/dev/stderr are present.

Answer 6 · 2015-11-04T01:13:52.000Z

I don't understand your previous comment -- I do have --src=java, followed by the package name to compile, com.google.thirdparty.ceylon.helloworld.

Ugh. Sorry. I simply misread the line I quoted. Pfff. I blame lack of sleep.

Answer 7 · 2015-11-04T08:56:44.000Z

@lukehutch could you add the option --verbose to the command line and send us the output? (Here if it's not to big or otherwise as a Gist or a pastebin, whatever you like)

Answer 8 · 2015-11-04T08:59:15.000Z

I'm at least not able to reproduce this locally, but I don't know what kind of environment the build system imposes.
To work around the problem you might trying to pass an absolute path to the --src argument or otherwise add the --cwd option with an absolute path to the folder where the project lives.

Answer 9 · 2015-11-05T22:08:10.000Z

@quintesse here are the results of compiling with --verbose:

Running locally (succeeds):
http://pastebin.com/YKWSJu3q

Running in the Forge build system sandbox (fails):
http://pastebin.com/QbmyTX3X

I haven't figured out how to get this working yet (I'm checking on the likelihood that this is CWD-related).

The problem is almost certainly specific to Forge (therefore not a Ceylon issue), but maybe filing the bug will be useful anyway by drawing attention to the comments on the Ceylon code which indicates this is due for an overhaul :-)

Answer 10 · 2015-11-05T22:48:36.000Z

@lukehutch Well it's never OK for our tools to blow up with a stack trace like that, so whatever is going on we need to figure out why and how it's happening so we can do better on the error reporting front.

Answer 11 · 2015-11-05T23:27:57.000Z

Looking into this some more, and in the Forge case:

pwd before the compile gives /build/work/26f626cc9c3b555dab4fcd5686b367c0/google3
The compiler option --src is set to java
The requested package to compile is com.google.thirdparty.ceylon.helloworld
There is in fact a file ./java/com/google/thirdparty/ceylon/helloworld/run.ceylon in the Forge build tree (i.e. it is at /build/work/26f626cc9c3b555dab4fcd5686b367c0/google3/java/com/google/thirdparty/ceylon/helloworld/run.ceylon)
And yet the error message is java/com/google/thirdparty/ceylon/helloworld/run.ceylon is not in the current source path.

Shouldn't it be looking for com/google/thirdparty/ceylon/helloworld/run.ceylon within the current source path (which was specified to be java), rather than java/com/google/thirdparty/ceylon/helloworld/run.ceylon? Or is it simply reporting the full path of the file it's looking for, relative to the current directory, and not relative to the source root?

Answer 12 · 2015-11-05T23:39:18.000Z

Well it's never OK for our tools to blow up with a stack trace like that

Only when used in a supported way, which this clearly isn't.

Which doesn't mean we don't want to fix it :)

Answer 13 · 2015-11-05T23:46:52.000Z

@lukehutch the successful log has in line 2 a list of arguments that is missing from the failed log, did that perhaps happen when you copy&pasted? (It would expect it to be there)

Answer 14 · 2015-11-05T23:47:53.000Z

I don't see how this is not "supported", all I'm doing is specifying my own src/out/cwd locations etc... (though the build environment is, admittedly, weird! Which is a good way to exercise code to find bugs...)

Answer 15 · 2015-11-05T23:50:40.000Z

@quintesse no, I definitely don't see that second line in the output of the Forge run, even if I re-run the code. Is it supposed to be written to stderr or stdout? (I think I have the output of both in what I pasted to pastebin...)

Answer 16 · 2015-11-05T23:54:44.000Z

@lukehutch because the supported way is using the shell script ;)
There's a bunch of environment variables in there that could affect the working of the code, as well as the fact that the build environment might be lacking in other details that a normal user account would have.

But like I said we're certainly interested in making it work.

The second line should be there yes, it's something that printed out very early in the process when verbose mode is enabled.

Answer 17 · 2015-11-05T23:58:16.000Z

Btw, did you try already with passing absolute paths to the --src command?

And secondly if that doesn't work add the --cmd?

Answer 18 · 2015-11-06T00:15:01.000Z

Or is it simply reporting the full path of the file it's looking for, relative to the current directory, and not relative to the source root?

Basically yes, it needs that part to be able to determine where it came from (files coming from a --resource path for example are treated differently than files coming from a --source path)

Answer 19 · 2015-11-06T01:50:59.000Z

I can't use absolute paths easily, because Forge puts a random hex key into the path for every new build, but I did --src=pwd/java --cwd=pwd``, and it didn't fix anything.

The only environment variable setting I see in the bash wrapper bin/ceylon that could conceivably affect compilation is:

 PREPEND_JAVA_OPTS="$PREPEND_JAVA_OPTS -Dcom.redhat.ceylon.common.tool.progname=$(basename $PRG)"

However, directly calling ceylon-bootstrap.jar with the main class specified works for local compilation, just not on Forge, so I don't think that's the issue.

It's possible there's a difference in the JVM on Forge, but I imagine Ceylon works on a range of different JVM versions.

Answer 20 · 2015-11-06T22:25:14.000Z

I got Ceylon to build, and if I print out the value of StandardLocation.SOURCE_PATH at the beginning of LanguageCompiler.getSrcDir(), I get as output (without the quotes): "SOURCE_PATH". This does not appear correct, I'm assuming it should be a path...

Answer 21 · 2015-11-06T22:27:34.000Z

I got Ceylon to build

What was the problem, FTR?

Answer 22 · 2015-11-06T22:39:53.000Z

I'm not sure -- I tried multiple incantations, using both the ant that came with Ubuntu, and a manually-downloaded ant. Ultimately I just blew away the ceylon-compiler directory and checked it out again from git, and it worked.

One thing that is not clearly explained in the docs is that after compiling ceylon-dist, you should check out ceylon-compiler into the ceylon-dist directory. (Or does it not need to be there?)

(I'm curious as to why ceylon-dist doesn't download and compile ceylon-compiler too..)

Answer 23 · 2015-11-06T22:45:58.000Z

What instructions are you following? These are the right ones:

https://github.com/ceylon/ceylon-dist#building-the-distribution

FTR, ant setup clones ceylon-compiler and everything else you need.

Answer 24 · 2015-11-06T23:09:49.000Z

Sorry, you're right. Those are the instructions I followed. The first time I followed them, I saw the other clone operations happen (ceylon-common, ceylon-model etc.), but looked for and must have missed spotting ceylon-compiler, so I cloned it myself. Apologies for the distraction from the core issue.

Answer 25 · 2015-11-06T23:56:17.000Z

I figured out what's going wrong here. The "SOURCE_PATH" thing was a red herring; ((JavacFileManager)fileManager).getLocation(StandardLocation.SOURCE_PATH) does the right thing when StandardLocation.SOURCE_PATH == "SOURCE_PATH".

The issue is that FileUtil.absoluteFile() calls file.getCanonicalFile() (and then if that fails, it calls file.getAbsoluteFile()). On Forge, file.getCanonicalFile() incorrectly turns /build/work/014abc80205c11d1f14689caa24071da/google3/java/com/google/thirdparty/ceylon/helloworld/run.ceylon into /build/cas/de5/de5422c502ba2107d2a138b888a8961a for some reason.

So this does appear to be a Google-specific bug, presumably to do with the JDK not being able to properly canonicalize paths across Google's distributed filesystem.

Sorry for the wild goose chase; I don't think there's anything you need to fix on your end. I'll open a bug in Google's bug tracker, and will close this unless I find some other issue. Thanks for the responsiveness anyway!

Answer 26 · 2015-11-07T00:00:52.000Z

OK, thanks for digging!

Answer 27 · 2015-11-07T00:23:06.000Z

Ha, I'd just added an extra log line to the Launcher exactly for that purpose, to see what getAbsoluteFile(".") would look like because the only think I could imagine going wrong was that method returning a wrong value somehow.

Keep us posted on the bug if you will, would be interesting to know what caused it!

Answer 28 · 2015-11-07T01:13:50.000Z

Ah, I heard back from the Google JDK team as to what's going on. Forge is a content-addressable storage, and every file is really a symlink to something that looks like /build/cas/md5sum. The advice I got was "What we usually do is either a) to remove the canonicalization (usually you don't need it), or b) to do it by following symlinks until the one immediately before the one in /build/cas".

So actually, this does raise a potential issue with the Ceylon compiler using getCanonicalPath: if getCanonicalPath causes links to be de-referenced, it is possible that the linked files will no longer have the right paths relative to the source directory. This is exactly what was happening in this case, but taken to an extreme, where all linked files are in a flat structure, keyed by MD5, but it's possible to imagine other cases where using symlinks could cause the Ceylon compiler to do the wrong thing.

I tried "a)" (specifically, replacing the call to file.getCanonicalFile() with file.getAbsoluteFile()), but then the Ceylon compiler bails with:

An exception has occurred in the compiler (ceylonc 1.2.0 (A Series Of Unlikely Explanations)).
java.lang.StringIndexOutOfBoundsException: String index out of range: -2
    at java.lang.String.substring(String.java:1918)
    at com.redhat.ceylon.compiler.typechecker.io.impl.Helper.computeRelativePath(Helper.java:44)
    at com.redhat.ceylon.compiler.typechecker.context.PhasedUnit.<init>(PhasedUnit.java:105)
    at com.redhat.ceylon.compiler.java.tools.CeylonPhasedUnit.<init>(CeylonPhasedUnit.java:43)
    at com.redhat.ceylon.compiler.java.tools.LanguageCompiler.ceylonParse(LanguageCompiler.java:485)
    at com.redhat.ceylon.compiler.java.tools.LanguageCompiler.parse(LanguageCompiler.java:366)
    at com.sun.tools.javac.main.JavaCompiler.parseFiles(JavaCompiler.java:910)
    at com.redhat.ceylon.compiler.java.tools.LanguageCompiler.parseFiles(LanguageCompiler.java:511)
    at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:826)
    at com.redhat.ceylon.compiler.java.tools.LanguageCompiler.compile(LanguageCompiler.java:264)
    at com.redhat.ceylon.compiler.java.launcher.Main.compile(Main.java:650)
    at com.redhat.ceylon.compiler.java.launcher.Main.compile(Main.java:566)
    at com.redhat.ceylon.compiler.java.launcher.Main.compile(Main.java:556)
    at com.redhat.ceylon.compiler.CeylonCompileTool.run(CeylonCompileTool.java:538)
    at com.redhat.ceylon.common.tools.CeylonTool.run(CeylonTool.java:491)
    at com.redhat.ceylon.common.tools.CeylonTool.execute(CeylonTool.java:380)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.redhat.ceylon.launcher.Launcher.runInJava7Checked(Launcher.java:114)
    at com.redhat.ceylon.launcher.Launcher.run(Launcher.java:41)
    at com.redhat.ceylon.launcher.Launcher.run(Launcher.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.redhat.ceylon.launcher.Bootstrap.run(Bootstrap.java:55)
    at com.redhat.ceylon.launcher.Bootstrap.main(Bootstrap.java:32)
[773ms] Program end

Answer 29 · 2015-11-07T01:25:17.000Z

As an example of what could go wrong, imagine a project root at MyProject/source that has a symlink from MyProject/source/com/xyz/package to MyOtherProject/source/com/xyz/package. Calling getCanonicalPath() will cause the source links to be based in MyOtherProject, while the source root is still in MyProject (because the source root occurs above the symlink).

I think the right thing to do is to never follow symlinks when absolutizing paths. You still need to handle ".." etc.

The "One Right Way" to handle paths in JDK >= 1.7 is using Path, not File. I recently switched a bunch of my own code from File to Path, and it's significantly nicer, more robust, more portable and safer to do things the Path way.

In this case, I think what you want is Path.toRealPath(LinkOption.NOFOLLOW_LINKS).

(Re-opening, since I think this is worth looking at further on the Ceylon end of things...)

Answer 30 · 2015-11-07T03:00:02.000Z

Opened new issue #2419 for Luke's suggestion and closing this.