shilad/wikibrain

compilation of wikAPIdia in wp-java.sh generates lib folders in each wikAPIdia sub-directory instead of home directory

Closed this issue · 15 comments

derianders commented

This issue results in a ClassNotFound exception during execution unless one of the lib folders is moved to the home directory, where the classpath says it should be.

shilad commented

wp-java.sh should detect child lib/ directories. Is yours not up-to-date? You may need to run the wp-java installer again. In an ideal world, I suppose this would happen automatically...

tobyli commented

@shilad I got exactly the same problem as well when I tried to fresh install.

shilad commented

What OS are you two running?

tobyli commented

@shilad Ubuntu Linux 12.04

derianders commented

I'm also using Ubuntu 12.04.

As of the most recent wp-java.sh in master, the classpath has the /lib
directory in the base directory:

Destination of compiled jars and dependencies

WP_LIB="${WP_LIB:-${WP_DIR}/lib}"

Specify default classpath. This will be updated later, and CLASSPATH

will be prepended to it.
WP_CLASSPATH="${WP_CLASSPATH:-${WP_LIB}/*}"

This seems reasonable to me though, because I don't know why the /lib
folders with the dependencies would need to exist in every child directory.
Also, the gitignore for the home directory contains /lib, while the
gitignores for the child directories don't. I think the problem is that
when they're being generated, they're being put in the child directories
instead of the home directory where they should be put. But I don't know
why that's happening for me and Toby, especially if it's working fine for
you.

Also, I was getting runtime exceptions during the execution of the
dumploader stage of the pipeline, because the program was trying to execute
the last semicolon of each of the SQL scripts in the resources/db folder
as a separate SQL statement. My workaround was just to delete the last
semicolon in each SQL file. Did either of you guys run into this issue, or
was it just me?

On Sat, Jan 25, 2014 at 1:27 AM, Toby Li notifications@github.com wrote:

@shilad https://github.com/shilad Ubuntu Linux 12.04

Reply to this email directly or view it on GitHubhttps://github.com//issues/147#issuecomment-33283188
.

tobyli commented

@derianders
I don't have the latter issue you mentioned

shilad commented

You'll notice that later down in the script this set of commands that tries to add on the runtime classpath.

# Update classpath with latest version of jars, etc.
for srcdir in $(find "${WP_DIR}" -type d -print | grep 'target/classes$'); do
    WP_CLASSPATH="${srcdir}:${WP_CLASSPATH}"
done
if [ -n "${CLASSPATH}" ]; then
    WP_CLASSPATH="${CLASSPATH}:${WP_CLASSPATH}"
fi

What's your exact error message? Also, what does the "executing...." echo statement from the bash script print out? Mine says:

$ ./wp-java.sh org.wikapidia.utils.ResourceInstaller
... a bunch of unimportant stuff, then...
executing "java" -cp "./wikAPIdia-wikidata/target/classes:./wikAPIdia-utils/target/classes:./wikAPIdia-sr/target/classes:./wikAPIdia-phrases/target/classes:./wikAPIdia-parser/target/classes:./wikAPIdia-matrix/target/classes:./wikAPIdia-mapper/target/classes:./wikAPIdia-lucene/target/classes:./wikAPIdia-loader/target/classes:./wikAPIdia-integration-tests/target/classes:./wikAPIdia-download/target/classes:./wikAPIdia-core/target/classes:./wikAPIdia-cookbook/target/classes:./lib/*" -server -Xmx2048m -server -ea org.wikapidia.utils.ResourceInstaller
backing up ./wp-java.sh to ./wp-java.sh.backup
shilad commented

@tobyli Are you using postgres? I introduced a bug that postgres didn't like consistent with the semi-colon problem you're describing at some point, but an update within the last week fixed it.

derianders commented

My "executing" statement says the exact same thing as yours, with the
classpath argument containing all the child directories' target classes as
well as the /lib folder in the current directory. Then I get the following
exception:

      Exception in thread "main" java.lang.NoClassDefFoundError:

org/apache/commons/cli/ParseException

because the apache commons library is in the /lib folder, which the
classpath says should be in the home directory but instead exists in each
of the child wikAPIdia directories (so typing "git status" shows
wikAPIdia-core/lib, wikAPIdia-loader/lib, etc. as untracked files, but
there is no /lib folder in the wikAPIdia directory at the first level).
Moving a lib folder from one of these child directories to the current
(wikAPIdia, from where I ran the script) directory fixes the issue.

To clarify, the problem wasn't that the .../target/classes folders weren't
being added to the classpath properly, it's just that the /lib folder
wasn't being generated in the correct place (the wikAPIdia folder).

On Sat, Jan 25, 2014 at 1:42 PM, Shilad Sen notifications@github.comwrote:

@tobyli https://github.com/tobyli Are you using postgres? I introduced
a bug that postgres didn't like consistent with what you're describing at
some point, but an update within the last week fixed it.

Reply to this email directly or view it on GitHubhttps://github.com//issues/147#issuecomment-33297937
.

shilad commented

Thanks for pointing me in the right direction, @tobyli and @derianders .

The problem was the destination directory for maven dependency jars was relative, not absolute so the libs were placed in each subdirectory instead of the one true lib/. I've added a bit of code from SO to change relative paths into absolute paths, and things seem to work better now.

I must have introduced this bug when I went to the new parent pom structure and not realized it because of the leftover lib/ directory.

Let me know if this works for you!

tobyli commented

@shilad
That makes sense...I realized that now everything starts from wikAPIdia-all instead of wikAPIdia/wikAPIdia-parent. This has always been a confusing part when programming with the wikAPIdia library :)

tobyli commented

That problem has been around for a while...my solution is to move all
library (JRE files) from the directiory of each module to the "lib" folder
in the home folder of wikapidia

On Fri, Mar 14, 2014 at 11:52 AM, gfenza notifications@github.com wrote:

Hi, sorry I downloaded the project today and I am having the same problem
described by dariander:

Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/commons/cli/ParseException

Is it solved?

thanks in advance

Reply to this email directly or view it on GitHubhttps://github.com//issues/147#issuecomment-37669948
.

Jiajun "Toby" Li
B.S. Student in Computer Science
Department of Computer Science and Engineering
University of Minnesota
Minneapolis, MN 55455

shilad commented

@tobyli Are you still having to move jars into place? I fixed this bug, so things should work now.

tobyli commented

That was the case when i installed wikAPIdia for Aaron (an undergrad at the
U) on a Ubuntu PC yesterday.

-Toby

On Sat, Mar 15, 2014 at 12:33 AM, Shilad Sen notifications@github.comwrote:

@tobyli https://github.com/tobyli Are you still having to move jars
into place? I fixed this bug, so things should work now.

Reply to this email directly or view it on GitHubhttps://github.com//issues/147#issuecomment-37717966
.

Jiajun "Toby" Li
B.S. Student in Computer Science
Department of Computer Science and Engineering
University of Minnesota
Minneapolis, MN 55455

gfenza commented

Hi, I am using windows and I am using win-bash to compile with wp-java.sh script.
In order to avoid NoClassDefFound problem I manually copied all jars in a lib folder under wikapidia root. In this way I don't have the problem of NoClassDefFound but I am having another problem which description is included in a new issue posted here: #153