Problems with Lucene loading on Mac OS X 10.9.3

Question

Problems with Lucene loading on Mac OS X 10.9.3

Closed this issue 10 years ago · 3 comments

Having some issues with the Lucene step in the pipeline loader. Duplicated on two machines both running 10.9.3 and default java.

INFO: Download complete: /Users/bjhecht/IdeaProjects/wikibrain/./.tmp/download2797307482081029940.tmp/wikidatawiki-20140508-wb_items_per_site.sql.gz
May 18, 2014 1:34:33 PM org.wikibrain.download.DumpFileDownloader downloadLanguageFiles
INFO: 7/7 file(s) downloaded
May 18, 2014 1:34:36 PM org.wikibrain.download.DumpFileDownloader downloadFrom
INFO: 7 files downloaded out of 7 files.
May 18, 2014 1:34:36 PM org.wikibrain.dao.load.PipelineLoader run
INFO: Successfully completed stage download
May 18, 2014 1:34:36 PM org.wikibrain.dao.load.PipelineLoader run
INFO: Beginning stage dumploader
May 18, 2014 1:34:37 PM org.wikibrain.core.lang.Language loadAllLanguages
INFO: loaded 285 languages
May 18, 2014 1:34:41 PM org.wikibrain.conf.Configurator registerProviders
INFO: configurator installed 37 providers for 21 classes
May 18, 2014 1:34:41 PM org.wikibrain.core.cmd.Env
INFO: using baseDir /Users/bjhecht/IdeaProjects/wikibrain/.
May 18, 2014 1:34:41 PM org.wikibrain.core.cmd.Env
INFO: using max vm heapsize of 4087MB
May 18, 2014 1:34:41 PM org.wikibrain.core.cmd.Env
INFO: using languages (SIMPLE,la,lad)
May 18, 2014 1:34:41 PM org.wikibrain.core.cmd.Env
INFO: using maxThreads 4
May 18, 2014 1:34:41 PM org.wikibrain.core.cmd.Env
INFO: using tmpDir ./.tmp
SLF4J: The requested version 1.6.99 by your slf4j binding is not compatible with [1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.9, 1.5.10]
SLF4J: See http://www.slf4j.org/codes.html#version_mismatch for further details.
May 18, 2014 1:34:42 PM org.wikibrain.utils.ParallelForEach$3 run
INFO: processing list element 1 of 3
May 18, 2014 1:34:42 PM org.wikibrain.dao.load.DumpLoader$1 call
INFO: processing file: ./download/la/20140518/lawiki-20140518-pages-articles.xml.bz2
May 18, 2014 1:34:42 PM org.wikibrain.dao.load.DumpLoader$1 call
INFO: processing file: ./download/simple/20140501/simplewiki-20140501-pages-articles.xml.bz2
May 18, 2014 1:34:42 PM org.wikibrain.dao.load.DumpLoader$1 call
INFO: processing file: ./download/lad/20140511/ladwiki-20140511-pages-articles.xml.bz2
Exception in thread "Thread-7" java.lang.NullPointerException
at org.wikibrain.core.dao.sql.FastLoader.insertBatches(FastLoader.java:162)
at org.wikibrain.core.dao.sql.FastLoader.access$000(FastLoader.java:27)
at org.wikibrain.core.dao.sql.FastLoader$1.run(FastLoader.java:69)

at java.lang.Thread.run(Thread.java:695)

"INFO: Beginning stage lucene
May 18, 2014 11:37:10 AM org.wikibrain.core.lang.Language loadAllLanguages
INFO: loaded 285 languages
May 18, 2014 11:37:13 AM org.wikibrain.conf.Configurator registerProviders
INFO: configurator installed 37 providers for 21 classes
May 18, 2014 11:37:13 AM org.wikibrain.core.cmd.Env
INFO: using baseDir /Users/bjhecht/Documents/IDEAProjects/wikibrain/.
May 18, 2014 11:37:13 AM org.wikibrain.core.cmd.Env
INFO: using max vm heapsize of 8175MB
May 18, 2014 11:37:13 AM org.wikibrain.core.cmd.Env
INFO: using languages (SIMPLE,la)
May 18, 2014 11:37:13 AM org.wikibrain.core.cmd.Env
INFO: using maxThreads 8
May 18, 2014 11:37:13 AM org.wikibrain.core.cmd.Env
INFO: using tmpDir ./.tmp
SLF4J: The requested version 1.6.99 by your slf4j binding is not compatible with [1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.9, 1.5.10]
SLF4J: See http://www.slf4j.org/codes.html#version_mismatch for further details.
command failed with exit code 139 :
ABORTING!

Process finished with exit code 139”

Answer 1 · 2014-05-18T23:58:46.000Z

I think these are different bugs. The first one seems like a wikibrain bug,
the second is a JVM bug (which may be triggered by the former wikibrain
bug). @tobyli the first message looks related to your recent commit. Could
you investigate?

Sent from my phone. Please excuse abbreviations and fat fingers.

On May 18, 2014, at 5:44 PM, Brent Hecht notifications@github.com wrote:

Having some issues with the Lucene step in the pipeline loader. Duplicated
on two machines both running 10.9.3 and default java.

INFO: Download complete:
/Users/bjhecht/IdeaProjects/wikibrain/./.tmp/download2797307482081029940.tmp/wikidatawiki-20140508-wb_items_per_site.sql.gz
May 18, 2014 1:34:33 PM org.wikibrain.download.DumpFileDownloader
downloadLanguageFiles
INFO: 7/7 file(s) downloaded
May 18, 2014 1:34:36 PM org.wikibrain.download.DumpFileDownloader
downloadFrom
INFO: 7 files downloaded out of 7 files.
May 18, 2014 1:34:36 PM org.wikibrain.dao.load.PipelineLoader run
INFO: Successfully completed stage download
May 18, 2014 1:34:36 PM org.wikibrain.dao.load.PipelineLoader run
INFO: Beginning stage dumploader
May 18, 2014 1:34:37 PM org.wikibrain.core.lang.Language loadAllLanguages
INFO: loaded 285 languages
May 18, 2014 1:34:41 PM org.wikibrain.conf.Configurator registerProviders
INFO: configurator installed 37 providers for 21 classes
May 18, 2014 1:34:41 PM org.wikibrain.core.cmd.Env
INFO: using baseDir /Users/bjhecht/IdeaProjects/wikibrain/.
May 18, 2014 1:34:41 PM org.wikibrain.core.cmd.Env
INFO: using max vm heapsize of 4087MB
May 18, 2014 1:34:41 PM org.wikibrain.core.cmd.Env
INFO: using languages (SIMPLE,la,lad)
May 18, 2014 1:34:41 PM org.wikibrain.core.cmd.Env
INFO: using maxThreads 4
May 18, 2014 1:34:41 PM org.wikibrain.core.cmd.Env
INFO: using tmpDir ./.tmp
SLF4J: The requested version 1.6.99 by your slf4j binding is not compatible
with [1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.9, 1.5.10]
SLF4J: See http://www.slf4j.org/codes.html#version_mismatch for further
details.
May 18, 2014 1:34:42 PM org.wikibrain.utils.ParallelForEach$3 run
INFO: processing list element 1 of 3
May 18, 2014 1:34:42 PM org.wikibrain.dao.load.DumpLoader$1 call
INFO: processing file:
./download/la/20140518/lawiki-20140518-pages-articles.xml.bz2
May 18, 2014 1:34:42 PM org.wikibrain.dao.load.DumpLoader$1 call
INFO: processing file:
./download/simple/20140501/simplewiki-20140501-pages-articles.xml.bz2
May 18, 2014 1:34:42 PM org.wikibrain.dao.load.DumpLoader$1 call
INFO: processing file:
./download/lad/20140511/ladwiki-20140511-pages-articles.xml.bz2
Exception in thread "Thread-7" java.lang.NullPointerException
at org.wikibrain.core.dao.sql.FastLoader.insertBatches(FastLoader.java:162)
at org.wikibrain.core.dao.sql.FastLoader.access$000(FastLoader.java:27)
at org.wikibrain.core.dao.sql.FastLoader$1.run(FastLoader.java:69)
at java.lang.Thread.run(Thread.java:695)

"INFO: Beginning stage lucene
May 18, 2014 11:37:10 AM org.wikibrain.core.lang.Language loadAllLanguages
INFO: loaded 285 languages
May 18, 2014 11:37:13 AM org.wikibrain.conf.Configurator registerProviders
INFO: configurator installed 37 providers for 21 classes
May 18, 2014 11:37:13 AM org.wikibrain.core.cmd.Env
INFO: using baseDir /Users/bjhecht/Documents/IDEAProjects/wikibrain/.
May 18, 2014 11:37:13 AM org.wikibrain.core.cmd.Env
INFO: using max vm heapsize of 8175MB
May 18, 2014 11:37:13 AM org.wikibrain.core.cmd.Env
INFO: using languages (SIMPLE,la)
May 18, 2014 11:37:13 AM org.wikibrain.core.cmd.Env
INFO: using maxThreads 8
May 18, 2014 11:37:13 AM org.wikibrain.core.cmd.Env
INFO: using tmpDir ./.tmp
SLF4J: The requested version 1.6.99 by your slf4j binding is not compatible
with [1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.9, 1.5.10]
SLF4J: See http://www.slf4j.org/codes.html#version_mismatch for further
details.
command failed with exit code 139 :
ABORTING!

Process finished with exit code 139”

—
Reply to this email directly or view it on
GitHubhttps://github.com//issues/173
.

Answer 2 · 2014-05-19T02:16:40.000Z

I can't replicate this error on my machine. However, I've added a line for checking the null pointer.

Answer 3 · 2014-05-19T02:21:15.000Z

Thanks!

On Sun, May 18, 2014 at 9:16 PM, Toby Li notifications@github.com wrote:

I can't replicate this error on my machine. However, I've added a line for
checking the null pointer.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/173#issuecomment-43461391
.