twitter/hadoop-lzo

Problem with hadoop v1.0.3

Closed this issue · 3 comments

ivihrov commented

Hi all,

I've followed the all steps from README.md
But now I have the next error
/usr/local/hadoop/lib $ hadoop jar hadoop-lzo.jar com.hadoop.compression.lzo.LzoIndexer /lzoInput

13/06/18 11:42:34 ERROR lzo.GPLNativeCodeLoader: Could not load native gpl library
java.lang.UnsatisfiedLinkError: /usr/local/hadoop-1.0.3/lib/native/Linux-amd64-64/libgplcompression.so: /usr/local/hadoop-1.0.3/lib/native/Linux-amd64-64/libgplcompression.so: undefined symbol: lzo1x_999_compress_level
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1807)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1732)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
....

Enviroment
hadoop 1.0.3
java version "1.6.0_32"
Java(TM) SE Runtime Environment (build 1.6.0_32-b05)
Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode)

Thanks for any ideas.

sjlee commented

I believe the location of your lzo dynamic library should be included in JAVA_LIBRARY_PATH as well. Could you check your value of JAVA_LIBRARY_PATH inside the hadoop script and see if it includes the location for your lzo dynamic library location?

ivihrov commented

Thanks sjlee. Yes, JAVA_LIBRARY_PATH includes lzo dynamic library.

Problem was in compling stage -

hduser@dm7 ~/test/twitter3/hadoop-lzo/src/native $ ldd -r .libs/libgplcompression.so
linux-vdso.so.1 => (0x00007fffc19ff000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f4e488e6000)
libc.so.6 => /lib64/libc.so.6 (0x00007f4e4855a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4e48cfe000)
undefined symbol: lzo1x_999_compress_level (.libs/libgplcompression.so)

Then in Makefile I manually set
CPPFLAGS = -I/usr/local/lzo-2.06/include/
LDFLAGS = -L/usr/local/lib/
LIBS = -ldl -llzo2

and delete -m$(JVM_DATA_MODEL) in
AM_CFLAGS = -g -Wall -fPIC -O2 -m$(JVM_DATA_MODEL)
because it causes error in compiling

hduser@dm7 ~/test/twitter/hadoop-lzo/src/native $ ldd -r .libs/libgplcompression.so
linux-vdso.so.1 => (0x00007fff621e3000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fb06aa56000)
liblzo2.so.2 => /usr/local/lzo-2.06/lib/liblzo2.so.2 (0x00007fb06a831000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb06a4a5000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb06ae6e000)

and now it's work.

hduser@dm7 ~ $ hadoop jar /usr/local/hadoop/lib/hadoop-lzo.jar com.hadoop.compression.lzo.LzoIndexer /lzoInput
13/06/20 12:05:31 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
13/06/20 12:05:31 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev c774424]
13/06/20 12:05:32 INFO lzo.LzoIndexer: LZO Indexing directory /lzoInput...
13/06/20 12:05:32 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file hdfs://dm9.adriver.x:54310/lzoInput/testhist3.lzo, size 2.26 GB...
13/06/20 12:06:01 INFO lzo.LzoIndexer: Completed LZO Indexing in 29.82 seconds (77.46 MB/s). Index size is 359.88 KB.

But now I have another problem. Then I use MapReduce for this file hdfs://dm9.adriver.x:54310/lzoInput/testhist3.lzo (2.26 GB) (it has hdfs://dm9.adriver.x:54310/lzoInput/testhist3.lzo.index)
Job is not splitted and I have only one Map task.

My Job code is:

job.setInputFormatClass(LzoTextInputFormat.class);
job.setMapperClass(myHBaseKVMapper.class);

Do you have any ideas why it happens?
Thanks.

ivihrov commented

I've solved problem - I've forgotten to delete old version of my Job jar

Thanks