No output when using index file
Closed this issue · 6 comments
I'm creating a index file via
`hadoop jar hadoop-lzo-0.4.21.jar com.hadoop.compression.lzo.DistributedLzoIndexer myInput.lzo`
This successfully runs and I get myInput.lzo.index
out. To test this I wrote a trivial map reduce program that reads in a file and then writes it out. When I set the myInput.lzo.index
as the input I get an empty output. When I use the original file myInput.lzo
everything seems to work and the decompressed version of the file is written to dfs.
myInput.lzo.index
is not an input file itself. An lzo index file itself exists to help the lzo input to handle the actual lzo file more efficiently.
Sorry, thanks for the clarification. A little confusing with the documentation saying "Running MR Jobs over Indexed Files: Now run any job, say wordcount, over the new file".
So how does one pass the index
file? I run the above, get the .index
file and then pass the original file through existing code and get a single split.
You don't directly pass the index file. The path of the index file is implicitly assumed to be (lzo file name).index (see https://github.com/twitter/hadoop-lzo/blob/master/src/main/java/com/hadoop/compression/lzo/LzoIndex.java#L41). The lzo file should be large enough to cause multiple splits.
If I decompress the file and run it I get multiple splits, keeping it compressed I always get 1 split. This is happening across hundreds of files. Does this mean I'm not creating the index file properly?
As long as you use the right input format (e.g. LzoTextInputFormat) and the file is big enough, you should see multiple splits. You might want to play with LzoIndex to see if you can find out more of your file.