Failed to merge file
Closed this issue · 2 comments
Hi,
I'm running Riak 2.0, with the latest bitcask backend, compiled from github code (as of today).
> code:which(riak_kv_bitcask_backend).
"/usr/lib64/riak/lib/basho-patches/riak_kv_bitcask_backend.beam"
I'm still seeing a lot of "Failed to merge" errors. My merging config is:
%% 4 days
{ expiry_secs, 345600 },
{io_mode, erlang},
%% 4 GB file size limit
{max_file_size, 16#100000000},
{merge_window, always},
{log_needs_merge, true},
%% we wait until the data file is 95% full of dead keys before merging
{frag_merge_trigger, 95},
%% we wait until dead bytes is 95% of file size (4GB), so 4294967296 * 95/100 = 4080218931
{dead_bytes_merge_trigger, 4080218931 },
%% thresholds: same
{ frag_threshold, 95 },
{ dead_bytes_threshold, 4080218931 },
%% merge expired keys only once a day
{expiry_grace_time, 86400}
And here is an error:
Failed to merge {["/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/95.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/93.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/92.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/91.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/90.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/138.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/130.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/129.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/128.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/127.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/126.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/124.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/123.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/122.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/121.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/120.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/119.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/117.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/116.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/115.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/114.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/113.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/112.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/111.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/109.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/107.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/105.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/103.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/101.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/99.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/97.bitcask.data"],["/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/95.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/93.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/92.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/91.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/90.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/138.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/130.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/129.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/128.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/127.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/126.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/124.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/123.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/122.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/121.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/120.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/119.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/117.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/116.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/115.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/114.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/113.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/112.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/111.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/109.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/107.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/105.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/103.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/101.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/99.bitcask.data","/var/lib/riak/bitcask_mult_events/753586781748746817198774991869333432010090217472/97.bitcask.data"]}: {generic_failure,error,function_clause,[{riak_kv_bitcask_backend,key_transform_to_1,[{tombstone,<<2,0,6,101,118,101,110,116,115,49,52,49,49,49,54,49,49,48,51,58,52,58,87,69,66,58,97,112,112,58,55>>}],[{file,"src/riak_kv_bitcask_backend.erl"},{line,99}]},{bitcask,'-expiry_merge/4-fun-0-',7,[{file,"src/bitcask.erl"},{line,1912}]},{bitcask_fileops,fold_hintfile_loop,5,[{file,"src/bitcask_fileops.erl"},{line,660}]},{bitcask_fileops,fold_file_loop,8,[{file,"src/bitcask_fileops.erl"},{line,720}]},{bitcask_fileops,fold_hintfile,3,[{file,"src/bitcask_fileops.erl"},{line,624}]},{bitcask,expiry_merge,4,[{file,"src/bitcask.erl"},{line,1915}]},{bitcask,merge1,4,[{file,"src/bitcask.erl"},{line,686}]},{bitcask,merge,3,[{file,"src/bitcask.erl"},{line,566}]}]}
@dams the fix for this is not on the develop branch of Bitcask, but on the 1.7 branch of Bitcask. It will be merged to develop once Riak 2.0.1 is out. You can remove that riak_kv_bitcask.beam from your basho-patches. To get it, make sure Bitcask is on the 1.7 branch and check with git log to see the fix checked in. There are two patches for riak_kv_bitcask.beam out there unfortunately, and I bet you only have the first attempt. I'm closing the ticket for now. Check on IRC or re-open if that doesn't solve the problem for you.
I was going to close this issue myself, realizing my mistake. Sorry for the noise !