eBay/HomeStore

index long running fails

Closed this issue · 2 comments

[06/22/24 07:25:20-07:00] [C] [12] [wb_cache.cpp:314] ******************** Assertion failure: =====> Expected '515' to be < to '515' down_buf=[Buf=0x60d00011ad20 index=0 state=1 create/dirty_cp=181/515 down_wait#=1 node=[0x621002d65600 id=4785078899048496 next=16888502897606726 nentries=168 level=0] ] up_buffer=[Buf=0x60d0000b3a20 index=0 state=1 create/dirty_cp=515/515 down_wait#=26 node=[0x621001e91200 id=12666378246946876 next=19421777688002604 nentries=116 level=1] ] should never have been created on same cp

bt
`#6 0x00007f9bd0149e96 in __GI___assert_fail (assertion=0x563e22ce6700 "0", file=0x563e22ce87e0 "/root/.conan/data/homestore/6.4.20-304///build/a9dcaf6078d4ec85d00647b2c1877a7a7aec2113/src/lib/index/wb_cache.cpp", line=314,
function=0x563e22ce9260 "void homestore::IndexWBCache::link_buf(const IndexBufferPtr&, const IndexBufferPtr&, bool, homestore::CPContext*)") at ./assert/assert.c:101
#7 0x0000563e20704906 in homestore::IndexWBCache::link_buf (this=0x611000034ac0, up_buf=std::shared_ptrhomestore::IndexBuffer (use count 29, weak count 1) = {...}, down_buf=std::shared_ptrhomestore::IndexBuffer (use count 4, weak count 1) = {...}, is_sibling_link=false,
cp_ctx=0x616000037280) at /root/.conan/data/homestore/6.4.20-304///build/a9dcaf6078d4ec85d00647b2c1877a7a7aec2113/src/lib/index/wb_cache.cpp:314
#8 0x0000563e206fe625 in homestore::IndexWBCache::transact_bufs (this=0x611000034ac0, index_ordinal=0, parent_buf=std::shared_ptrhomestore::IndexBuffer (use count 29, weak count 1) = {...}, child_buf=std::shared_ptrhomestore::IndexBuffer (use count 4, weak count 1) = {...},
new_node_bufs=..., freed_node_bufs=..., cp_ctx=0x616000037280) at /root/.conan/data/homestore/6.4.20-304///build/a9dcaf6078d4ec85d00647b2c1877a7a7aec2113/src/lib/index/wb_cache.cpp:243
#9 0x0000563e1fa02afd in homestore::IndexTable<TestFixedKey, TestFixedValue>::transact_nodes (this=0x613000028990, new_nodes=..., freed_nodes=..., left_child_node=..., parent_node=..., context=0x616000037280)
at /root/.conan/data/homestore/6.4.20-304///build/a9dcaf6078d4ec85d00647b2c1877a7a7aec2113/src/tests/../include/homestore/index/index_table.hpp:167
#10 0x0000563e1f70d97f in homestore::Btree<TestFixedKey, TestFixedValue>::split_node (this=0x613000028998, parent_node=..., child_node=..., parent_ind=17, out_split_key=0x7f9bc8610db0, context=0x616000037280)
at /root/.conan/data/homestore/6.4.20-304///build/a9dcaf6078d4ec85d00647b2c1877a7a7aec2113/src/tests/../include/homestore/btree/detail/btree_mutate_impl.ipp:295
#11 0x0000563e1f4598be in homestore::Btree<TestFixedKey, TestFixedValue>::do_puthomestore::BtreeSinglePutRequest (this=0x613000028998, my_node=..., curlock=homestore::locktype_t::WRITE, req=...)
at /root/.conan/data/homestore/6.4.20-304///build/a9dcaf6078d4ec85d00647b2c1877a7a7aec2113/src/tests/../include/homestore/btree/detail/btree_mutate_impl.ipp:104
#12 0x0000563e1f45fcf0 in homestore::Btree<TestFixedKey, TestFixedValue>::do_puthomestore::BtreeSinglePutRequest (this=0x613000028998, my_node=..., curlock=homestore::locktype_t::NONE, req=...)
at /root/.conan/data/homestore/6.4.20-304///build/a9dcaf6078d4ec85d00647b2c1877a7a7aec2113/src/tests/../include/homestore/btree/detail/btree_mutate_impl.ipp:159
#13 0x0000563e1f2e217b in homestore::Btree<TestFixedKey, TestFixedValue>::puthomestore::BtreeSinglePutRequest (this=0x613000028998, put_req=...) at /root/.conan/data/homestore/6.4.20-304///build/a9dcaf6078d4ec85d00647b2c1877a7a7aec2113/src/tests/../include/homestore/btree/btree.ipp:119
#14 0x0000563e1f1ffd3a in homestore::IndexTable<TestFixedKey, TestFixedValue>::puthomestore::BtreeSinglePutRequest (this=0x613000028990, put_req=...)
at /root/.conan/data/homestore/6.4.20-304///build/a9dcaf6078d4ec85d00647b2c1877a7a7aec2113/src/tests/../include/homestore/index/index_table.hpp:80
#15 0x0000563e1f168ceb in BtreeTestHelper::do_put (this=0x626000000110, k=396617, put_type=homestore::btree_put_type::INSERT, value=..., expect_success=true)
at /root/.conan/data/homestore/6.4.20-304///build/a9dcaf6078d4ec85d00647b2c1877a7a7aec2113/src/tests/btree_helpers/btree_test_helper.hpp:392
#16 0x0000563e1f164b16 in BtreeTestHelper::put_random (this=0x626000000110) at /root/.conan/data/homestore/6.4.20-304///build/a9dcaf6078d4ec85d00647b2c1877a7a7aec2113/src/tests/btree_helpers/btree_test_helper.hpp:171

`

Summary

The direct cause of this issue is that the up_buf is not correctly traced back to the real one during the link_buf process. This may happen in the case where a parent node is split, and later within the same CP, one of its original children - which was created in an earlier CP and just transmitted to the new parent node - also gets split.

Modifying this if statement could solve it, but its potential impact to other scenarios still needs further evaluation and testing.

Analysis

Analysis chart: https://lucid.app/lucidchart/b49a93b5-863b-4c69-b108-1cea20fc9326/edit?beaconFlowId=020F90D1DBA44395&invitationId=inv_2a4d0928-083d-409a-82f9-5eba8dcd4a81&page=0_0#

The coredump file reveals the following tree structure:

Index Long-running Issue

where the white nodes were created in earlier CPs and the colored ones were generated during the current CP.

During the current CP, events occurred in the following sequence:

  1. P1 split into P1 & P2
  2. L1 split into L1 & L2
  3. L1 split into L1 & L3

For the given tree structure, a dependency graph we are expecting should be like

Index Long-running Issue - Tree Structure (1)

However, when linking parent buf P2 with child buf L1 during step 2, it would fail to trace back the up buffer to the 'real' P1 since L1 is created in an earlier CP and the condition cannot be met. As a result, the up_buf of L1 would be incorrectly pointed to P2.

Then in the next step, when L1 got split again, this error led to the crash. link_buf would find it having an up_buf which was created in the current CP - and then fail the assertion.

Therefore, as mentioned in the summary, the direct cause of this incorrect dependency chain is that the tracing back logic for the real up buffer is not working properly.

Coredump Info

Root

(gdb) print *(class homestore::IndexBuffer*)0x60d00001abb0
$7 = {<sisl::ObjLifeCounter<homestore::IndexBuffer>> = {static s_created = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 28001}, static is_always_lock_free = true}, static s_alive = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 3004}, static is_always_lock_free = true}, static s_type = {m_dummy = 0}},
  _vptr.IndexBuffer = 0x563e23bc6ec8 <vtable for homestore::IndexBuffer+16>, 
  m_blkid = {s = {m_is_multi = 0, m_blk_num = 4, m_nblks = 1, m_chunk_num = 12}}, m_dirtied_cp_id = 515, m_created_cp_id = 0,
  m_state = std::atomic<homestore::index_buf_state_t> = { homestore::index_buf_state_t::DIRTY }, m_bytes = 0x62100011d200 "\253\001\274N\034",
  m_up_buffer = std::shared_ptr<homestore::IndexBuffer> (empty) = {get() = 0x0}, m_wait_for_down_buffers = {m_count = std::atomic<int> = { 5 }}, m_down_buffers = std::vector of length 5, capacity 8 = {
    std::weak_ptr<homestore::IndexBuffer> (use count 57, weak count 1) = {get() = 0x60d0000878e0}, 
    std::weak_ptr<homestore::IndexBuffer> (use count 70, weak count 1) = {get() = 0x60d0000f5e70}, 
    std::weak_ptr<homestore::IndexBuffer> (use count 77, weak count 1) = {get() = 0x60d00009d710}, 
    std::weak_ptr<homestore::IndexBuffer> (use count 75, weak count 1) = {get() = 0x60d000031e30}, 
    std::weak_ptr<homestore::IndexBuffer> (use count 73, weak count 1) = {get() = 0x60d000100f90}},
  m_prev_up_buffer = std::shared_ptr<homestore::IndexBuffer> (empty) = {get() = 0x0}, m_crash_flag_on = false, m_index_ordinal = 0, m_is_meta_buf = 0 '\000', m_node_freed = false}

(gdb) print *(struct homestore::persistent_hdr_t*)0x62100011d200
$8 = {magic = 171 '\253', version = 1 '\001', checksum = 20156, nentries = 28, leaf = 0, node_deleted = 0, node_id = 3377704015495176, next_node = 18446744073709551615, node_gen = 56, link_version = 0, edge_info = {m_bnodeid = 10133103456550978, m_link_version = 0}, modified_cp_id = 515, level = 2, node_size = 4095, node_type = 0 '\000', reserved = "\000\000"}

P1

(gdb) print {homestore::IndexBuffer}0x60d0000878e0
$1 = {<sisl::ObjLifeCounter<homestore::IndexBuffer>> = {static s_created = {<std::__atomic_base<long>> = {
        static _S_alignment = 8, _M_i = 28001}, static is_always_lock_free = true},
    static s_alive = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 3004}, static is_always_lock_free = true},
    static s_type = {m_dummy = 0}}, _vptr.IndexBuffer = 0x563e23bc6ec8 <vtable for homestore::IndexBuffer+16>, m_blkid = {s = {
      m_is_multi = 0, m_blk_num = 19, m_nblks = 1, m_chunk_num = 86}}, m_dirtied_cp_id = 515, m_created_cp_id = 15,
  m_state = std::atomic<homestore::index_buf_state_t> = { homestore::index_buf_state_t::DIRTY },
  m_bytes = 0x62100157c200 "\253\001\f\256p",
  m_up_buffer = std::shared_ptr<homestore::IndexBuffer> (use count 7, weak count 0) = {get() = 0x60d00001abb0},
  m_wait_for_down_buffers = {m_count = std::atomic<int> = { 56 }}, m_down_buffers = std::vector of length 56, capacity 64 = {
    std::weak_ptr<homestore::IndexBuffer> (use count 5, weak count 1) = {get() = 0x60d000040900},
    std::weak_ptr<homestore::IndexBuffer> (use count 5, weak count 1) = {get() = 0x60d000109810},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00009fe10},
    std::weak_ptr<homestore::IndexBuffer> (use count 5, weak count 1) = {get() = 0x60d00011f300},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000b4c00},
    std::weak_ptr<homestore::IndexBuffer> (use count 5, weak count 1) = {get() = 0x60d0000c1e70},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000098df0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000dc010},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000f7ef0},
    std::weak_ptr<homestore::IndexBuffer> (use count 3, weak count 1) = {get() = 0x60d000121450},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00006a820},
    std::weak_ptr<homestore::IndexBuffer> (use count 5, weak count 1) = {get() = 0x60d00012ee10},
    std::weak_ptr<homestore::IndexBuffer> (use count 5, weak count 1) = {get() = 0x60d000048620},
    std::weak_ptr<homestore::IndexBuffer> (use count 5, weak count 1) = {get() = 0x60d0000a6e30},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00005eee0},
    std::weak_ptr<homestore::IndexBuffer> (use count 3, weak count 2) = {get() = 0x60d000045970},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000118070},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0001670b0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00016ab30},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000da540},
    std::weak_ptr<homestore::IndexBuffer> (use count 5, weak count 1) = {get() = 0x60d000042640},
    std::weak_ptr<homestore::IndexBuffer> (use count 5, weak count 1) = {get() = 0x60d000048d70},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000a6880},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00016c6d0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000ca550},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000073720},
    std::weak_ptr<homestore::IndexBuffer> (use count 5, weak count 1) = {get() = 0x60d000145ef0},
    std::weak_ptr<homestore::IndexBuffer> (use count 3, weak count 1) = {get() = 0x60d000014140},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000cecd0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0001729f0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000431a0},
    std::weak_ptr<homestore::IndexBuffer> (use count 29, weak count 1) = {get() = 0x60d0000b3a20},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00012d410},
    std::weak_ptr<homestore::IndexBuffer> (use count 3, weak count 1) = {get() = 0x60d0000cdbc0},
    std::weak_ptr<homestore::IndexBuffer> (use count 5, weak count 1) = {get() = 0x60d00006e780},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00016cbb0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0001732e0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0001419e0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000d0940},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000131e00},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000e1560},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000168840},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000e2670},
    std::weak_ptr<homestore::IndexBuffer> (use count 3, weak count 1) = {get() = 0x60d0000e94f0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000a56a0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000107930},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000d7fe0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00016c7a0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000164330},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000b43e0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000c4300},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000e4b00},
    std::weak_ptr<homestore::IndexBuffer> (use count 3, weak count 1) = {get() = 0x60d0000f28d0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000c8330},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000160a50},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00015fef0}},
  m_prev_up_buffer = std::shared_ptr<homestore::IndexBuffer> (empty) = {get() = 0x0}, m_crash_flag_on = false,
  m_index_ordinal = 0, m_is_meta_buf = 0 '\000', m_node_freed = false}
(gdb) print {homestore::persistent_hdr_t}0x62100157c200
$2 = {magic = 171 '\253', version = 1 '\001', checksum = 44556, nentries = 112, leaf = 0, node_deleted = 0, node_id = 24206852292083750, next_node = 12666378246946876, node_gen = 560, link_version = 3, edge_info = {m_bnodeid = 18446744073709551615, m_link_version = 0}, modified_cp_id = 515, level = 1, node_size = 4095, node_type = 0 '\000', reserved = "\000\000"}

P2

(gdb) print *(class homestore::IndexBuffer*)0x60d0000b3a20
$6 = {
  <sisl::ObjLifeCounter<homestore::IndexBuffer>> = {
    static s_created = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 28001}, static is_always_lock_free = true}, static s_alive = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 3004}, static is_always_lock_free = true}, static s_type = {m_dummy = 0}},
  _vptr.IndexBuffer = 0x563e23bc6ec8 <vtable for homestore::IndexBuffer+16>,
  m_blkid = {s = {m_is_multi = 0, m_blk_num = 30, m_nblks = 1, m_chunk_num = 45}}, m_dirtied_cp_id = 515, m_created_cp_id = 515,
  m_state = std::atomic<homestore::index_buf_state_t> = { homestore::index_buf_state_t::DIRTY }, m_bytes = 0x621001e91200 "\253\001\035Zt",
  m_up_buffer = std::shared_ptr<homestore::IndexBuffer> (use count 57, weak count 1) = {get() = 0x60d0000878e0},
  m_wait_for_down_buffers = {m_count = std::atomic<int> = { 26 }},
  m_down_buffers = std::vector of length 26, capacity 32 = {
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000ddd50},
    std::weak_ptr<homestore::IndexBuffer> (use count 5, weak count 1) = {get() = 0x60d00005ff20},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000170630},
    std::weak_ptr<homestore::IndexBuffer> (use count 3, weak count 1) = {get() = 0x60d00010f4b0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000aa160},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00011ad20},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000114860},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000160980},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00010d840},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0001223c0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000417a0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0001451f0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000f8f30},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000088fa0},
    std::weak_ptr<homestore::IndexBuffer> (use count 3, weak count 1) = {get() = 0x60d0000e2b50},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00015eeb0},
    std::weak_ptr<homestore::IndexBuffer> (use count 3, weak count 1) = {get() = 0x60d0000841a0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000049a70},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000118960},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000171f60},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00012b530},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00010b140},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000143e70},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d0000357e0},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d000143a60},
    std::weak_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00009af40}},
  m_prev_up_buffer = std::shared_ptr<homestore::IndexBuffer> (empty) = {get() = 0x0}, m_crash_flag_on = false,
  m_index_ordinal = 0, m_is_meta_buf = 0 '\000', m_node_freed = false}

(gdb) print *(struct homestore::persistent_hdr_t*)0x621001e91200
$7 = {magic = 171 '\253', version = 1 '\001', checksum = 23069, nentries = 116, leaf = 0, node_deleted = 0, node_id = 12666378246946876, next_node = 19421777688002604, node_gen = 72, link_version = 0, edge_info = {m_bnodeid = 18446744073709551615, m_link_version = 0}, modified_cp_id = 515, level = 1, node_size = 4095, node_type = 0 '\000', reserved = "\000\000"}

L1

(gdb) print *(class homestore::IndexBuffer*)0x60d00011ad20
$14 = {<sisl::ObjLifeCounter<homestore::IndexBuffer>> = {
  static s_created = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 28001}, static is_always_lock_free = true}, static s_alive = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 3004}, static is_always_lock_free = true}, static s_type = {m_dummy = 0}},
  _vptr.IndexBuffer = 0x563e23bc6ec8 <vtable for homestore::IndexBuffer+16>,
  m_blkid = {s = {m_is_multi = 0, m_blk_num = 24, m_nblks = 1, m_chunk_num = 17}}, m_dirtied_cp_id = 515, m_created_cp_id = 181,
  m_state = std::atomic<homestore::index_buf_state_t> = { homestore::index_buf_state_t::DIRTY }, m_bytes = 0x621002d65600 "\253\001\366\225\250",
  m_up_buffer = std::shared_ptr<homestore::IndexBuffer> (use count 29, weak count 1) = {get() = 0x60d0000b3a20},
  m_wait_for_down_buffers = {m_count = std::atomic<int> = { 1 }},
  m_down_buffers = std::vector of length 1, capacity 1 = {std::weak_ptr<homestore::IndexBuffer> (use count 2, weak count 1) = {get() = 0x60d00003b620}},
  m_prev_up_buffer = std::shared_ptr<homestore::IndexBuffer> (empty) = {get() = 0x0}, m_crash_flag_on = false, m_index_ordinal = 0, m_is_meta_buf = 0 '\000', m_node_freed = false}

(gdb) print *(struct homestore::persistent_hdr_t*)0x621002d65600
$15 = {magic = 171 '\253', version = 1 '\001', checksum = 38390, nentries = 168, leaf = 1, node_deleted = 0, node_id = 4785078899048496, next_node = 16888502897606726, node_gen = 9368, link_version = 2, edge_info = {m_bnodeid = 18446744073709551615, m_link_version = 0}, modified_cp_id = 515, level = 0, node_size = 4095, node_type = 0 '\000', reserved = "\000\000"}

L2

(gdb) print *(class homestore::IndexBuffer*)0x60d00003b620
$1 = {<sisl::ObjLifeCounter<homestore::IndexBuffer>> = {static s_created = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 28001}, static is_always_lock_free = true},
    static s_alive = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 3004}, static is_always_lock_free = true}, static s_type = {m_dummy = 0}},
  _vptr.IndexBuffer = 0x563e23bc6ec8 <vtable for homestore::IndexBuffer+16>, m_blkid = {s = {m_is_multi = 0, m_blk_num = 31, m_nblks = 1, m_chunk_num = 48}}, m_dirtied_cp_id = 515, m_created_cp_id = 515,
  m_state = std::atomic<homestore::index_buf_state_t> = { homestore::index_buf_state_t::DIRTY }, m_bytes = 0x621000746a00 "\253\001C\033\267",
  m_up_buffer = std::shared_ptr<homestore::IndexBuffer> (use count 4, weak count 1) = {get() = 0x60d00011ad20}, m_wait_for_down_buffers = {m_count = std::atomic<int> = { 0 }},
  m_down_buffers = std::vector of length 0, capacity 0, m_prev_up_buffer = std::shared_ptr<homestore::IndexBuffer> (empty) = {get() = 0x0}, m_crash_flag_on = false, m_index_ordinal = 0, m_is_meta_buf = 0 '\000',
  m_node_freed = false}

(gdb) print {homestore::persistent_hdr_t}0x621000746a00
$2 = {magic = 171 '\253', version = 1 '\001', checksum = 6979, nentries = 183, leaf = 1, node_deleted = 0, node_id = 13510803177078846, next_node = 16325552944185382, node_gen = 1682, link_version = 0, edge_info = {
    m_bnodeid = 18446744073709551615, m_link_version = 0}, modified_cp_id = 515, level = 0, node_size = 4095, node_type = 0 '\000', reserved = "\000\000"}

L3

(gdb) print {homestore::IndexBuffer}0x60d0000f7940
$6 = {<sisl::ObjLifeCounter<homestore::IndexBuffer>> = {static s_created = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 28001}, static is_always_lock_free = true}, static s_alive = {<std::__atomic_base<long>> = {static _S_alignment = 8, _M_i = 3004}, static is_always_lock_free = true}, static s_type = {m_dummy = 0}},
  _vptr.IndexBuffer = 0x563e23bc6ec8 <vtable for homestore::IndexBuffer+16>, 
  m_blkid = {s = {m_is_multi = 0, m_blk_num = 35, m_nblks = 1, m_chunk_num = 60}}, m_dirtied_cp_id = 515, m_created_cp_id = 515,
  m_state = std::atomic<homestore::index_buf_state_t> = { homestore::index_buf_state_t::DIRTY }, m_bytes = 0x621000a08e00 "\253\001\200\373\247", 
  m_up_buffer = std::shared_ptr<homestore::IndexBuffer> (empty) = {get() = 0x0}, m_wait_for_down_buffers = {m_count = std::atomic<int> = { 0 }}, 
  m_down_buffers = std::vector of length 0, capacity 0, m_prev_up_buffer = std::shared_ptr<homestore::IndexBuffer> (empty) = {get() = 0x0}, 
  m_crash_flag_on = false, m_index_ordinal = 0, m_is_meta_buf = 0 '\000', m_node_freed = false}

(gdb) print {homestore::persistent_hdr_t}0x621000a08e00
$7 = {magic = 171 '\253', version = 1 '\001', checksum = 64384, nentries = 167, leaf = 1, node_deleted = 0, node_id = 16888502897606726, next_node = 13510803177078846, node_gen = 1, link_version = 0, edge_info = {m_bnodeid = 18446744073709551615, m_link_version = 0}, modified_cp_id = 515, level = 0, node_size = 4095, node_type = 0 '\000', reserved = "\000\000"}