Is it possible to use memory mapping to load / save a dense hash map?
Opened this issue · 4 comments
GoogleCodeExporter commented
What steps will reproduce the problem?
If a dense hash map contains a large number of key-value pairs (both POD), is
it possible to use a memory mapping file to load / save it?
The data eats memory a log, and I need to load them in multiple processes. If
memory mapping is supported, the footprint can be shared between processes,
which saves a lot of memory.
I'm not familiar with dense hash map's implementation, just ask here for
whether it is possible, or someone has tried this.
Thank you in advance.
What is the expected output? What do you see instead?
What version of the product are you using? On what operating system?
Please provide any additional information below.
Original issue reported on code.google.com by huas...@gmail.com
on 14 Jan 2012 at 3:45
GoogleCodeExporter commented
An interesting idea! You'd have a read-only hashtable then, is that right?
This is not possible right now. I'm wondering if you could do something
similar using shared memory (which I recognize is a lot more finicky to work
with) but even that I think would be pretty tricky, and involve writing a
custom allocator.
If you end up trying to write something like this, I'd be happy to look at a
patch! I don't know how intrusive such a change would be. In practice, you'd
probably want just to mmap the table array (in
densehashtable.h:dense_hashtable), and have each process own its own copy of
all the other data (everything else is small). Then you'd want to write your
own version of Serialize() and Unserialize() that mmap in the table or
something.
Original comment by csilv...@gmail.com
on 17 Jan 2012 at 7:24
- Added labels: Priority-Medium, Type-Enhancement
GoogleCodeExporter commented
If the table array value type is POD (without pointer type), it's straight
forward to save the table array's memory across processes using either mmap or
shared memory.
I'll try to implement Serialize() and Unserialize() with mmap first, and if
success, put the patch here. Please help me review it then.
Original comment by huas...@gmail.com
on 18 Jan 2012 at 4:16
GoogleCodeExporter commented
Hi
Have you considered to handle concurrent-write issue? Or Does it just support
read?
Original comment by baibaic...@gmail.com
on 18 Jan 2012 at 5:43
GoogleCodeExporter commented
why it will wrong
when iter is large 10000000
my english is bad
hope someone can help me
Original comment by pw...@sina.com
on 13 Jul 2012 at 9:44
Attachments: