vlsi/compactmap

If the order of keys is inconsistent during deserialization, OOM will occur.

Closed this issue · 6 comments

first deserialization, if key order is:
key1
key2
key3

sencond deserialization, if key order is:
key2
key3
key1

when do deserialize multiple times use compactmap , will trigger OOM

vlsi commented

Do you mean serialize-deserialize within the same JVM or across JVMs?

@vlsi in my case , it is across JVMs;
but i write a simple code to test this scenes, it will reappear

oom simple code(keys are out of order):

public class CompactRandomOrder {
    private static final int ENTRY_CARD = 1000;
    private static final int LOOP_COUNT = 100000;
    private static final int MAP_SIZE = 50;
    private static Random random = new Random(ENTRY_CARD);
    public static void main(String[] args) throws InterruptedException {
        List<String> keyList = new ArrayList<>();
        List<String> valueList = new ArrayList<>();
        for (int i = 0; i < ENTRY_CARD; i++) {
            keyList.add(i + "#keyKeykeyKeykeyKeykeyKeykeyKeykeyKeykeyKeykeyKeykeyKey" + random.nextInt());
            valueList.add(i + "#valueValuevalueValuevalueValuevalueValuevalueValue" + random.nextInt());
        }
        int idx;
        List<Map> listMap = new ArrayList<>(LOOP_COUNT);
        for (int i = 0; i < LOOP_COUNT; i++) {
            CompactHashMap map = new CompactHashMap();
            for (int j = 0; j < MAP_SIZE; j++) {
                idx = Math.abs(random.nextInt()) % ENTRY_CARD;
                map.put(keyList.get(idx), valueList.get(idx));
            }
            listMap.add(map);
        }

        System.out.println("has filled");
        Thread.sleep(200000);
    }
}

normal code (keys are sequential)

public class CompactOrderly {
    private static final int ENTRY_CARD = 1000;
    private static final int LOOP_COUNT = 100000;
    private static final int MAP_SIZE = 50;
    private static Random random = new Random(ENTRY_CARD);
    @SneakyThrows
    public static void main(String[] args)  {
        List<String> keyList = new ArrayList<>();
        List<String> valueList = new ArrayList<>();
        for (int i = 0; i < ENTRY_CARD; i++) {
            keyList.add(i + "#keyKeykeyKeykeyKeykeyKeykeyKeykeyKeykeyKeykeyKeykeyKey" + random.nextInt());
            valueList.add(i + "#valueValuevalueValuevalueValuevalueValuevalueValue" + random.nextInt());
        }
        int idx;
        List<Map> listMap = new ArrayList<>(LOOP_COUNT);
        for (int i = 0; i < LOOP_COUNT; i++) {
            CompactHashMap map = new CompactHashMap();
            idx = Math.abs(random.nextInt()) % (ENTRY_CARD - MAP_SIZE);
            for (int j = idx; j < ENTRY_CARD; j++) {
                if (j - idx >= 50) {
                    break;
                }
                map.put(keyList.get(j), valueList.get(j));
            }
            listMap.add(map);
        }


        System.out.println("has filled");
        Thread.sleep(200000);
    }
}


image

@vlsi any idea?

vlsi commented

I've published 2.0.1, and it should appear on Central soon.

vlsi commented

Thanks for figuring out the root cause.