In version 0.9.36, RoaringBitmap fixed compatibility issues with CRoaring

[wzqiang1332]

RoaringBitmap/RoaringBitmap@0bebf92

When set Roaring64NavigableMap. SERIALIZATION_MODE = Roaring64NavigableMap. SERIALIZATION_MODE_PORTABLE, Java's Roaring64NavigableMap is consistent with CRoaring's binary format.

[zhicwu]

Thanks a lot @wzqiang1332! I was thinking to fix this one in weekend :p

Would you mind to add some integtation tests to prove the change works? If you have extra time, maybe we can port the code and drop the dependency.

[standup-jb]

In version 0.9.36, RoaringBitmap fixed compatibility issues with CRoaring。

这里的version 0.9.36 是指代什么的version？clickhouse集群的还是clickhouse-jdbc的？

[standup-jb]

In version 0.9.36, RoaringBitmap fixed compatibility issues with CRoaring。

这里的version 0.9.36 是指代什么的version？clickhouse集群的还是clickhouse-jdbc的？

@wzqiang1332 @zhicwu

[standup-jb]

In version 0.9.36, RoaringBitmap fixed compatibility issues with CRoaring。
这里的version 0.9.36 是指代什么的version？clickhouse集群的还是clickhouse-jdbc的？

@wzqiang1332 @zhicwu

I Got it https:/RoaringBitmap/RoaringBitmap

[standup-jb]

@wzqiang1332 @zhicwu 我用了0.9.36版本的Roaring64NavigableMap，但是这个问题还是没有解决。当bitmap里面的数少于32个的时候没问题，大于32个保存到ck后，依旧出错。我的代码如下

` Roaring64NavigableMap map = new Roaring64NavigableMap();
for (int i=0;i<100;i++){
map.add(i);
}
System.out.println(map.serializedSizeInBytes());
Roaring64NavigableMap.SERIALIZATION_MODE = Roaring64NavigableMap.SERIALIZATION_MODE_PORTABLE;
ClickhouseUtil.init();
ClickhouseUtil.save("1_2","20230101",map);

public static void init()  {
    try {
        Properties properties = new Properties();
        properties.setProperty("user", USERNAME);
        properties.setProperty("password",PASSWORD);
        properties.setProperty("continueBatchOnError", "true");
        clickHouseDataSource = new ClickHouseDataSource("jdbc:ch://ip:8123/abTest",properties);
        System.out.printf("create ck datasource finished ");
    }catch (Exception e){
        System.out.printf("create ck datasource error : message %s ",e.getMessage());
    }
}

public static void save(String xabtestId,String dt, Roaring64NavigableMap bitmap) throws SQLException {
    ClickHouseConnection connection = clickHouseDataSource.getConnection();
    PreparedStatement statement = connection.prepareStatement(
            "insert into abTest.xabtest_dispatch_bitmap_v2 " +
            "select xabtest_id,dt,create_at,bitmap from " +
            "input('\n" +
            "xabtest_id String, dt String, create_at DateTime , bitmap AggregateFunction" +
            "(groupBitmap, UInt64)')");
    statement.setString(1,xabtestId);
    statement.setString(2,dt);
    statement.setDate(3,new Date(System.currentTimeMillis()));
    statement.setObject(4, ClickHouseBitmap.wrap(bitmap, ClickHouseDataType.UInt64));
    //statement.setObject(4, new ClickHouseBitmap.ClickHouseRoaring64NavigableMap(bitmap, ClickHouseDataType.UInt64));
    statement.execute();

// statement.addBatch();
// int[] result = statement.executeBatch();
// System.out.printf("xabtestId:%s result:%s \n",xabtestId, Arrays.toString(result));
}
}`

[standup-jb]

RoaringBitmap/RoaringBitmap@0bebf92

When set Roaring64NavigableMap. SERIALIZATION_MODE = Roaring64NavigableMap. SERIALIZATION_MODE_PORTABLE, Java's Roaring64NavigableMap is consistent with CRoaring's binary format.

Can you show us your test code , I still meet this bug. I used the version 0.9.36 .

[qwemicheal]

I think before 0.41 fix this. You could manually reverse the highToBitmap using code like the following.

reverseHighInt(Roaring64NavigableMap map) {
        Long cardinality = map.getLongCardinality();
        if (cardinality>32){
        Field highToBitmapField = Roaring64NavigableMap.class.getDeclaredField("highToBitmap");
        AccessibleObject.setAccessible(new AccessibleObject[]{highToBitmapField}, true);
        NavigableMap<Integer, BitmapDataProvider> highToBitmap =
                (NavigableMap<Integer, BitmapDataProvider>) highToBitmapField.get(map);
        Set<Integer> oldKeys = new HashSet<>(highToBitmap.keySet());
        NavigableMap<Integer, BitmapDataProvider> keyThatNeedReverse = new TreeMap<>();
        NavigableSet<Integer> keyThatNeedRemove = new TreeSet<>();
        oldKeys.forEach(key -> {
            BitmapDataProvider provider = highToBitmap.get(key);
            keyThatNeedRemove.add(key);
            int high = Integer.reverseBytes(key);
            keyThatNeedReverse.put(high,provider);
        });
        for(Integer key: keyThatNeedRemove){
            highToBitmap.remove(key);
        }
        for (Integer reversedKey : keyThatNeedReverse.keySet()) {
            highToBitmap.put(reversedKey, keyThatNeedReverse.get(reversedKey));
        }
    }
}