wuxb45/remixdb

compile error

Closed this issue · 6 comments

AnonyK1 commented

First of all, thank you for sharing your works : )

I tried to compile with clang-12 and kernel version 5.12.10 , however, I still see errors

[root@localhost remixdb]# make M=j xdbdemo.out
xdbdemo.out <= xdbdemo.c lib.c kv.c wh.c blkio.c sst.c xdb.c 
clang -std=gnu18   -march=native -mtune=native -pthread -Wall -Wextra -Wshadow  -fno-builtin-memcpy -fno-builtin-memmove -fno-builtin-memcmp -DNDEBUG -g3 -O3 -flto -fno-stack-protector -ferror-limit=3  -rdynamic -o xdbdemo.out xdbdemo.c lib.c kv.c wh.c blkio.c sst.c xdb.c  -lm -ljemalloc -lrt
lib.c:710:59: error: use of undeclared identifier 'MAP_HUGE_SHIFT'
  return pages_do_alloc(sz, MAP_PRIVATE | MAP_ANONYMOUS | PAGES_FLAGS_1G);
                                                          ^
lib.c:696:47: note: expanded from macro 'PAGES_FLAGS_1G'
#define PAGES_FLAGS_1G ((MAP_HUGETLB | (30 << MAP_HUGE_SHIFT)))
                                              ^
lib.c:724:59: error: use of undeclared identifier 'MAP_HUGE_SHIFT'
  return pages_do_alloc(sz, MAP_PRIVATE | MAP_ANONYMOUS | PAGES_FLAGS_2M);
                                                          ^
lib.c:697:47: note: expanded from macro 'PAGES_FLAGS_2M'
#define PAGES_FLAGS_2M ((MAP_HUGETLB | (21 << MAP_HUGE_SHIFT)))
                                              ^
2 errors generated.
make: *** [Makefile.common:180: xdbdemo.out] Error 1

And I add header in lib.h

#include <asm-generic/mman-common.h>

and new errors:

[root@localhost remixdb]# make M=j xdbdemo.out
xdbdemo.out <= xdbdemo.c lib.c kv.c wh.c blkio.c sst.c xdb.c 
clang -std=gnu18   -march=native -mtune=native -pthread -Wall -Wextra -Wshadow  -fno-builtin-memcpy -fno-builtin-memmove -fno-builtin-memcmp -DNDEBUG -g3 -O3 -flto -fno-stack-protector -ferror-limit=3  -rdynamic -o xdbdemo.out xdbdemo.c lib.c kv.c wh.c blkio.c sst.c xdb.c  -lm -ljemalloc -lrt
lib.c:2015:10: error: always_inline function '_pext_u32' requires target feature 'bmi2', but would be inlined into function 'm128_movemask_u16' that is compiled without support for 'bmi2'
  return _pext_u32((u32)_mm_movemask_epi8(v), 0xaaaau);
         ^
1 error generated.
make: *** [Makefile.common:180: xdbdemo.out] Error 1
wuxb45 commented

What is your Linux distro? according to man 2 mmap the macros should be available with recent Linux kernels and headers.
If you can't make it work, just change the flags to MAP_HUGETLB or even 0.

#define PAGES_FLAGS_1G ((MAP_HUGETLB))    // or just 0
#define PAGES_FLAGS_2M ((MAP_HUGETLB))    // or just 0
wuxb45 commented

It might be a missing _GNU_SOURCE in xdbdemo.out. I have pushed a new commit but I don't know if it fixes the problem since I cannot reproduce the error in my Archlinux.

AnonyK1 commented

What is your Linux distro? according to man 2 mmap the macros should be available with recent Linux kernels and headers.
If you can't make it work, just change the flags to MAP_HUGETLB or even 0.

#define PAGES_FLAGS_1G ((MAP_HUGETLB))    // or just 0
#define PAGES_FLAGS_2M ((MAP_HUGETLB))    // or just 0

Thanks for your reply!
My linux distro is CentOS 7.4. According to man 2 mmap, MAP_HUGE_SHIFT is not available.

[root@localhost remixdb]# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.4.1708 (Core) 
Release:        7.4.1708
Codename:       Core
  • So I change the flags to MAP_HUGETLB and pull the new commit, and the second error still exsits.
[root@localhost remixdb]# make M=j xdbdemo.out
xdbdemo.out <= xdbdemo.c lib.c kv.c wh.c blkio.c sst.c xdb.c 
clang -std=gnu18   -march=native -mtune=native -pthread -Wall -Wextra -Wshadow  -fno-builtin-memcpy -fno-builtin-memmove -fno-builtin-memcmp -DNDEBUG -g3 -O3 -flto -fno-stack-protector -ferror-limit=3  -rdynamic -o xdbdemo.out xdbdemo.c lib.c kv.c wh.c blkio.c sst.c xdb.c  -lm -ljemalloc -lrt
lib.c:2019:10: error: always_inline function '_pext_u32' requires target feature 'bmi2', but would be inlined into function 'm128_movemask_u16' that is compiled without support for 'bmi2'
  return _pext_u32((u32)_mm_movemask_epi8(v), 0xaaaau);
         ^
1 error generated.
make: *** [Makefile.common:180: xdbdemo.out] Error 1
AnonyK1 commented

After check, the cpu in our server may not support instruction bmi2

#include <stdio.h>

int main()
{
    __builtin_cpu_init();
    if (__builtin_cpu_supports("bmi2"))
    {
        printf("BMI2 supported \n");
    } else {
        printf("BMI2 not supported \n");
    }
    return 0;
}
  • Output:
BMI2 not supported
  • Our cpu info
     24  Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz
wuxb45 commented

It looks the pext are only in two unused functions. I have removed these functions and it now compiles with make ARCH=ivybridge M=j on my machine. Please try the latest commit.

AnonyK1 commented

@wuxb45 Thanks for your support!
After fixing some issues of clang-env and using latest commit, I can run remixdb correctly!
Thanks a lot!

Best Regards

  • 🚀1