Apollo+LLVM Notes

Tips

!!! Do not use dev_start.sh to start dev docker after rebooting the machine

For the first time of starting Apollo dev docker, you can use bash docker/scripts/dev_start.sh. However, after rebooting, if you still use dev_start.sh, you will lose all previous modifications within the docker, because the dev docker container will be deleted. Using the following commands can avoid such issue.

Apollo 3.0

docker run -it -d --rm --name apollo_localization_volume apolloauto/apollo:localization_volume-x86_64-latest
docker run -it -d --rm --name apollo_yolo3d_volume apolloauto/apollo:yolo3d_volume-x86_64-latest
docker run -it -d --rm --name apollo_map_volume-sunnyvale_big_loop apolloauto/apollo:map_volume-sunnyvale_big_loop-latest
docker run -it -d --rm --name apollo_map_volume-sunnyvale_loop apolloauto/apollo:map_volume-sunnyvale_loop-latest

# Make sure that you can see `apollo_dev` in `docker ps -a` as `EXITED`
docker start apollo_dev

bash docker/scripts/dev_into.sh

Apollo 5.0

# Copy `dev_restart.sh` in this directory to Apollo's root directory
cp dev_restart.sh <path to apollo>/docker/scripts

# Make sure that you execute the following commands in Apollo's root directory
cd <path to apollo>
bash docker/scripts/dev_restart.sh

bash docker/scripts/dev_into.sh

May need to delete inconsistent gtest header file

sudo mv /usr/include/gtest /usr/include/gtest_bak

Compile Apollo using LLVM

(All following commands are assumed to be executed in Apollo dev docker)

Install LLVM

LLVM 3.4

# LLVM 3.4
sudo apt install llvm-3.4 llvm-3.4-dev clang-3.4 libclang-3.4-dev

# Create soft links
sudo ln -sf /usr/bin/llvm-config-3.4 /usr/bin/llvm-config
sudo ln -sf /usr/bin/llvm-link-3.4 /usr/bin/llvm-link

LLVM 8 (Recommended)

# LLVM 8
sudo apt install llvm-8 llvm-8-dev clang-8 libclang-8-dev

# Create soft links
sudo ln -sf /usr/bin/llvm-config-8 /usr/bin/llvm-config
sudo ln -sf /usr/bin/llvm-link-8 /usr/bin/llvm-link

Install whole-program-llvm
```
sudo pip install wllvm
```
Copy wllvm directory to /apollo/tools
```
cp -r wllvm /apollo/tools
```
Before compiling (Apollo 5.0 only)
1. Modify modules/planning/reference_line/spiral_problem_interface.h
  
  Change constexpr static size_t N = 10; at line 140 from private to public.
2. Modify cyber/scheduler/policy/classic_context.cc
  
  Add alignas(CACHELINE_SIZE) for line 32~35, see the following:
```
alignas(CACHELINE_SIZE) GRP_WQ_MUTEX ClassicContext::mtx_wq_;
alignas(CACHELINE_SIZE) GRP_WQ_CV ClassicContext::cv_wq_;
alignas(CACHELINE_SIZE) RQ_LOCK_GROUP ClassicContext::rq_locks_;
alignas(CACHELINE_SIZE) CR_GROUP ClassicContext::cr_group_;
```
3. Disable compiler options
  
  Comment out Line 65~Bottom in tools/bazel.rc.

Compile Apollo

cd /apollo
mkdir wllvm_bc  # This directory will contain all seperate bitcode files
bash apollo.sh build --copt=-mavx2 --cxxopt=-mavx2 --copt=-mno-sse3 --crosstool_top=tools/wllvm:toolchain

Those copt and cxxopt can be removed, if your machine supports the corresponding instruction sets.

Or compile single module (e.g., planning module)

## Apollo 3.0
bazel build --define ARCH=x86_64 --define CAN_CARD=fake_can --cxxopt=-DUSE_ESD_CAN=false --copt=-mavx2 --copt=-mno-sse3 --cxxopt=-DCPU_ONLY --crosstool_top=tools/wllvm:toolchain //modules/planning:planning --compilation_mode=dbg

## Apollo 5.0
bash apollo.sh build_planning --cxxopt=-mavx2 --copt=-mno-sse3 --crosstool_top=tools/wllvm:toolchain  # recommended

# Or
bazel build --ram_utilization_factor 80 --define ARCH=x86_64 --define CAN_CARD=fake_can --cxxopt=-DUSE_ESD_CAN=false --copt=-mavx2 --copt=-mno-sse3 --cxxopt=-DCPU_ONLY --experimental_multi_threaded_digest --crosstool_top=tools/wllvm:toolchain --compilation_mode=dbg //modules/planning:libplanning_component.so

Extract bitcode file (e.g., planning module)

Apollo 3.0

cd /apollo/bazel-bin/modules/planning
extract-bc planning

# Check output
file planning.bc
llvm-dis planning.bc

Apollo 3.5 & 5.0

sudo apt install python3-pip
sudo pip3 install sh
python3 ApolloBitcodeExtractor.py

Usage of ApolloBitcodeExtractor.py:

usage: ApolloBitcodeExtractor.py [-h] [-t TARGET] [-p PROJECT] [-o OUTPUT]

Apollo LLVM bitcode extractor for any Bazel targets

optional arguments:
  -h, --help            show this help message and exit
  -t TARGET, --target TARGET
                        A Bazel build target (default:
                        //modules/planning:libplanning_component.so)
  -p PROJECT, --project PROJECT
                        The root directory of a Bazel project (default:
                        /apollo)
  -o OUTPUT, --output OUTPUT
                        The name of the output bitcode (default: output.bc)

h1994st/ApolloLLVMCompile

Apollo+LLVM Notes

Tips

Compile Apollo using LLVM