Notes

This application runs in SGX SIMULATION mode, if you need to run it in hardware mode, please first ensure that the CPU on your device is newer than the 6th generation (Skylake), and that the SGX feature is fully enabled by the BIOS settings. Also, to run the application, you need to install Intel's SGX SDK and SGX drivers for the Linux OS.

Some Guidances For Writing SGX Applications

Enclave functions should be declared in ./include/enclave/*.hh.
Unstrusted functions should be declared in ./include/app/*.hh.
Wrapper functions such as ecall_xxx(), ocall_xxx() should also be defined in ./include/app/*.hh.
The trusted interfaces declared in enclave.edl should be implemented in file ./src/enclave/*.cc, and the untrusted interfaces should be implemented in file ./src/app/*.cc.
The proxy functions are auto-generated by sgx_edger8r, do not modify them.
The enclave memory is very limited, so any malloced memory must be freed to prevent memory leakage in the enclave. Typical errors indicating OOM in the enclave are:
- SIGSEGV -> segmentation fault
- SIGABRT -> std::bad_alloc
- SIGILL -> illegal hardware instruction
You could always set the SGX environment arguments by adding one command to your shell's rc:

echo "source $SGX_SDK/environment" >> $HOME/.zshrc; # or $HOME/.bashrc

Remote Attestation

This version ~~enables~~ disables the Remote Attestation technology for Intel SGX. We only take use of the session key computation framework of SGX.
To serialize all the messages that are being transported via network, Google Protobuf is a must.
Also, to start an encrypted session and make the application easier to use, we deployed the Google RPC (Remote Procedure Call).

Build our project

Prerequisites

First, install the following dependencies:

libspdlog (for logging)
libgflags (for command line parsing)
liblz4 (for compressing)

To build our project, you may also need to install Intel SGX SDK for Linux system, and you also need to install gRPC framework for the connection between the client and the server, and we strongly recommend that one should install gRPC via source rather than package manager. One could clone the repository from gRPC's official github page:

git clone --recurse-submodules https://github.com/grpc/grpc $HOME/grpc

Configure it with cmake and install it on the computer:

cd $HOME/grpc
mkdir -p cmake/build
pushd cmake/build
cmake -DgRPC_INSTALL=ON \                
      -DgRPC_BUILD_TESTS=OFF \
      # Do not link to libssl provided by the system.
      -DgRPC_SSL_PROVIDER=package \ 
      # Change it to your own preferred directory, but better not install globally.
      -DCMAKE_INSTALL_PREFIX=/usr/local \
      -DBUILD_SHARED_LIBS=ON \
      ../..
make -j
sudo make install
popd

For more information, interested readers are referred to this. Note: IF GRPC is as shared library, you CANNOT run SGX in hardware mode because libsgx_quote_ex.so depends on libprotobuf.so.10 which conflicts with libprotobuf.so.3 required by libgrpc++_reflection.so. In this case, please remove -DBUILD_SHARED_LIBS=ON and link to the static libraries. Also, it is recommended that gRPC should be installed in a seperate directory such as ~/.local to prevent confliction with other dynamic libraries.

Also, you may also need to set the environment for the SGX SDK so that you can properly build the enclave:

source <SGX_PATH>/sgxsdk/environment

In addition, you should manually set the correct path for gRPC for loading the correct gRPC libraries in client.mk and server.mk.

Build the server:

make -j server

Build the client:

make -j client

Build all:

make -j all

Run the project

Extra care must be paid when one tries to properly run the server and the client. Before running, please make sure that the path of loaded libraries is correctly set, and the ssl key is generated:

. ./env.sh
sh -c ./keygen.sh

Otherwise, the system will throw an exception indicating that the libsample_libcrypto and libservice_provider cannot be found.

If you upgraded gRPC, the old CPP source files auto-generated by the protobuf-compiler should be completely deleted to prevent bugs, and if you changed some macros in the header, you may also need a fresh build.

Finally, you could run the client and the server by

./build/bin/server.bin --address=0.0.0.0 --port=5678 --seg_size=0 --cache_enabled=1 --log_level=2 --log_to_stderr
https_proxy="" ./build/bin/client.bin --type 2 --bucket_size=128 --number=500000 -way=32 --access_num=1  --port=5678 --log_to_stderr
# The environment variable $https_proxy cannot be set when running client.
# If you want to specify the parameters, run ./build/bin/server.bin --help .

Make sure that your system does not use any proxy; otherwise the client cannot connect to the local server.

All the logs will be output to the directory ./log.

Debug the project

The compiled binary file of client residing ./build/bin/client.bin can be debugged with arbitrary tools including gdb, lldb. However, for the server, we need to use sgx-sgx to load all the debug symbols in the enclave. Otherwise the debug tool will report symbol missing error upon trapping into the memory space owned by the enclave if one types bt full. For example:

error: ld-2.27.so 0x7fffffff0005c896: adding range [0x14870-0x14876) which has a base that is less than the function's low PC 0x14f60. Please file a bug and attach the file at the start of this error message

To check the memory used by the enclave, you must create a global .gdbinit in your user directory:

touch ~/.gdbinit
echo \
"set auto-load local-gdbinit on
add-auto-load-safe-path </path/to/project>/.gdbinit" >> ~/.gdbinit

Then the sgx-gdb will print the memory usage information when the enclave is destroyed:

Enclave: "./build/server/enclave/enclave_signed.so"
  [Peak stack used]: 16 KB
  [Peak heap used]:  2648 KB
  [Peak reserved memory used]:  0 KB