#################################
PENCIL Image Processing Benchmark
#################################
# Prerequisites
################
- An OpenCL driver, header files and libraries.
- The following packages:
* build-essential
* cmake
* TBB (Threading Building Blocks)
- A PENCIL compiler (PPCG for example).
- The PRL runtime library (included in PPCG).
- The PENCIL header files (included in PPCG).
- OpenCV (2.4.9.1 or above):
* Tested with 2.4.9.1 and 2.4.10.1
* You can install the OpenCV packages (not tested)
or download and compile the sources (as described below).
* Download OpenCV
git clone https://github.com/Itseez/opencv.git
* Checkout 2.4.9.1 or newer:
cd opencv
git checkout 2.4.10.1
cd ..
* If you are running the benchmark on ARM Mali GPU you need to apply the patch
0001-Make-image-filtering-erode-dilate-work-with-ARM-Mali.patch
provided with the benchmark to OpenCV. This patch changes the workgroup sizes to smaller ones so that OpenCV kernels can run on Mali (which only accepts small workgroups).
cd opencv
git apply ../pencil-benchmarks-imageproc/0001-Make-image-filtering-erode-dilate-work-with-ARM-Mali.patch
cd ..
* Configure build with CMake:
mkdir opencv-build
cd opencv-build
cmake ../opencv
* Details:
- If your CPU supports AVX/SSEx instruction sets, you can add the following to the "cmake ../opencv" command:
-DENABLE_AVX=ON -DENABLE_SSE41=ON -DENABLE_SSE42=ON -DENABLE_SSSE3=ON -DENABLE_SSE3=ON
- If your CPU is ARM, set the appropriate NEON/VFP switches.
- Optional: add TBB usage with -DWITH_TBB=ON (requires libtbb-dev package).
- Make sure the necessary modules are to be built - CMake should print a (long) status with a "To be built:" list, it should contain core, ocl and highgui (plus their dependencies).
* Build OpenCV
make all -j12
* Details: You can replace the number 12 with the number of threads of your processor(s)
* Optionally, install OpenCV as a system library:
sudo make install
############
# Building
############
Two methods are possible. You can either use bash scripts (in scripts/) or
use CMake.
# Building Using Bash Scripts (recommended):
##############################################
We recommend this method as you can use the same scripts to perform auto-tuning
which is not possible with the Cmake method.
- Set the variables in scripts/scripts_config.conf to define the paths of the
libraries, headers and tools used in the benchmark.
PENCIL_COMPILER_BINARY: the full path of the PENCIL compiler binary
PENCIL_INCLUDE_DIR: path of the directory containing "pencil.h"
PRL_LIB_DIR: path of the directory containing the PRL library (libprl.so)
PRL_INCLUDE_DIR: path of the directory containing "prl.h"
OPENCL_LIB_DIR: path of the directory containing the OpenCL library
OPENCL_INCLUDE_DIR: path of the OpenCL header files
OPENCV_INCLUDE_DIR: path of the OpenCV header files
OPENCV_LIB_DIR: path of the OpenCV library files
LIST_OF_KERNELS: defines a blank separated list of the kernels that
should be compiled or autotuned.
Please look into the file scripts/scripts_config.conf for an example
and for more details.
You can use the default values of the variables except the path variables
which should be set correctly.
- You can use the following script to build the benchmark:
./scripts/compile_and_run_kernels.sh
A log is generated in build/benchmark_building_log.txt
# Building Using CMake :
#########################
To build the benchmark, you need:
- Compile benchmark:
Crate a directory for out-of-source build, run cmake and make:
- CMake can be configured from a GUI window using cmake-gui. (recommended: list all parameters that can be change)
- OpenCL might not be found by the CMake script. In this case, set OPENCL_LIBRARY manually. It is a required parameter for ARM Mali: list BOTH libOpenCL.so and libmali.so.
- Based on your OpenCL device, it is recommended to set up PENCIL_DEFAULT_FLAGS_* variables:
The default values are set to be the common denominator to allow it to work everywhere, but the performance is not optimal.
PENCIL_DEFAULT_FLAGS_BLOCKSIZE should be set to the maximum workgroup sizes. For instance, AMD Radeon cards should use "16,16" (256 threads total)
PENCIL_DEFAULT_FLAGS_LOCAL_MEMORY_SIZE is the amount of used __local memory. Alternatively set to 0 to disable __local memory usage (useful for ARM Mali).
These parameters are reported by the tool clinfo: First is "Max work items[x]:" (individually) and "Max work group size:" (the multiply of values); second is "Max local memory:"
- Optionally, as performance tuning try turning
PENCIL_DEFAULT_FLAGS_MAXFUSE,
PENCIL_DEFAULT_FLAGS_NO_SEPARATE_COMP,
PENCIL_DEFAULT_FLAGS_DISABLE_PRIVATE
flags ON or OFF.
- If something is not installed at the standard paths (/usr, /usr/local), then run cmake-gui and edit the variables (should be straight-forward)
- Set CMAKE_BUILD_TYPE to Release for optimized build, set to Debug to build without optimizations and with debug info.
mkdir pencil-benchmarks-imageproc-build
cd pencil-benchmarks-imageproc-build
cmake -DCMAKE_BUILD_TYPE=Release -DOPENCL_LIBRARY=/path/to/libOpenCL.so -DPENCIL_DEFAULT_FLAGS_BLOCKSIZE="A,B" -DPENCIL_DEFAULT_FLAGS_LOCAL_MEMORY_SIZE=C path/to/pencil-benchmarks-imageproc-repo
make all -j12
The CMake cache is persistent between builds, you only need to supply a parameter when you want to override it.
- (Optional) Tune per-kernel parameters:
We usually need to support separate tile-grid-block sizes for different kernels, and allows access to advanced parameters of the polyhedral compiler.
You can supply these parameters using a corresponding PENCIL_FLAGS_* parameter, separate for every benchmark item. If this is not supplied, the default flags are used.
- Warning: Due to CMake restrictions, some characters (quotes, backslash) needs escaping. Also, the semicolon character cannot be used in these strings.
instead of: --sizes={kernel[i]->tile[8,8];kernel[i]->grid[8,8];kernel[i]->block[8,8]}
use: --sizes=\"{kernel[i]->tile[8,8]}\" --sizes=\"{kernel[i]->grid[8,8]}\" --sizes=\"{kernel[i]->block[8,8]}\"
- Build benchmark:
cd repository_root_path
make all -j12
- Examples of tested hardware parameters (to use with CMake)
AMD Radeon R9 290
-DPENCIL_DEFAULT_FLAGS_BLOCKSIZE="16,16" -DPENCIL_DEFAULT_FLAGS_LOCAL_MEMORY_SIZE=32768 -DPENCIL_DEFAULT_FLAGS_MAXFUSE=ON -DPENCIL_DEFAULT_FLAGS_NO_SEPARATE_COMP=ON -DPENCIL_DEFAULT_FLAGS_DISABLE_PRIVATE=OFF
nVidia Tesla M2050
-DPENCIL_DEFAULT_FLAGS_BLOCKSIZE="32,32" -DPENCIL_DEFAULT_FLAGS_LOCAL_MEMORY_SIZE=49152 -DPENCIL_DEFAULT_FLAGS_MAXFUSE=ON -DPENCIL_DEFAULT_FLAGS_NO_SEPARATE_COMP=ON -DPENCIL_DEFAULT_FLAGS_DISABLE_PRIVATE=OFF
Intel HD Graphics 4000
-DPENCIL_DEFAULT_FLAGS_BLOCKSIZE="32,16" -DPENCIL_DEFAULT_FLAGS_LOCAL_MEMORY_SIZE=65536 -DPENCIL_DEFAULT_FLAGS_MAXFUSE=ON -DPENCIL_DEFAULT_FLAGS_NO_SEPARATE_COMP=ON -DPENCIL_DEFAULT_FLAGS_DISABLE_PRIVATE=OFF
ARM Mali T628
-DPENCIL_DEFAULT_FLAGS_BLOCKSIZE="8,8" -DPENCIL_DEFAULT_FLAGS_LOCAL_MEMORY_SIZE=0 -DPENCIL_DEFAULT_FLAGS_MAXFUSE=ON -DPENCIL_DEFAULT_FLAGS_NO_SEPARATE_COMP=ON -DPENCIL_DEFAULT_FLAGS_DISABLE_PRIVATE=OFF
Please, share the parameters to your device here
# Auto-tuning and Time Measurements
######################################
- You can use the following script to perform auto-tuning for the
PPCG PENCIL compiler.
./scripts/ppcg_tuning.sh
The script generates many PPCG options internaly, and
launches PPCG with each one of these options, then will compile each
file generated by PPCG and will execute it.
The execution times measured for each kernel are reported in
build/output_time.$KERNEL_NAME.csv (a list of all the options explored
by the auto-tuner is provided along with the execution time for each
option).
The best execution times for each kernel are reported in
build/output_time.csv
The best PPCG options obtained through tuning are reported in
the file build/best_optimizations_log.sh
A log is generated in build/benchmark_building_log.txt
- You can use the following script to build the benchmark using
the default PPCG options:
./scripts/compile_and_run_kernels.sh
The script will generate the file build/output_time.csv that
constains the timings for each kernel.
In the generated timings file, the total_execution_time_reference
column indicates the total execution time for the OpenCV calls (this
includes the data copy time and the kernel execution time but does
not include kernel compilation time, the OpenCV kernel is precompiled
by being run once before time measurements are taken).
The total_execution_time_optimized column indicates the total execution
time for the PPCG generated code (this includes the data copy times, the
kernel execution time, the time spent by any host code generated by the
PENCIL compiler but does not include kernel compilation time, kernels are
precompiled also by being run once before taking time measurements).
kernel_only_execution_time_optimized column indicates kernel execution
time measured using OpenCL profiling (does not include data copies and
kernel compilation).
The script uses default PPCG options except for the workgroup sizes
where workgroups of size 16x16 are set instead of the default 32x32
workgroups which may not work on some architectures. Please note that in some cases (e.g. for ARM Mali) you might need to set an even lower value.
- You can also use the previous script to build the kernels with specific
PPCG options. In this case you need to pass a file containing the PPCG
options as an argument.
./scripts/compile_and_run_kernels.sh <ppcg_options_file>
The previous command compiles and runs the kernels using the options
defined in <ppcg_options_file> (for example you can use the file
scripts/ppcg_preset_options/best_optimizations_log.Nvidia_GTX470.sh).
Such a file is obtained in general using autotuning (as described above
the auto-tuning script generates a file containing the best optimization
options that were found during auto-tuning and which you can reuse if you
don't want to redo autotuning).
<ppcg_options_file> provides an option for each kernel listed in
$LIST_OF_KERNELS. The option that corresponds to the i'th kernel
in the $LIST_OF_KERNELS is stored in best_optimization_options[$i].
The folder scripts/ppcg_preset_options/ contains many files with preset
options.
# Running Individual Kernels Manually
#######################################
cd build/
./ppcg_test_KERNEL_NAME <path_to_image>
Details:
- Example images are in images/
- If you use CMake, it will also produce a test_* files.
- These binaries call the original PENCIL code not the code generated by the PENCIL compiler.
- Each executable runs a different operator with all three (C++, OpenCL, PENCIL) implementations.
- Results are cross-checked, and if there is no difference
(within a small allowed precision error), total times are
reported at the end.
# Troubleshooting
##################
- In case of a compilation/execution error, please check the log file in
build/benchmark_building_log.txt
- If you get an execution error while runing a script, you should run
a single kernel to see what is exactly the error message, to run the
resize kernel for example:
cd build/
./ppcg_test_resize ../images/M104_ngc4594_sombrero_galaxy_hi-res.jpg
this will run the resize kernel (OpenCV CPU, OpenCV OpenCL and the code
generated using the PENCIL compiler) on the M104_ngc4594_sombrero_galaxy_hi-res.jpg
image.
- You may get a CL_INVALID_WORK_GROUP_SIZE error, this means that the
workgroup sizes set by the PENCIL compiler are too big for your
architecture.
You can reduce the workgroup sizes for autotuning by removing big
workgroup sizes from the variable POSSIBLE_BLOCK_SIZES
If you are using the compile_and_run_kernels.sh script, you can set
the options of the PENCIL compiler by creating a file similar to
the file scripts/ppcg_preset_options/ppcg_default_options.sh