PEBIL (pebil) - README
2.2 Binary Instrumentation Packages for Gathering Computational Trace
The PMaC Prediction Framework relies on binary instrumentation to gather information about the computational work done during an application run. The information in a computational trace is the floating point operation count, memory operation count, and simulated cache hit rates (stored per basic block for each level of the caches in the memory hierarchy) of the target machine(s). Each basic block in an application is given a unique ID that is assigned to it in a consistent manner during each instrumentation pass. The unique ID for every basic block in the application is consistent as long as the executable is kept the same . The simulated cache hit rates are computed on-the-fly during the trace by inputing the address stream of the instrumented running application into a cache simulator. The specifications for the cache or caches to be simulated are specified in the CacheStructures.h and describe in more detail in section 2.2.1.3.
To gather computation traces, PMaC provides two instrumentation tools. PEBIL is a binary editing tool for X86/Linux systems (described in this document) and PMaCInstrumentor is a binary editing tool for IBM Power/AIX systems.
2.2.1 PEBIL
PEBIL is a binary instrumentation toolkit that operates on ELF binaries on Linux for x86/x86_64 processors. PEBIL has a C++ API that provides the means to inject code and data into a binary file. It also provides several pre-implemented tools for basic block counting and cache simulation for a set of memory hierarchies.
2.2.1.1 Installation for Application Tracing
Host Systems: Any X86/Linux
The source code for the PEBIL library and instrumentation tools is available at through PMaC Laboratories by e-mailing help@pmaclabs.com.
The directory structure of distribution is as follows:
$ cd /home/userid/tools/PEBIL
$ ls
bin/ include/ instcode/ Makefile scripts/ testapps/ tools/ docs/ etc/ lib/ src/
The include, src, tools, and instcode directories contain the source code for instrumentation related libraries and instrumentation tools. The bin and lib directories contain the executables and shared libraries after building the distribution. The scripts directory contains additional PERL scripts to assist application tracing for high level instrumentation activities.
After retrieving the source, users can change to the top directory of the distribution and follow the instructions in the INSTALL to build the distribution. After building the executables and shared libraries, it is also advised to link or copy the scripts in the scripts directory to the bin directory of the distribution so that the scripts will be found in user's path as well. The following is an example of this procedure:
$ cd /home/userid/tools/PEBIL
$ gmake clean
$ gmake
$ cd bin
$ ln -s ../scripts/selectSimBlocks.pl selectSimBlocks.pl
$ ln -s ../scripts/traceHelper.pl traceHelper.pl
The Makefile at the top directory /home/userid/tools/PEBIL iterate over sub directories and run the appropriate make commands. Alternatively, users can follow the same steps manually. Instead of gmake at the top directory, the users can run the following. Note that the order of making src should come before the order of making tools directory. For an installed distribution to work, minimal set of steps is to make src, tools and instcode directories.
$ cd /home/userid/tools/PEBIL
$ cd ./src
$ gmake
$ cd ../tools
$ gmake
$ cd ../instcode
$ gmake
Manually following the make steps is particularly useful when the default configuration (configuration that comes with the source code) is not used for building the instcode directory. There are various ways of building the code under this directory and the available compilation macros are described in details in Section 2.2.1.2.
To use PEBIL, users need to include the bin directory of distribution to their path. Thus, if a particular distribution needs to be used, the user needs to set the path variables accordingly before using that distribution as the name of the executable and additional PERL scripts would be the same. When installed properly, the PEBIL executable, pebil, should be in user's path. It can be checked using the �which� command of Unix from a different directory (you may need to source .cshrc/.tcshrc or .profile/.bashrc files for paths to be adjusted). The --help option will print a brief usage information.
$ source ~/.cshrc
$ which pebil
/home/userid/tools/PEBIL/bin/pebil
$ pebil --help
usage : pebil
--typ (ide|dat|bbt|cnt|jbb|sim|csc)
--app <executable_path>
--inp <block_unique_ids> <-- valid for sim/csc
[--lib <shared_lib_topdir>]
[--ext <output_suffix>]
[--dtl]
[--lpi] <-- valid for sim/csc
[--phs <phase_no>] <-- valid for sim/csc
[--dfp <pattern_file>] <-- valid for sim/csc
[--help]
Brief Descriptions for Options:
===============================
--typ : required for all. Instrumentation type.
--app : required for all. Executable path.
--lib : optional for all. Instrumentation shared library top
directory. default is $PEBIL_LIB
--ext : optional for all. File extension for output files.
default is (typ)inst, such as jbbinst for type jbb.
--dtl : optional for all. Enable detailed .static file with line numbers
and filenames. default is no details.
--inp : required for sim/csc. File including list of block ids to instrument
--lpi : optional for sim/csc. Loop level block inclusion for
cache simulation. default is no.
--phs : optional for sim/csc. Phase number. defaults to no phase,
otherwise, .phase.N. is included in output file names
--dfp : optional for sim/csc. dfpattern file. defaults to no dfpattern file,
2.2.1.2 Installation for MultiMAPS Tracing
Host Systems: Any X86/Linux
In the framework, the MultiMAPS data for a target system needs to be augmented with the cooresponding hit rates. To accomplish this, MultiMAPS needs to be traced with the target system�s cache structure. The default PEBIL installation, which comes configured for application tracing, uses address stream sampling for cache simulation. Rather than using every address in the address stream of an application run, it uses only a subset of addresses in the address stream and simulates them for the target memory subsystems. This method of sampling the address stream has been used to reduce the overhead of instrumentation. For MultiMAPS tracing, however, the entire address stream needs to be consumed for cache simulations, therefore sampling should not be used. Hence, separate installations of PEBIL are needed for tracing MultiMAPS for cache hit rates.
To install PEBIL for MultiMAPS tracing, you need to edit the instcode/Makefile file under the PEBIL installation directory (a different directory than the one installed for application tracing) and repeat the installation process described in Section 2.2.1.1 for this new directory. To install pebil for MultiMAPS tracing, user needs to replace the EXTENDED_SAMPLING compilation macro in instcode/Makefile with the NO_SAMPLING_MODE compilation macro. Assuming the source code for PEBIL for MultiMAPS tracing is under /home/userid/mmaps user needs to;
$ cd /home/userid/mmaps/PEBIL
$ vi instcode/Makefile
### change extended sampling mode to no sampling mode
#EXTRA_CFLGS = -DEXTENDED_SAMPLING -DPER_SET_RECENT -DVICTIM_CACHE
EXTRA_CFLGS = -DNO_SAMPLING_MODE -DPER_SET_RECENT -DVICTIM_CACHE
$ gmake
2.2.1.3 Using a Different Set of Memory Hierarchies (Caches)
The PEBIL source is distributed with a set of memory hierarchies to be used in cache simulation runs. However, it also is flexible enough to easily allow a different set of memory hierarchies with the installation. PEBIL provides a PERL script under the scripts directory of the distribution, scripts/generateCaches.pl to do this.
This script takes an input file of memory hierarchy specifications for a set of systems and generates a C header file that contains these same specifications as C code. The user can also specify the stride system to use from one of the caches given in the input file if stride information will be collected. Otherwise, the default action of the script is to add a 2MB direct cache that is used for gathering stride information. Information on how to use the generateCaches.pl script can be seen by using the --help flag. The following example shows the --help flag:
$ /home/userid /PEBIL/scripts/generateCaches.pl --help
usage : scripts/generateCaches.pl
[--sys-file <cache_desc_file>] <-- defaults to CacheDescriptions.txt
[--out-file <outfile>] <-- defaults to CacheStructures.h
[--stride-sys <system idx>] <-- defaults to 0, 2 MB direct cache
[--nostd] ? static allocation of cache structures and use of safe lib functions
[--puniq] ? print unique cache information.
[--help]
[--readme]
A sample input to generateCaches.pl script is given in the text box below. Everything after �#� sign is assumed to be a comment and is ignored. Each line in the input file defines a memory hierarchy by listing the sysid, number of cache levels and the specifications of each cache. The sysid needs to be greater than 0 and unique. It is used by the prediction database to deferentiate different systems and cache structures. Each cache is defined by 4 attributes: the cache size (which can be given in bytes or in KB or MBs), the associativity, the line size in bytes and the replacement policy. The replacement policy can be lru, lru_vc, dir or ran where lru is the LRU pseudo implementation, lru_vc is the LRU pseudo implementation for victim caches, dir is the direct addressed and ran is a specific random replacement.
Note that each cache specification in the memory hierarchy description file needs to be per computation unit (core or processors). For instance, if L1 is private but L2 is shared between cores or processors, the L2 specification needs to be given per core/processor. Since there is no easy way of dividing the shared caches per computation unit, the simplest way is to divide caches evenly among the sharing units.
The output of this script is a C header file that will be compiled into the shared libraries under the instcode directory for the cache simulator rewriting tools. So if the user wants to use different caches structures or memory hierarchies for application and MultiMAPS tracing than the set of hierarchies distributed with the source (very likely), then before installing PEBIL as described in Sections 2.2.1.1 and 2.2.1.2 they need to create memory hierarchy specifications and generate the C header file for those specifications using scripts/generateCaches.pl.
For instance if the user wants to install new cache structures listed in a file named MyCaches.txt, they would perform the following steps
$ cd /home/userid /PEBIL/instcode
$ vi MyCaches,txt
### add your cache specifications to this file such as
# id level size assoc line repl size assoc line repl size assoc line repl
1 2 256KB 8 128 lru 4MB 8 128 lru_vc
$ ../scripts/generateCaches.pl --sys-file ./MyCaches.txt
Processing input file ---- MyCaches.txt
Output file will be ---- CacheStructures.h
*** DONE *** SUCCESS *** SUCCESS *** SUCCESS *****************
$ gmake clean
$ gmake
Note that this script generates a file named CacheStructures.h by default since CacheStructures.h is the name of the file used in the source code of the instrumentation libraries. If you decide to use some other file as output, please copy that file to CacheStructures.h in the instcode directory before building the code in the instcode directory again. More importantly, since cache structures are embedded into the PEBIL installation as source code, for each different set of cache structures, a new installation of PEBIL is required.
2.4 Tracing with PEBIL or PMaCInstrumentor
The computational traces for an application run are gathered with the aid of binary instrumentation. During application tracing, a trace file for each task in the application is gathered. Such a trace file includes cache hit rates across the specified set of cache structures for each basic block that was executed by that task during the application run. Since it is not practical to perform cache simulation on all basic blocks in the application (there might be hundreds of thousands), the computational traces are collected in two steps. During the first step, the executable is instrumented to count the number executions for each basic block (JBB Tracing). During the second step, only the most frequently executed basic blocks are chosen to be instrumented with cache simulation code (Cache Simulation). This means that to gather the computational traces for an application, it needs to be run twice using two different instrumented binaries. For the complete computational trace, all of the trace files generated during these two steps need to be used by the PMaC Prediction Framework.
The computational traces for an application are gathered by instrumenting the application binary using either PMaCInstrumentor or PEBIL. To help and ease the gathering of computational traces, these tools each provide a PERL script called traceHelper.pl under the scripts directory (and under bin directory also if the installation steps are followed). This script is designed to guide the user to instrument executables and to collect the computational traces that are used to generate performance predictions. The following two subsections describe the steps of computational trace collection in details.
For the remainder of this section we will assume that pmacInst is installed at /site/pmac_tools/Instrumentor, the application to be instrumented is compiled at /home/userid/app and the name of the executable is app.exe that is under the same directory, and the executable is run from the /home/userid/run directory.
For pmacinst or pebil to be used for computation tracing, it inserts calls to shared libraries from the lib directory of the installation into the executable to dump the traces to the disk in addition to inserting the cache simulation code. The traceHelper.pl script allows the user to specify the path to those shared libraries using --inst_lib option every time it is executed . The value for the --inst_lib option should point to the top directory of PMaCInstrumentor/PEBIL installation, not the lib directory under the installation.
2.4.1 JBB Tracing
JBB Tracing instruments the executable to insert a counter at every basic block. Thus during the execution of the instrumented executable, it counts the number of times each basic block is executed. For JBB tracing, user needs to instrument the executable using traceHelper.pl and run the executable in the same fashion as original. To instrument the executable for JBB tracing, the user can run the following:
$ /site/pmac_tools/Instrumentor/scripts/traceHelper.pl
--action jbbinst
--application app --dataset standard --cpu_count 64
--exec_file /home/userid/app/app.exe
--pmacinst_dir $HOME/mytraces
--inst_lib /site/pmac_tools/PMaCInstrumentor
If successful, the traceHelper.pl script will create a directory, named pmacTRACE, under $HOME/mytraces. All tracing-related files that are generated will be copied to this directory. Under this directory, using the action type given by --action option, traceHelper.pl will create subdirectories and save the tracing related files including the instrumented executable and the static files that are needed later for post-processing the traces. Here is an example layout of the directories after this step.
$ ls $HOME/mytraces/pmacTRACE/
jbbinst
$ ls $HOME/mytraces/pmacTRACE/jbbinst/
app_standard_0064
$ ls $HOME/mytraces/pmacTRACE/jbbinst/app_standard_0064/
App.exe.jbbinst app.exe.jbbinst.static
Note that the --exec_file option points to the path to the executable file and the --pmacinst_dir option points to the top directory where all traces will be collected.
The app.exe.jbbinst file under $HOME/mytraces/pmacTRACE/jbbinst/app_standard_0064/ directory is the instrumented executable and user needs to run this executable the same as the original to generate basic block execution counts. If run successfully, it will generate trace files in the run directory as follows (assuming the run directory is /home/userid/run). The number of trace files generated should match the number of tasks in the application run. Furthermore, the user should expect the execution of the jbb-instrumented application to take a factor of 1.5-2.0 times longer than the original application run.
$ ls /home/userid/run/*.meta*.jbbinst
app.exe.meta_0000.jbbinst
app.exe.meta_0001.jbbinst
.............................................
App.exe.meta_0064.jbbinst
If the trace files do not include the rank id as part of name as above (0000,0001, �etc.) but include some other non-consecutive numbers (process IDs), it indicates that PMaC timers are not linked in to the executable properly. Make sure the times are inserted properly. Similarly, if the instrumented run does not generate the trace files, that indicates the termination of run was not recognized by the instrumentation code and the timers need to be included. In both cases, the RankPid file in the run directory is not expected to exist, indicating that the timers are not included properly.
The user needs to collect these trace files under the mytraces directory by running the following command.
$ /site/pmac_tools/Instrumentor/scripts/traceHelper.pl
--action jbbcoll
--application app --dataset standard --cpu_count 64
--exec_name app.exe
--jbb_trc_dir /home/userid/run
--pmacinst_dir $HOME/mytraces
If the script runs successfully, it will generate a subdirectory under $HOME/mytraces/pmacTRACE with the action name. Under that directory, there will be an application-specific directory to where the script will copy the trace files.
$ ls $HOME/mytraces/pmacTRACE/
jbbcoll jbbinst
$ ls $HOME/mytraces/pmacTRACE/jbbcoll/
app_standard_0064
$ ls $HOME/mytraces/pmacTRACE/jbbcoll/app_standard_0064/
app.exe.meta_0000.jbbinst
app.exe.meta_0001.jbbinst
......................................
app.exe.meta_0064.jbbinst
Note that rather than a path to the executable, this command takes the executable name using the --exec_name option. Note also that the --jbb_trc_dir points to the directory where the *.meta*.jbbinst trace files are located.
2.4.2 Cache Simulation
The JBB tracing generates files that contain basic block execution counts. Since it is not practical to instrument all basic blocks for cache simulation, PMaC framework chooses a subset of the most frequently executed basic blocks that cover only a certain percentile of all basic block execution counts. The user does not need to know anything about how they are chosen as the traceHelper.pl script automatically calls the selectSimBlocks.pl script to choose the blocks to instrument for cache simulation. The next steps for computational trace collection is to instrument the executable for cache simulation, run the instrumented executable and then collect the cache simulation traces generated.
The user needs to ensure that the following are true before continuing with this step; First, the selectSimBlocks.pl script is his path (please refer to Section 2.1.1.1). Second, the target cache structures and memory hierarchies are installed for cache simulation rather than the sample structures shipped by default with the distribution (please refer to Section 2.1.1.3).
First, the user needs to instrument the executable for cache simulation. This can be done by running the following command.
$ /site/pmac_tools/Instrumentor/scripts/traceHelper.pl
--action siminst
--application app --dataset standard --cpu_count 64
--exec_file /home/userid/app/app.exe
--jbb_trc_dir $HOME/mytraces/pmacTRACE/jbbcoll/app_standard_0064/
--jbb_static $HOME/mytraces/pmacTRACE/jbbinst/app_standard_0064/app.exe.jbbinst.static
--phase_no 1 --phase_count 1
--pmacinst_dir $HOME/mytraces
--inst_lib /site/pmac_tools/Instrumentor
If successful, this command will instrument the executable with cache simulation code and save the generated files under the $HOME/mytraces directory as follows.
$ ls $HOME/mytraces/pmacTRACE/
jbbcoll jbbinst siminst
$ ls $HOME/mytraces/pmacTRACE/siminst/
app_standard_0064
$ ls $HOME/mytraces/pmacTRACE/siminst/app_standard_0064/
p01
$ ls $HOME/mytraces/pmacTRACE/siminst/app_standard_0064/p01/
app.phase.1o1.0064.jbbinst.lbb
app.exe.phase.1.0064.siminst
app.exe.phase.1.0064.siminst.static
The app.exe.phase.1.0064.siminst file is the instrumented executable for cache simulations. The user needs to run this executable the same way as the original executable to generate traces for cache simulation. Note that unlike JBB tracing, to instrument the executable for cache simulation, we also passed number of phases (using the --phase_count option) and phase number (using the --phase_no option) to the helper script. This is due to fact that cache simulation is computationally expensive and may introduce slowdowns of 6 to 10-fold during the run of the instrumented executable. In the HPC resource being used to run the instrumented executable, the upper limit in execution imposed by the system may not be large enough to complete tracing of the application in full. So users can divide the cache simulation into multiple phases, instrument the executable to perform cache simulation for only those blocks selected for a particular phase, then run each phase's instrumented executable. Note that the pmacTRACE directory structures uses the phase number for saving the cache simulation files generated (but most of the tracing we have conducted up to now has required only 1 phase).
If the user runs the instrumented executable app.exe.phase.1.0064.siminst in a manner similar to the original executable and if the run completes successfully, it should generate trace files as follows. The number of trace files should match the number of tasks in the application run.
$ ls /home/userid/run/*phase*.meta*.siminst
app.exe.phase.1.meta_0000.0064.siminst
app.exe.phase.1.meta_0001.0064.siminst
...................................................................
app.exe.phase.1.meta_0063.0064.siminst
If the trace files do not include the rank id as part of name as above (0000,0001, and son on) but include some other non-consecutive numbers (process IDs), it indicates that PMaC timers are not linked in to the executable properly as described in Section4.1 . Make sure the times are inserted properly. Similarly, if the instrumented run does not generate the trace files, that indicates the termination of run was not recognized by the instrumentation code and the timers need to be included. In both cases, the RankPid file in the run directory is not expected to exist, indicating that the timers are not included properly.
After running the instrumented executable, the user needs to collect these traces using the simcoll action (which is functionally similar to what the jbbcoll action does for jbb tracing).
$ /site/pmac_tools/Instrumentor/scripts/traceHelper.pl
--action simcoll
--application app --dataset standard --cpu_count 64
--exec_name app.exe
--sim_trc_dir /home/userid/run
--pmacinst_dir $HOME/mytraces
--phase_no 1
Note that the --phase_no option is passed to the helper script to indicate which phase's traces are being collected under the $HOME/mytraces/pmacTRACE directory. If ran successfully, the trace files will be copied and the directory structure will be.
$ ls $HOME/mytraces/pmacTRACE/
jbbcoll jbbinst simcoll siminst
$ ls $HOME/mytraces/pmacTRACE/simcoll/
app_standard_0064
$ ls $HOME/mytraces/pmacTRACE/simcoll/app_standard_0064/
p01
$ ls $HOME/mytraces/pmacTRACE/simcoll/app_standard_0064/p01/
app.exe.phase.1.meta_0000.0064.siminst
app.exe.phase.1.meta_0001.0064.siminst
......................................................................
app.exe.phase.1.meta_0063.0064.siminst
After this step, all traces for the application for a particular dataset, size and CPU count should be complete. For each application or dataset/size/CPU count that will be traced for performance prediction, similar steps are required. The same directory for the --pmacinst_dir option can be used for the other applications, datasets, sizes and CPU counts to collect all trace files conveniently under one roof even though this is not a requirement.