Integration with HPCToolkit
adayton1 opened this issue · 11 comments
Are there bindings to export Caliper annotations to HPCToolkit? If not, it would be very helpful to have (and my guess is that it would be fairly simple to add).
@daboehme, is there any tutorial on how to add a service? I've looked at the following pages, and so far all have been too abstract or just a list of APIs rather than a tutorial to actually aid in implementing a service:
http://software.llnl.gov/Caliper/CaliperBasics.html#third-party-tool-support-nvidia-nsight-intel-vtune
http://software.llnl.gov/Caliper/ThirdPartyTools.html
http://software.llnl.gov/Caliper/workflow.html
http://software.llnl.gov/Caliper/services.html
http://software.llnl.gov/Caliper/DeveloperGuide.html
The hpctoolkit interface is really simple:
hpctoolkit_sampling_start();
hpctoolkit_sampling_end();
hpctoolkit_sampling_is_active();
Basically, I'm looking to add a service that will work with region filtering so that hpctoolkit will only profile the regions I care about. http://software.llnl.gov/Caliper/RegionFiltering.html
Even if you could point me to a service that I could start from, that would be helpful. There's just so many and they don't all implement the same interface, so it's hard to know where to start from.
Hi @adayton1, I don't have a good tutorial here, but for this type of service you can look at the nvtx, roctx, and vtune services. Essentially you'll derive from the AnnotationBinding class and overwrite the on_begin, on_end, and service_tag methods. Shouldn't be too difficult. The AnnotationBinding class takes care of the filtering. I'm happy to help if you run into any issues.
@daboehme, can you explain the difference between a properly nested attribute and a non-nested attribute?
https://github.com/LLNL/Caliper/blob/master/src/services/nvtx/Nvtx.cpp#L122
@daboehme, can you explain the difference between a properly nested attribute and a non-nested attribute?
Hi @adayton1 , the nested attributes form the "main" region hierarchy. The regions created by the Caliper annotation macros all fall into this category. You can create additional non-nested attributes that can overlap the main hierarchy, but it's a relatively rarely used feature. By default the AnnotationBinding class only forwards nested attributes.
Is a service class a singleton? Do I have to worry about thread safety in on_begin and on_end?
Also, is any error checking done behind the scenes (e.g. CALI_MARK_END("meh") where there was no CALI_MARK_BEGIN("meh"))?
Hi @adayton1, the service class gets one object for each Caliper channel where it's active, but there is typically only one. In fact it doesn't make sense to have a binding service active multiple times. The on_begin/on_end functions can be invoked from multiple threads depending on how the target code is instrumented, so yes a good implementation should make sure to be thread safe. And yes, Caliper does check if the regions match with the CALI_MARK_BEGIN/END macros and error out if they don't.
I'm not sure the AnnotationBinding class is going to work. NVTX and RocTX have a stack like API. HPCToolkit does not - it just has start and stop. So on an exclude region, I would need to call stop on the begin marker and potentially start on the end marker if an outer level is supposed to be timed. But if I understand correctly, on_begin and on_end for my service won't even be called for an excluded region. Or am I misunderstanding something?
Correct, on_begin and on_end won't be called for excluded regions. But you're right, you'd have to make sure to just start/stop once at the correct stack level.