Energy profiling tools: NVML-based measurement tool

[ethan-puyaubreau]

This PR adds NVML-based GPU energy monitoring to the Kokkos Energy Profiler:

Features:

• Integrated Energy Profiling: Combines timing measurements with GPU power monitoring in a single profiler library
• NVML Provider: NVMLProvider class for GPU power monitoring with device discovery and power usage APIs
• Background Power Sampling: PowerSampler class using daemon-based background monitoring for continuous power data collection
• Multi-GPU Support: Individual device queries for systems with multiple GPUs

Implementation:

• Main profiler library: kp_energy_profiler (integrates timing + power monitoring)
• NVML provider: nvml_provider.cpp/hpp for GPU power monitoring
• Power sampler: power_sampler.cpp/hpp for background sampling with daemon
• Timing utilities: Enhanced timing infrastructure with power data export

Dependencies:

• Requires CUDA 12.6+ with NVML support
• Depends on: Energy profiling tools: Add Daemon class for periodic task execution #300 and Energy profiling tools: Core infrastructure with timing tool and export capabilities #299

Usage:
The profiler automatically detects available GPUs and begins power monitoring during initialization. Power data is collected in the background and exported alongside timing data in CSV format.

[romintomasetti]

Have you seen https://docs.nvidia.com/nsight-systems/UserGuide/index.html#nvml-power-and-temperature-metrics-preview ?

It would be great to actually better document this tool and compare it to what nsigh-systems may provide. Does your tool make anything special to help correlate Kokkos regions with consumption, trigger anything special ? Or is it "just" launching a thread that samples the energy consumption ?

[romintomasetti]

Also, what about asynchronicity ? I mean:

{
    Kokkos::Profiling::ScopeRegion region("my region");

    Kokkos::parallel_for(Kokkos::RangePolicy(exec, 0, N), ...); // async
}

If the tool reports the consumption of the my region region, if you're not Kokkos::fenceing, what is the meaning of the measurements reported by Kokkos Tools, especially when the kernel is actually running after the scoped region ends ?

[ethan-puyaubreau]

Hello! I've seen this page one time but since I focused more on Variorum support at some point I haven't had the chance of reading it further. As of now, the tool is only launching a thread that samples the energy consumption.

[ethan-puyaubreau]

Also, what about asynchronicity ? I mean:

{
    Kokkos::Profiling::ScopeRegion region("my region");

    Kokkos::parallel_for(Kokkos::RangePolicy(exec, 0, N), ...); // async
}

If the tool reports the consumption of the my region region, if you're not Kokkos::fenceing, what is the meaning of the measurements reported by Kokkos Tools, especially when the kernel is actually running after the scoped region ends ?

For now, the system doesn't do anything special to correlate power consumption with a specific region or kernel. The timing system is meant to help visualize what the current situation is, so there's room for improvement (meaning adding more correlation using multiple metrics), especially since the daemon system would still allow for method/data propagation

[JBludau]

@romintomasetti we will check if Nsight does actually do something useful. It at least sounds like the exact same thing we are doing.
About the fencing: Nvidia at the moment allows power measurements every 100ms. But it seems to return only the power average of the last 25ms of that window, see http://arxiv.org/abs/2312.02741. Thus the tool will at the current state of Nvidas tools only be useful for measuring entire regions and even then it will need repetitions with shifts and post processing in order to get anything that is relatable to an algorithm. Due to these problems the tool currently does not require fencing, this should be done by the user.

[maartenarnst]

Could be good to add the AMD tool to the mix:

• https://github.com/ROCm/rocprofiler-systems?tab=readme-ov-file#gpu-metrics

[JBludau]

on our ToDo list :-)

[vlkale]

@JBludau Do you still have plans to look at the AMD Profiling metrics, as mentioned above? Do you have updates you can share - particularly those that are pertinent to this PR - from your end?

Thanks!

[JBludau]

This pr has been split into smaller ones to make it easier to review. Once we have these in, we can think about adding something for amd

[romintomasetti]

I would not merge this PR without an example.

And I would be strongly against merging this one before we get proper Cuda testing, e.g. #271.