Python Profiling - Function profiling with cProfile
The python cProfile builtin function profiler works with the bwpy module. To use it with your script, it can be as simple as:
$ aprun -n 1 python -m cProfile <myscript.py>
Optionally add the -o <bw.profile> and your bw.profile may be analyzed with any of the python profile viewers you may install in a virtualenv or via the python pstats module:
$ python
>>> import pstats
>>> p = pstats.Stats('bw.profile')
>>> p.sort_stats('cumulative').print_stats(5)
Wed Apr 18 10:21:57 2018 bw.profile
610602 function calls (605262 primitive calls) in 913.990 seconds
Ordered by: cumulative time
List reduced from 2296 to 5 due to restriction <5>
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.062 0.062 913.991 913.991 ep_task_errors_bw.py:8()
1 0.000 0.000 908.962 908.962 ep_task_errors_bw.py:208(main)
3 900.292 300.097 900.292 300.097 {time.sleep}
3 0.166 0.055 7.414 2.471 ep_task_errors_bw.py:104(my_endpoint_manager_task_list)
5825 0.004 0.000 6.978 0.001 {next}
>>>
Python Profiling - line level profiling with line_profiler
You may setup the line_profiler in a virtualenv as shown and use it to profile functions you decorate with @profile .
arnoldg@h2ologin2:~> module load bwpy
arnoldg@h2ologin2:~> virtualenv line_profiler
Using base prefix '/mnt/bwpy/single/usr'
New python executable in /mnt/a/u/staff/arnoldg/line_profiler/bin/python3.5
Also creating executable in /mnt/a/u/staff/arnoldg/line_profiler/bin/python
Installing setuptools, pip, wheel...done.
arnoldg@h2ologin2:~> source line_profiler/bin/activate
(line_profiler) arnoldg@h2ologin2:~> pip install line_profiler
Collecting line_profiler
Downloading https://files.pythonhosted.org/packages/14/fc/ecf4e238bb601ff829068e5a72cd1bd67b0ee0ae379db172eb6a0779c6b6/line_profiler-2.1.2.tar.gz (83kB)
100% |################################| 92kB 3.9MB/s
...
(line_profiler) arnoldg@h2ologin2:~> kernprof
Usage: kernprof [-s setupfile] [-o output_file_path] scriptfile [arg] ...
# switching into a batch job here...
(line_profiler) arnoldg@h2ologin2:~> grep --after-context=5 @profile ep_task_errors_bw.py
@profile
def my_endpoint_manager_task_list(tclient, endpoint):
"""
Get tasks from an endpoint, then look through them for error events.
Also mark as SRC, DEST, or SRC_DEST as the case may be.
"""
(line_profiler) arnoldg@h2ologin2:~> aprun -n 1 kernprof -l ep_task_errors_bw.py
...
(line_profiler) arnoldg@h2ologin2:~> python -m line_profiler ep_task_errors_bw.py.lprof
Total time: 7.17466 s
File: ep_task_errors_bw.py
Function: my_endpoint_manager_task_list at line 103
Line # Hits Time Per Hit % Time Line Contents
==============================================================
103 @profile
104 def my_endpoint_manager_task_list(tclient, endpoint):
105 """
106 Get tasks from an endpoint, then look through them for error events.
107 Also mark as SRC, DEST, or SRC_DEST as the case may be.
108 """
109 3 20.0 6.7 0.0 source_total_files = 0
110 3 13.0 4.3 0.0 dest_total_files = 0
111 3 9.0 3.0 0.0 source_total_bps = 0
112 3 8.0 2.7 0.0 dest_total_bps = 0
113 3 8.0 2.7 0.0 source_total_tasks = 0
114 3 8.0 2.7 0.0 dest_total_tasks = 0
115
116 3 17.0 5.7 0.0 for task in tclient.endpoint_manager_task_list(filter_endpoint=endpoint,
117 3 9.0 3.0 0.0 filter_status="ACTIVE",
118 44 1631294.0 37074.9 22.7 num_results=None):
119 41 431.0 10.5 0.0 if task["destination_endpoint_id"] == endpoint:
120 3 9.0 3.0 0.0 endpoint_is = "DEST"
121 3 26.0 8.7 0.0 dest_total_files += task["files"]
122 3 26.0 8.7 0.0 dest_total_bps += task["effective_bytes_per_second"]
123 3 10.0 3.3 0.0 dest_total_tasks += 1
References:
cProfile: https://docs.python.org/3/library/profile.html
line_profiler: https://github.com/rkern/line_profiler/blob/master/README.rst
Python timers: https://docs.python.org/3/library/time.html#time.perf_counter
