Great overview of Python profiling
I'm very interested in Python optimization and profiling (see Quick profiling in Python and Profiling Python code with cProfile and pstats). So I'm always on the look out for a good discussion on these topics.
Luckily I ran across this article by Huy Nguyen. It covers some great profiling tools I've discussed before such as line_profiler as well as some topics like memory_profiler which I have been meaning to discuss on this blog.
Also, the tool for tracking down memory leaks, objgraph, is something I've never seen before and will be investigating further.
The one key takeaway from this article is a little insight that's obvious but easily forgotten. It's related to the Unix utility time and the 3 'types' of 'time returned (emphasis is my own):
- real - refers to the actual elasped time
- user - refers to the amount of cpu time spent outside of kernel
- sys -refers to the amount of cpu time spent inside kernel specific functions
You can get a sense of how many cpu cycles your program used up regardless of other programs running on the system by adding together the sys and user times.
If the sum of sys and user times is much less than real time, then you can guess that most your program’s performance issues are most likely related to IO waits.
Published: 02-22-2013 14:48:54