[Yum-devel] [HOWTO]: profiling with cProfile and kcachegrind

[Terje Rosten]

Quick howto to do profiling of yum with cProfile and
kcachegrind. (Might be added to Wiki?)

First install some software (kcachegrind and dot (from graphviz))

$ yum -y install /usr/bin/kcachegrind /usr/bin/dot

Then get lsprofcalltree

$ mkdir ~/py-2.5
$ cd ~/py-2.5
$ wget http://www.gnome.org/~johan/lsprofcalltree.py

We need some stuff from Python-2.5 (if not working in rawhide or
distro with fresh Python bits).

  Install buildrequires:

  $ yum install readline-devel libtermcap-devel ncurses-devel gdbm-devel \
              expat-devel tcl-devel tk-devel tix-devel bzip2-devel \
              sqlite-devel db4-devel

  Download, unpack, build and install Python-2.5:

  $ wget http://www.python.org/ftp/python/2.5/Python-2.5.tar.bz2
  $ tar xjvf Python-2.5.tar.bz2
  $ cd Python-2.5
  $ ./configure --prefix=/dev/shm/py-2.5 \
           --enable-ipv6 --enable-unicode=ucs4 --enable-shared
  $ make && make install

  Make links in our python dir:

  $ cd ~/py-2.5
  $ ln -s /dev/shm/py-2.5/lib/python2.5/lib-dynload/_lsprof.so
  $ ln -s /dev/shm/py-2.5/lib/python2.5/cProfile.py
  $ ln -s /dev/shm/py-2.5/lib/python2.5/pstats.py
  $ ln -s /dev/shm/py-2.5/lib/libpython2.5.so.1.0

Now we are ready for the profiling stuff, get a yum release:

$ wget http://linux.duke.edu/yum/download/3.0/yum-3.0.2.tar.gz
$ tar xzvf yum-3.0.2.tar.gz
$ cd yum-3.0.2

Then apply the following yum profiling patch:
(Seth: apply to CVS if you want)

diff -u -r1.103 yummain.py
--- yummain.py	19 Dec 2006 03:43:31 -0000	1.103
+++ yummain.py	3 Jan 2007 18:11:20 -0000
@@ -25,6 +25,7 @@
 from yum import logginglevels
 import cli
 
+PROFILING = False
 
 def main(args):
     """This does all the real work"""
@@ -181,14 +182,22 @@
 
     verbose_logger.log(logginglevels.INFO_2, 'Complete!')
     unlock()
-    sys.exit(0)
+    if not PROFILING:
+        sys.exit(0)
 
     
 if __name__ == "__main__":
-    #import hotshot
-    #p = hotshot.Profile(os.path.expanduser("~/yum.prof"))
-    #p.run('main(sys.argv[1:])')
-    #p.close()    
+    if PROFILING:
+        import os
+        import cProfile
+        import lsprofcalltree
+        p = cProfile.Profile()
+        p.run('main(sys.argv[1:])')
+        k = lsprofcalltree.KCacheGrind(p)
+        data = open(os.path.expanduser('~/yum-prof.kgrind'), 'w+')
+        k.output(data)
+        data.close()
+        sys.exit(0)
     try:
         main(sys.argv[1:])
     except KeyboardInterrupt, e:

Then set PROFILING to True in yummain.py.

To generate data, try e.g.

$ rpm -e xpdf
$ LD_LIBRARY_PATH=~/py-2.5  PYTHONPATH=~/py-2.5  python yummain.py -y install xpdf

Start kcachegrind and watch the data in it's full glory:

$ cd
$ kcachegrind yum-prof.kgrind

( Before doing a new run, move ~/yum-prof.kgrind )


 - Terje