Profiling Python with valgrind

Posted on October 11, 2013 by Thomas Cokelaer

1. Under Linux, install valgrind as follows (Fedora):

sudo yum install valgrind kcachegrind

and pyprof2calltree:

pip install pyprof2calltree

2. create a Python script (let us call it run.py)
3. Create the file containing the profile information using valgrind/callgrind

python -m cProfile  -o profile_data.pyprof run.py
pyprof2calltree -i profile_data.pyprof  # this converts the stats into a callgrind format

4. Visualisation

kcachegrind profile_data.pyprof.log

Alternatively, for visualisation, you can also use gprof2dot and graphviz (dot):

gprof2dot.py --format=callgrind --output=out.dot profile_data.pyprof.log
dot -Tsvg out.dot -o graph.svg
Posted in Python | Tagged , | 1 Comment

svn: can’t upgrade

Posted on September 10, 2013 by Thomas Cokelaer

I’ve recently update my distribution and copied back an old SVN repo into the new distributio. This led to issues with respect to different SVN version. The old one used SVN 1.6 and the new one is 1.7
In general, the command:

svn upgrade

works well but then I got this error message:

svn: E155019: Can't upgrade '....../spectrum/trunk' as it is not a pre-1.7 working copy root, the root is 'spectrum'

The solution is written in the error message: you are not in the root directory of the SVN repository. You must upgrade the entire repository not only a sub directory. So, just type (in this example):

cd ..
svn upgrade
Posted in Computer Science, Linux | Tagged | 1 Comment

VIM : copy and paste reformats the text with lots of spaces

Posted on September 2, 2013 by Thomas Cokelaer

When I paste a piece of code or text from a copy (or the clipboard), lots of extra spaces are added at the start of each new line.

The solution is to turn off the autoindent:

:set paste

After you pasted your code, turn off the paste-mode:

:set nopaste
Posted in Linux | Tagged | 1 Comment

Fedora 19 post installation

Posted on September 1, 2013 by Thomas Cokelaer

Another failed upgrade from Fedora… This time while upgrading from release 15 to 16. Although, I followed the instructions from Fedora web site, I ultimately end up (after a reboot) with a grub prompt. Quite dreadful. After a few hours I manage to boot somehow but could not manage to fix the grub.conf after another couple of hours. Even though I ,anaged to get F16; I will then need to upgrade fro, 16 to 17 and then 17 to 18 and finally 18 to 19. So, I decided to reinstall directly from scratch with the latest Fedora 19. quite a frustrating experience I should say.

Here are the post installation procedure I followed.

Packages installation with yumex

First, update your existing packages.

sudo yum update

Then install yumex that provides a user interface to yum:

sudo yum install yumex

Firefox related

Firefox requires the flash plugin for the videos. Many web sites (e.g., Fedora FAQs) provides help on this subject. You need to install the plugin yourself. First, you need to add the repositories:

sudo yum install http://linuxdownload.adobe.com/adobe-release/adobe-release-x86_64-1.0-1.noarch.rpm -y

and then the plugin:

sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux
sudo yum install flash-plugin -y

Reference: https://fedoraproject.org/wiki/Flash

Install VLC player

VLC is a versatile mp3/video player but it requires to add a repository to the existing yum repositories, which is done as follows:

su -c 'rpm -Uvh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm'

Firefox has also many addons. I use the following ones:

xmarks
firebug
adblock plus

How to install acroread

First, you need to install the repositories

sudo rpm -ivh http://linuxdownload.adobe.com/adobe-release/adobe-release-i386-1.0-1.noarch.rpm
sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux

and then for Fedora 17/18/19 (on x86_64 bit system, 32-bit dependencies is also installed) type:

sudo yum install nspluginwrapper.i686 AdobeReader_enu

Note that it also installs gcc, readline and quit a few useful packages for software developers.

Gnome issues in Fedora 19

where is the desktop ?

Install the following package:

sudo yum install gnome-tweak-tool

Launch gnome-tweak-tool (from your account , not the sudo account), select Desktop. Set “Have File Manager handle the desktop” to ON.

How to maximize the windows

By default, there are no more minimize/maximize buttons. again , you can start gnome-tweak-tool.

Then, go to Shell and see the “Arrangements of buttons on the titlebar” options and select “ALL”.

Some extra packages to install

Some useful packages (general)

yum install thunderbird

Some useful packages (software)

yum install git svn kile okular dos2unix wget

Note that installing kile installs texlive

sudo yum install vim gvim  meld

Some python related packages

yum install ipython python-virtualenv R-core R-devel PyQt4 PyQt4-devel libpng-devel blas blas-devel lapack lapack-devel readline-devel coverage text-preview
  • readline-devel is required by rpy2
  • libpng-devel for matplotlib installation
  • blas blas-devel for scipy
  • tex-preview for sphinx 
sudo rpm -Uvh http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm

Install VLC player

VLC is a versatile mp3/video player but it requires to add a repository to the existing yum repositories, which is done as follows:

su -c 'rpm -Uvh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm'
sudo yum install vlc

Install java JRE

Simply go to the website and download section and click on the link that correspond to your system (32 bits/64 bits).
java web site

Other specific dependencies

sudo yum install libXp.x86_64

Continue reading →

Posted in Linux | Tagged | Leave a comment

How to mount Sony NWZ walkman under Fedora

Posted on September 1, 2013 by Thomas Cokelaer

I recently bought a mp3/video SONY NWZ walkman (one of B172F or B173F). I plugged in the USB into the computer but it was not detected. It does not appear on the desktop and cannot be seen in a shell either.

It looks like a general issue under Linux at the moment. From ubuntu forum, one suggestion is to seqrch for the label in the /dev directory:

ls /dev/disk/by-label -lah

to figure out the device name. However, the SONY device was not found; only hard disk related:

DellUtility  _Fedora-17-x86_6  OS  RECOVERY

However, I could find it in the by-id

ls /dev/disk/by-id/

Here is the name as it appear on my computer (amongst others):

usb-SONY_WALKMAN_1042D501543287-0:0

From there, you can mount the USB device as follows (note the slash before :0):

 sudo mkdir /media/Walkman
 sudo mount /dev/disk/by-id/usb-SONY_WALKMAN_1042D501543287-0\:0  /media/Walkman
Posted in Linux | Tagged , | 2 Comments

canviz, jake and npm

Posted on August 23, 2013 by Thomas Cokelaer

I wanted to play with canviz (http://www.ryandesign.com/canviz/) to play around with graphviz and javascript.

It’s still not clear what I’ll be able to do quickly with it but first I needed to install it.

I first tried from the source tar ball provided on the web site (http://canviz.googlecode.com/). This led to many issues that I’m provided here below. The outcome is that after an hour, I could not install it. So, I decided to start from the SVN,

svn checkout http://canviz.googlecode.com/svn/canviz/trunk/ canviz-read-only

and again no success: the javascript somehow was not working. The old version from the examples worked but not the new ones (see below). Finally, I found a link to the GIT repo and with that version, it worked out of the box:

git clone https://github.com/koops/canviz

 

Installing canviz from the source using npm and jake

!! This did not work at the end but some information here below could be useful.

In order to install canviz from the source, you need to install jake. But before, you need another tool called npm, an executable installed with Node.js (http://nodejs.org/download/).

tar xvfz node-v0.10.17.tar.gz
cd node-v0.10.17
./configure
make
sudo make install

Then, you can either install jake from the source file or using npm, which I realised later on. So, here is what I did to install jake from the source. First, get the source file from https://github.com/mde/jake

# installing jake 
git clone git://github.com/mde/jake.git
cd ./jake
make
sudo make install

Then, you can come back to the canviz installation from source.

Just type the following command in the canviz directory:

jake

I got this error:

[prompt] jake
jake aborted.
Error: Cannot find module 'browserify'
    at Function.Module._resolveFilename (module.js:338:15)
(See full trace by running task with --trace)

Here, a module is missing. You can install missing modules with npm:

npm install browserify

There are a couple of other modules to install. There was not problem except canvas:

make: Entering directory `/home/cokelaer/Downloads/canviz-read-only/node_modules/canvas/build'
  CXX(target) Release/obj.target/canvas/src/Canvas.o
In file included from ../src/Canvas.cc:8:0:
../src/PNG.h: In function ‘cairo_status_t canvas_write_png(cairo_surface_t*, png_rw_ptr, void*)’:
../src/PNG.h:139:10: error: ‘CAIRO_FORMAT_RGB30’ was not declared in this scope
make: *** [Release/obj.target/canvas/src/Canvas.o] Error 1
make: Leaving directory `/home/cokelaer/Downloads/canviz-read-only/node_modules/canvas/build'
continue....

Here the solution was to install a specific version of canvas as specified in the file package.json.

npm install canvas@1.0.4

Some references:

  • http://www.cappuccino-project.org/blog/2010/04/introducing-jake-a-build-tool-for-javascript.html
  • http://nodejs.org/download/
  • http://code.google.com/p/canviz/wiki/HowToUse
  • http://stackoverflow.com/questions/14472522/cannot-find-module-browserify
Posted in Computer Science | Tagged , | Leave a comment

Python: Error: new-line character – how to open the file in universal-newline mode ?

Posted on August 22, 2013 by Thomas Cokelaer

I was playing with the CSV module from the Python standard library. Typically using a code such as:

    import csv
    import zipfile
    zip = zipfile.ZipFile("file.zip")
    zipdata = zip.open("ExistingFile.csv")
    data_iter = csv.reader(zipdata, delimiter=",")
    data_iter.next()

So far so good, it did what I wanted that is reading a CSV file stored in a ZIP archive.

Then, I opened another ZIP archive and got this error:

---------------------------------------------------------------------------
Error                                     Traceback (most recent call last)
<ipython-input-19-d6c927ad0840> in <module>()
----> 1 data_iter.next()
Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?

It appears that the archive was created under MacOSX and I work under Linux so I suspected an issue with the end-of-line convention that are different under different systems (e.g., \n under Linux, \r under Mac and \r\n under Windows).

The solution is to open the file in universal mode. Replace

zipdata = zip.open("ExistingFile.csv")  # default mode is "r"

by

zipdata = zip.open("ExistingFile.csv", "rU")
Posted in Python | Tagged | Leave a comment

SVN: how to clean up a directory from unversioned files

Posted on July 31, 2013 by Thomas Cokelaer

Here is the command to delete all the files in a SVN directory that are not in the SVN server repository.

The first part (svn status) returns all files with the ? tag (meaning they don’t belong to the repository). The –no–ignore should be used if you have some set some files to be ignored in your SVN configuration file.

The “grep” command is used to deal with the question tag.

finally, the “xargs ” commands perform the cleanup.

Be aware that the final option “-rf” means delete directories and don’t have for confirmation so to be used with care !!!

svn status --no-ignore | grep '^\?' | sed 's/^\?     //'  | xargs rm -rf
Posted in Linux | Tagged | 1 Comment

R package dependencies: difference between imports, depends, suggests, enhances

Posted on July 31, 2013 by Thomas Cokelaer

When you want to assemble R codes into a package, you will need to write a file called DESCRIPTION. This file contains some metadata and keywords to be used and some keywords are definitely confusing (at least to me). Here below, I try to emphasize the difference between them (imports, enhances, depends,…).

An advanced DESCRIPTION file would look like:

Package: a_package_name
Type: Package
Title: your clever title
Version: 1.7.6
Date: 2013-07-30
Author: list of authors separated by commas
Maintainer: yourname &lt;username@whatever.org&gt;
Depends: R (>= 2.15.0), RBGL, graph
Suggests: RUnit, BiocGenerics, igraph
biocViews: Bioinformatics, TimeCourse
Description: this package does blablabla
License: GPL-3
LazyLoad: yes
SystemRequirements: anotherPackage version >= 2.2

Package, Type, Title, Version, Date, Author, Maintainer, License are obvious. biocViews contains a list of keywords (bioconductor related).

For the others, it is still a bit confusinf to me but here is what I understood.

  • Depends: list here all packages that are required by your package including version if needed (checked if the packages are installed when calling R CMD INSTALL)
  • Suggests: list here packages that are used in the vignettes (when calling R CMD check)
  • SystemRequirements: list package required but not automatically installed when running R CMD INSTALL

There are two others that could be used as well:

  • Enhances: packages that could improve the performance but not strictly required.
  • Imports: If you use a namespace and added import(“package1”), you don’t need to use “Imports: package1” in the DESCRIPTION file. If you use “Depends”, you don’t need the import. However, using “Imports” is safer than “depends” with respect to scenario where other loaded packages may mask existing functions. Note also that using namespace brings only data or functions apparently. Examples would surely help to explain these different statements. Personally I always use depends and had no issues so far but simply because the packages are simple enough.
Posted in R, Uncategorized | Tagged | 1 Comment

Python: list of empty lists creates references not copy

Posted on July 29, 2013 by Thomas Cokelaer

In Python, you can create a list with an element repeated N times quite easily with the following syntax:

    >>> a = [1] * 5
    >>> print(a)
    [1,1,1,1,1]

Here we create a list with a simple element (1) not an object. As soon as you manipulate objects, you need to be cautious:

    >>> a = [[1]] * 5
    >>> print(a)
    [[1],[1],[1],[1],[1]]

so far so good but if you change an element, you change them all:

   >>> a[0].append(2)
   >>> print(a)
   [[1,2],[1,2],[1,2],[1,2],[1,2]]

The reason being that in [[1]], the first element is duplicated using a reference not a copy.
Instead, you could use:

    >>> a = [[1] for i in range(5)]
Posted in Python | Tagged | Leave a comment