python: how to merge two dictionaries

Let us suppose two dictionaries storing ages of different individuals:

If you do mind losing the contents of either list1 or list2 variable, you can update one of the other as follows:

Now list1 variable contains:

while list2 is unchanged.

Usually, this is not what you want though. Instead, you would prefer to create a third variable keeping list1 and list2 unchanged.

In Python 3.5 or greater, you can use the following syntax:

In Python 2 or 3.4 and below, you need to copy one of the variable and update it:

The second method is more generic and would be more backward compatible (if you plan to provide your code to Python 2 users. Indeed, it would work for Python 2 and 3. However, it would be slower for Python 3.5 users (and above).

Posted in Python | Tagged | Leave a comment

Search for a pattern in a set of files using find and grep commands

A common task for developers is to search for a pattern amongst a bunch of files that are in different directories.

For instance, you are looking for the pattern “import sys” within a set of Python files. Those files are in sub directories mixed with other documents.

You can use the find command (to look for files ending with the py extension) and redirect those files to the grep command to search for the pattern “import sys” within all files found by the find command:

find . -name "*py" | xargs grep "import sys"

Note the double quotes and the use of the xargs command to scan the content of the files (not their names).

Of course, you can use all kind of wildcards:

find . -name "*py" | xargs grep "import sys"

Posted in Linux | Tagged , | Leave a comment

okular: export annotations in the PDF file

One open source software to add annotations under Linux is okular (

One can add annotations easily (go to Tools, tick review, or just type F6).

Then, it is time to save your document or to send it to a collaborator but wait a minute: we do not see the annotations when using acroread reader !! No worries, plenty of resources tell you to go to File/save as

Seems to work indeed. You quit, open the file and there you can see the annotations. Now, I use xpdf to read the PDF file. And here nothing. Oh, and I send the PDF to a journal review; they include the PDF inside another one and there no annotations either….

Final solution: instead of File/Save as, just print the document in … a PDF file: File/Print

This worked for me.

Posted in Linux | Tagged | 1 Comment

No more space left on /tmp under Fedora

Under Fedora, one of my software requires more than 4Gb of temporary space and I realised that the /tmp directory is limited to 4Gb. In order to increase the /tmp directory, just edit the /etc/fstab file and add this line (to extend to 8Gb instead of 4Gb):

Then, as root:

That’s it. You can check that you have now 8Gb available in /tmp by typing

Posted in Linux | Leave a comment

blasr (pacbio) installation on fedora box

I wanted to use blasr tool for Pacbio mapping and had difficulties in using or installing the tool. I first use a local installation of the tool on the provided cluster and it look like the installation was quite old. I then use a bioconda version. The latest version (5.3) was not working on my system (missing library) and this was reported. The previous version worked (5.2) but was missing a needed configuration flag. So, I decided to follow the instructions from pacbio to install a local version. That was not straightforward but finally got it to work. My platform is Fedora 23 and the instructions were given for ubuntu or centos 6.

First, download the source code:

This command installs zlib, bzip2, boost and hdf5 to start with.
The first issue arised from an error in the compilation of the hdf5 dependency due to missing iostream in the compilation of the hdf5 library

The solution was to edit the Makefile in workspace/hdf5-1.8.16/c++
and to comment this line (adding a # character in front))

Come back to the main pitchfork directory, type the make command again (see above). Another failure due to similar compilation error related to the namespace occured. Again, I edited the Makefile in
workspace/hdf5-1.8.16/c++/test/Makefile and commented the same line.

Next, I got a linking issue

This was solved by installing the stdc++ static library. I figure out the solution by typing:

to see that none of the standard path could find the library despite the presence of the /usr/lib/

Again, type

I got

so again, needed to find the culprit: a Makefile and it was in
In workspace/hdf5-1.8.16/c++/hl/test/Makefile . Again comment the same line as shown above.

Back to blasr, the make then failed when trying to install ncurses.

Here I tried a different strategy and tried to use the packages installed with my conda environement. To do so, I edited the and added:

Then, same issue with samtools, so added

Then, there was an error in the ./bin/pitchfork module due to a Python 3 issue (I had already switch all print “” to print(“”). Here, the issue was

Just replace _out[0] with str(_out[0])

Then, blasr compiles successfully…time to run it; The executable seems to be in workspace/blasr/blasr:

./workspace/blasr/blasr: error while loading shared libraries: cannot open shared object file: No such file or directory

Here, you need to add a bunch of path to your LD_LIBRARY_PATH:

Posted in biology | Tagged , | Leave a comment

swapping two columns with awk keeping tabulation

Assuming you have a data file with N columns and you want to swap the first and second one, just type:

To keep the tabulation intact, you need to specify the separator.

Incoming separator is defined by -F $’\t’ and the separator for the output is defined by OFS=$’\t’.

Posted in Linux | Tagged | Leave a comment

set a default version to an environment module

Context: “The environment modules package provides for an easy dynamic modification of a user’s environment via modulefiles. which typically instruct the module command to alter or set shell environment variables such as PATH, MANPATH, etc. as well as define aliases over a variety of shells.”

Reference: modules.sourceforge

When you type “module av PKGNAME” you get a list of available version for the PKGNAME. You may see the (default) one.

By default, the highest lexicographically sorted modulefile under the directory is set as the default. However, you can change this behaviour by creating a .version file in that directory which has a format like the following

Posted in Linux | Tagged | Leave a comment

fedora aeroplane mode always on

Under Fedora 23, somehow the aeroplane mode went on (plane icon shown in top right corner). I tried to switch it off but could not. After a reboot, the aeroplane is still switch on and there is no way to switch it off. So, neither wire connection nor wifi connection can be switch on.

After a while and some browsing, this solution worked for me:

Note, however, that on my system rfkill was not installed… so i got the RPM (rfkill-0.5.6.fc23.x86_64.rpm) from another source, copied it on a usb key and install it as follows:

Posted in Linux | Tagged | Leave a comment

packet_write_wait: Connection to XXX : Broken pipe

In general, I can connect to a remote machine via SSH without any troubles but occasionally in some particular setup, although I can connect to the remote machine and type commands in a shell, the connection is broken after a few minutes of connection.

This happens when we do not interact with the terminal (i.e., we do not type anything on the keyboard).

In such case, the following error message appears:


On the host, add those lines in the file .ssh/config

If the file config does not exist, just create it.

The ServerAliveInternal pings the server with a null packet every 30 seconds. As for the second line, I have not check whether it is required or not.

Posted in Linux | Tagged | 1 Comment

pytest-cov collects no data on Travis

I recently switched from nosetests to pytest and pytest-cov for one of my Python project (

Locally, everything seemed to work fine and once on Travis, all tests passed as expected.

Yet, before the coverage report, I noticed this warning:

and more importantly, the final coverage reported was 0% !!

I later realised that one file in the test/ directory was present locally but not in the github source code. After a commit, travis found the missing file and the coverage came back to normal.

So, this seems important for pytest-cov

Posted in Python | Tagged , , | 5 Comments