corrplot function in Python

We’ve implemented a corrplot function in Python, which is available in BioKit package

– github https://github.com/biokit/biokit
– pypi https://pypi.python.org/pypi/biokit

For illustration, let us create some random data sets:

import pandas as pd
import string
letters = string.uppercase[0:15]
df = pd.DataFrame(dict(( (k, np.random.random(10)+ord(k)-65) for k in letters)))
df = df.corr()

The correlation matrix is stored in the Pandas dataframe called df. It is now straightforward to create some nice plots:

from biokit.viz import corrplot
c = corrplot.Corrplot(df)
c.plot()

corrplot

See this notebook for more examples.

Posted in Python | Tagged , , , | 4 Comments

About matplotlib colormap and how to get RGB values of the map

matplotlib comes with lots of colormaps. First, let us see where and how to find them. Then, we will see how to extract individual colors (e.g., in RGB) from the colormap itself.

Let us first creat an image

from matplotlib import imshow
import numpy as np
imshow(np.random.rand(10,10))

By default, the jet colormap is used. You should see something like this

post_sept_2014_1

You can chnage the colormap easily but you need to know the name of the colormap first. Some colormaps are available as functions. For instance the copper one:

from matplotlib import imshow
copper()

post_sept_2014_2

How can we know the names of the colormap ? Check the documentation of colormaps itself for details or to obtain the full list of colormap names, type

from matplotlib import imshow
colormaps()

as mentionned above, some colormaps are available as functions. If not, you should use the function set_cmap:

from matplotlib import imshow
from matplotlib import set_cmap
set_cmap('copper')
set_cmap('Accent')

Finally, if you want to extract the color contained in a colormap, use the cm function:

from matplotlib import imshow
>>> from matplotlib import cm
>>> cm.jet(0)
(0, 0, 0.5, 1)

colormaps are usually encoded with N=256 colors. To figure out the first one, we use the code as above. If you want the latest one, use cm.jet(255)
Note that the returned objet is tuple of 4 items. The last one being the transparency.

Posted in Python | Tagged | 1 Comment

git and github : skip password typing with ssh

Assuming you have already provided your ssh key in your github settings, you can avoid typing your username and password when when pushing content into your github account, by cloning the SSH repository instead of the default HTTP one.

ssh://git@github.com/username/repo.git

instead of

https://github.com/username/repo.git
Posted in Computer Science | Tagged | Leave a comment

Installing Python2.6 in a virtual environment

First, install python2.6 executable. see previous post if you have issues when compiling (gcc not found).

Make sure you have installed python-dev with apt-get or yum.

Then, following the instructions on the same topic but for Python 3 in a virtualenv,

type:

 
virtualenv --python=/home/<user>/Software/Python2.6/mybuild/bin/python2.6 virtualenv2.6 --no-setuptools
cd virtualenv2.6

Note the usage of –no-setuptools option. I had error related to pip and setuptools that could not be installed.

So, now we need to install them manually, but before, let us activate the newly created virtualenv:

cd virtualenv2.6
source bin/activate

and now, let us download and install setuptools:

wget https://bootstrap.pypa.io/ez_setup.py -O - | python
unzip setuptools-5.6.zip 
cd setuptools-5.6/
python setup.py install

and finally, let us install pip

easy_install-2.6 pip

that’s it we are ready to use python2.6

Posted in Python, Software development | Tagged , | 2 Comments

Installation of Python 2.6 error fixed

I wanted to install python2.6 in a virtual environment. The first step was to compile python 2.6 executable from the source code.

Once the source are extracted, just type:

./configure --prefix=<PATH_WHERE_TO_INSTALL>
make

If you are lucky, everything is smooth and you then just need to type

make install

In my case, under Linux Fedora, I got these error message:

gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -DSVNVERSION=\"`LC_ALL=C svnversion .`\" -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c
gcc: error: directory": No such file or directory
make: *** [Modules/getbuildinfo.o] Error 1

Looking into the c source file, I figured out that the issue is related to a SVN problem. Indeed, the SVNVERSION variable needs to be defined in your environement.

Figure out what is the svn version first (svn –version) and type something like:

export SVNVERSION=1.7.8  # or whatever is your version
./configure
make

It worked for me.

Posted in Python | Tagged , | 11 Comments

Installing another Python version into virtualenv

The idea is that we do not want to install a new version of Python on the system (e.g., for testing purpose), which may interfere with your entire distribution.

Instead, we want to create a virtual environment with virtualenv tool.

Let us assume you have python 2.7 and you want to play with Python 3.4

1. First, the Python executable

cd
mkdir Software
cd Software
wget  https://www.python.org/ftp/python/3.4.1/Python-3.4.1.tar.xz
unxz Python-3.4.1.tar.xz
tar xvf Python-3.4.1.tar
cd Python-3.4.1
./configure --prefix=/home/user/Software/Python-3.4.1/mybuild
make
make install

2. VirtualEnv setup

If you decide to install python3.4 (latest version), make sure you also have the latest version of virtualenv (1.11). I used 1.9 and got this kind of errors:

ImportError: No module named '_collections_abc'
...
ERROR: The executable py34/bin/python3 is not functioning
ERROR: It thinks sys.prefix is ....
ERROR: virtualenv is not compatible with this system or executable

updating virtualenv with

pip install --upgrade virtualenv

fixed the issue.

On another computer, I got this error message:

ImportError: cannot import name 'HTTPSHandler'

which was due to a missing openssl library:

sudo yum install openssl openssl-devel

openssl-devel is the package to install under Fedora. maybe differently named on other distributions.

Let us now create the virtualenv, which is just like a normal directory (let us call it python3)

virtualenv --python=/home/&lt;user&gt;/Software/Python3.4/mybuild/bin/python3.4 virtualenv3
cd virtualenv3

3. Initialise the virtualenv and testing

cd virtualenv3
source bin/activate
python

You should now be able to check the python version to be 3.41

Python 3.4.1 (default, Aug 16 2014, 18:35:43)
[GCC 4.7.2 20120921 (Red Hat 4.7.2-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Posted in Python, Software development | Tagged | 6 Comments

How to add badges in your pypi project

1. pypi badge

To get the pypi badge, go to https://badge.fury.io/for/py page and enter the name of a valid package. Select the code you’d like to insert into your documentation (e.g., RST) and you check get a image like this:

https://badge.fury.io/py/pypiview.svg

2. Number of downloads

Nothing special to do here. Use the link https://crate.io/packages/ and append your package name.

https://pypip.in/d/pypiview/badge.png

3. Continuous integration on Travis

This one is definitely the most difficult but also the most rewarding since you will be able to test your entire software and coverage on different distributions.

First, open an account on https://travis-ci.org/ . It is quite straightforward to synchronise your github with travis. You will see all your repositories and will need to specify those you want to include into the build process. To do so, cick on the repository of your choice in the https://travis-ci.org/profile/username

For a build to start, you will need to commit/push a change in the github and a file in your repository (same level as setup.py and README.rst) called .travis.yml

You can check that the syntax of your file is correct using travis-lint executable.

 travis-lint .travis.yml

This executable can be installed easily using the gem utility:

gem install travis-lint

Here is an example of such as file:

language: python
python:
  - "3.3"
  - "2.7"
  - "2.6"
before_install:
  - "export DISPLAY=:99.0"
  - "sh -e /etc/init.d/xvfb start"
# command to install dependencies, e.g. pip install -r requirements.txt --use-mirrors
install: 
  - pip install .
  - pip install nose coverage
  - pip install coveralls
# # command to run tests, e.g. python setup.py test
script:  
  - python setup.py nosetests --with-coverage --cover-package pypiview
 
after_sucess:
  coveralls

From now one, each time you commit/push a change travis will start a new build. If you do not want to trigger a new build (because you’ve just change a print statement), include this line into your commit statement:

[ci skip]

If you want coverage (next section), you need to use the script and after_sucess sections otherwise just comment or remove them.

Note also that if you use matplotlib in your test, the build may fails with an error related to the DISPLAY. This is the reason for the before_install section and the statement related to the DISPLAY.

Having your first build to pass without errors may take you some time and iterations… Finally, you’ll get a badge as follows

https://secure.travis-ci.org/cokelaer/pypiview.png

4. Coverage

Once you’ve managed to have a succesfuly build on Travis, it is time to look at the coverage. This is very important that the build succeeded otherwise, this step will not work.

Go to https://coveralls.io, create an account and add your github directory. You may need to synchronise your repo.
There, click on an icon called (view on coveralls) on the same line as your pacakge. In the new page, look for the repo token
, which is a list of random character and number. Keep it for later.

You will need to trigger travis CI by committed some changes.

this should then trigger the coveralls website where the badge should now be ready.

THere is another way, which is more manual; I did not manage but instructions are See https://github.com/coagulant/coveralls-python for help.

The final badge looks like:

https://coveralls.io/repos/cokelaer/pypiview/badge.png?branch=master

Quality

an easy one now. Get a badge for the quality of your code (style) by joining landscape website https://landscape.io

add the repository you wish to test going to the bottom of this page

https://landscape.io/repository/add

https://landscape.io/github/cokelaer/pypiview/master/landscape.png

Tasks

Another easy one is the task badge from waffle. Open an account and then go to

https://waffle.io/cokelaer/PROJECT

https://badge.waffle.io/cokelaer/pypiview.png?label=ready&title=Ready

Update pypi

Finally, you need to update your README.rst , which should be in your github repository with the following code (adapted to your needs of course)

.. image:: https://badge.fury.io/py/PROJECT.svg
    :target: https://badge.fury.io/py/PROJECT.svg
 
.. image:: https://pypip.in/d/PROJECT/badge.png
    :target: https://crate.io/packages/PROJECT/
 
.. image:: https://secure.travis-ci.org/USERNAME/PROJECT.png
    :target: http://travis-ci.org/USENAME/PROJECT
 
.. image:: https://coveralls.io/repos/USERNAME/PROJECT/badge.png?branch=master 
    :target: https://coveralls.io/r/USERNAME/PROJECT?branch=master 
 
.. image:: https://landscape.io/github/USERNAME/PROJECT/master/landscape.png
   :target: https://landscape.io/github/USERNAME/PROJECT/master
 
.. image:: https://badge.waffle.io/USERNAME/PROJETC.png?label=ready&amp;title=Ready 
   :target: https://waffle.io/USERNAME/PROJECT
Posted in Python | 3 Comments

Migrating from a Subversion repository to GitHub

I’ve just moved a SVN repository to github. I was a bit afraid of spending time and lost the history but fortunately, I came upon this great post http://www.samaxes.com/2013/11/move-from-svn-to-git/ , followed the instructions and it worked almost like a charm.

Here is a summary of the instructions but please visit the original post if you want to understand what you are doing. The following being just a recipe rather than a thorough explanation of what is going on behind each instructions.

Let us assume that the name of the SVN repository is MYPROJECT and you have it locally in SVNDIR and that you will name it on github with the same name MYPROJECT.

First, we need to figure out the names of the contributors in the SVN directory and their respective names in the github repository:

 cd SVNDIR
svn log --xml | grep -P "^<author" | sort -u | perl -pe 's/&lt;author&gt;(.*?)&lt;\/author&gt;/$1 = /' &gt; authors.txt

edit the authors.txt to add the github author names. Keep it safe. We will use it later on. The format looks like

svnuser1 = Firstname Surname <email>

Note that you may have a svnuser called www-data. I kept it but put my firstname/surnma/email to replace it in the new git repo.

Now, let us copy the SVN into a new directory and make sure this new directory is cleaned up from useless files. We can do that by just cloning the SVN repository using git itself:

  cd ~/TEMP 
  git svn clone https://subversion.assembla.com/svn/MYPROJECT --no-metadata --stdlayout --authors-file=authors.txt MYPROJECT
 cd MYPROJECT

If you end up with a

git: 'svn' is not a git command. See 'git --help'.

just install git-svn tool, which is probably missing on your system.

Here of course, you need to provide the proper svn repository. To obtain that information type svn info in your SVNDIR directory. The file authors.txt is the one created earlier.

Note here, that you will get everything (trunk/tags/branches) and the following code will clean up things for you so you end up with a
proper gitbuh repository

 git for-each-ref refs/remotes/tags | cut -d / -f 4- | grep -v @ | while read tagname; do git tag "$tagname" "tags/$tagname"; git branch -r -d "tags/$tagname"; done
 
 git for-each-ref refs/remotes | cut -d / -f 3- | grep -v @ | while read branchname; do git branch "$branchname" "refs/remotes/$branchname"; git branch -r -d "$branchname"; done

Before going to the next steps, do not forget to ceate the MYPROJET repository on github itself !!

The next step took me a while to understand. In the original post,

 git remote add origin git@my-git-server:myrepository.git

I thought I had to do the following replacements:

  • git by my username but no ! keep it as it is.
  • my-git-server is github.com, which is correct
  • myrepository.git by MYPROJECT but no, it should be USERNAME/MYPROJECT

So, just type something like that shoul work:

  git remote add origin git@github.com:USERNAME/MYPROJECT
  git push origin --all -v

I also had issues with permission at that stage. The following command helped me somehow to figure out it was a SSH key missing on github

    ssh -vT git@github.com
    ssh-add -l

Hoping this helps.

References:

  • https://help.github.com/articles/generating-ssh-keys
  • http://www.samaxes.com/2013/11/move-from-svn-to-git/
  • http://imakewebthings.com/blog/github-pages-email/
  • https://help.github.com/articles/error-repository-not-found
Posted in Uncategorized | Leave a comment

How to calculate mean in AWK

Another trick with awk to compute mean of a column.

Imagine that you have a file with many lines and you want to compute the mean of the fourth item on each line. E.g:

Lowest value found .2
Lowest value found .4
Lowest value found .6

Using the awk tool under Linux, type:

awk '{x+=$4; next} END{print x/NR}'

and you should get the answer.

Posted in Linux | Tagged | Leave a comment

How to send svn diff to meld

On one hand meld provides a nice GUI to visualise the differences between 2 files.

On the other hand, with SVN diff command, you can obtain the differences between 2 versions of the same file so you end up with one file.

In order to combine the two, you can simply type:

svn diff --diff-cmd='meld' -r4550 yourfile
Posted in Linux | Tagged , | 7 Comments