How to use subprocess with PIPEs

In Python, the subprocess module allows you to execute linux/unix commands directly from Python. The official Python documentation is very useful would you need to go further. Yet it may be a bit scary for newbies. Syntax may also change from one Python version to the other.

Here is a solution to use subprocess to encode piped commands with Python 3.7. For example imagine you want to encode this command that contains a pipe and a redirection:

    zcat file1.dat.gz file2.dat.gz | pigz > out.gz

Here is python example on how to implement this unix command using Python subprocess. The first call to the subprocess encodes the pipe. The second command encodes the redirection:

p1 = subprocess.Popen(["zcat", "file1.dat.gz", "file2.dat.gz"], 
                       stdout=subprocess.PIPE)
fout = open('out.gz', 'wb')
p2 = subprocess.run(['pigz'], stdin=p1.stdout, stdout=fout)

Note that the first subprocess uses the Popen method while the second call uses the run method. If you use the run() method for the pipe itself,

p1 = subprocess.run(['zcat', 'file1.dat.gz', 'file2.dat.gz'],
    stdout=subprocess.PIPE)
fout = open('out.gz', 'wb')
p2 = subprocess.run(['pigz'], stdin=p1.stdout, stdout=fout)

you would get this kind of error:

AttributeError: 'bytes' object has no attribute 'fileno'

Please follow and like us: