Monday, August 10, 2009

PEP Submission

Now that the coding is more or less complete, I have produced and submitted the following P.E.P. I have not officially been assigned a P.E.P. number but I stuck 3144 on there as the newest one when I began writing this one was 3143.

PEP: 3144
Title: Asynchronous I/O For subprocess.Popen
Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson
Status: Draft
Type: Standards Track
Requires: 324
Created: 04-08-2009
Content-Type: text/plain
Python-Version: 3.2

Abstract:
In its present form, the subprocess.Popen implementation is prone to
dead-locking and blocking of the parent Python script while waiting on
data from the child process.

Copyright:
This P.E.P. is licensed under the Open Publication License;
http://www.opencontent.org/openpub/.

Motivation:
A search for "python asynchronous subprocess" will turn up numerous
accounts of people wanting to execute a child process and communicate
with it from time to time reading only the data that is available
instead of blocking to wait for the program to produce data [1] [2]
[3]. The current behavior of the subprocess module is that when a user
sends or receives data via the stdin, stderr and stdout file objects,
dead locks are common and documented [4] [5]. While communicate can be
used to alleviate some of the buffering issues, it will still cause
the parent process to block while attempting to read data when none is
available to be read from the child process.

Rationale:
There is a documented need for asynchronous, non-blocking
functionality in subprocess.Popen [6] [7, comments] [2] [3]. Inclusion
of the code would improve the utility of the Python standard library
that can be used on Unix based and Windows builds of Python.
Practically every I/O object in Python has a file-like wrapper of some
sort. Sockets already act as such and for strings there is StringIO.
Popen can be made to act like a file by simply using the methods
attached the the subprocess.Popen.stderr, stdout and stdin file-like
objects. But when using the read and write methods of those options,
you do not have the benefit of asynchronous I/O. In the proposed
solution the wrapper wraps the asynchronous methods to mimic a file
object.

Reference Implementation:
I have been maintaining a Google Code repository that contains all of
my changes including tests and documentation [9] as well as blog
detailing the problems I have come across in the development process
[10].

I have been working on implementing non-blocking asynchronous I/O in
the subprocess.Popen module as well as a wrapper class for
subprocess.Popen that makes it so that an executed process can take
the place of a file by duplicating all of the methods and attributes
that file objects have.

[1] http://mail.python.org/pipermail/python-bugs-list/2006-December/036524.html
[2] http://ivory.idyll.org/blog/feb-07/problems-with-subprocess
[3] http://stackoverflow.com/questions/636561/how-can-i-run-an-external-command-asynchronously-from-python
[4] http://docs.python.org/library/subprocess.html#subprocess.Popen.wait
[5] http://docs.python.org/library/subprocess.html#subprocess.Popen.kill
[6] http://bugs.python.org/issue1191964
[7] http://code.activestate.com/recipes/440554/
[8] http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5#437
[9] http://code.google.com/p/subprocdev
[10] http://subdev.blogspot.com/

2 comments:

  1. Here here! [BUMP]

    ReplyDelete
  2. Good work. Hope this PEP gets accepted soon, as subprocess in its current blocking form is such a royal pain.

    ReplyDelete