PEP Submission

Now that the coding is more or less complete, I have produced and submitted the following P.E.P. I have not officially been assigned a P.E.P. number but I stuck 3144 on there as the newest one when I began writing this one was 3143.

PEP: 3144
Title: Asynchronous I/O For subprocess.Popen
Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson
Status: Draft
Type: Standards Track
Requires: 324
Created: 04-08-2009
Content-Type: text/plain
Python-Version: 3.2

Abstract:
In its present form, the subprocess.Popen implementation is prone to
dead-locking and blocking of the parent Python script while waiting on
data from the child process.

Copyright:
This P.E.P. is licensed under the Open Publication License;
http://www.opencontent.org/openpub/.

Motivation:
A search for "python asynchronous subprocess" will turn up numerous
accounts of people wanting to execute a child process and communicate
with it from time to time reading only the data that is available
instead of blocking to wait for the program to produce data [1] [2]
[3]. The current behavior of the subprocess module is that when a user
sends or receives data via the stdin, stderr and stdout file objects,
dead locks are common and documented [4] [5]. While communicate can be
used to alleviate some of the buffering issues, it will still cause
the parent process to block while attempting to read data when none is
available to be read from the child process.

Rationale:
There is a documented need for asynchronous, non-blocking
functionality in subprocess.Popen [6] [7, comments] [2] [3]. Inclusion
of the code would improve the utility of the Python standard library
that can be used on Unix based and Windows builds of Python.
Practically every I/O object in Python has a file-like wrapper of some
sort. Sockets already act as such and for strings there is StringIO.
Popen can be made to act like a file by simply using the methods
attached the the subprocess.Popen.stderr, stdout and stdin file-like
objects. But when using the read and write methods of those options,
you do not have the benefit of asynchronous I/O. In the proposed
solution the wrapper wraps the asynchronous methods to mimic a file
object.

Reference Implementation:
I have been maintaining a Google Code repository that contains all of
my changes including tests and documentation [9] as well as blog
detailing the problems I have come across in the development process
[10].

I have been working on implementing non-blocking asynchronous I/O in
the subprocess.Popen module as well as a wrapper class for
subprocess.Popen that makes it so that an executed process can take
the place of a file by duplicating all of the methods and attributes
that file objects have.

[1] http://mail.python.org/pipermail/python-bugs-list/2006-December/036524.html
[2] http://ivory.idyll.org/blog/feb-07/problems-with-subprocess
[3] http://stackoverflow.com/questions/636561/how-can-i-run-an-external-command-asynchronously-from-python
[4] http://docs.python.org/library/subprocess.html#subprocess.Popen.wait
[5] http://docs.python.org/library/subprocess.html#subprocess.Popen.kill
[6] http://bugs.python.org/issue1191964
[7] http://code.activestate.com/recipes/440554/
[8] http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5#437
[9] http://code.google.com/p/subprocdev
[10] http://subdev.blogspot.com/

Prep the PEP

The CRLF Windows issues is the only thing holding me back from producing a patch and PEP and submitting it for inclusion into the Python Core. I posted on comp.lang.python and until I get a response, I will be gathering sources, references and planning out my PEP as well as looking over my code and checking to see where optimizations can be made.

Unrelated

While chatting with an Argentinean buddy, he used an amusing portmanteau. He referred to "Windows" as "Guindous," a combination of the word Windows and the Spanish verb "guindar" or "to freeze." I was rather amused and felt like sharing the term with people.

stdout

I discovered that Wing IDE has beta releases with Python 3.1 support and have been using them to debug my code. My unit tests for my subprocess.Popen changes now work flawlessly on both Linux and Windows. The only things holding me back now are issues with stdout being opened in text mode by libc. See the following message from Amaury on Python-Dev:

> Ah so any streams opened in text mode on Windows will read '\n' newlines as
> '\r\n'?

No, it is the libc stdout which is opened in text mode. This simple program:
int main() {
printf("Hello\n");
}
when run like this:
program > out
will create a file ending with \r\n.

ReadFile and WriteFile (and other functions from the win32 API) are
unaware of this, and faithfully transmit the bytes without
modification.

This is causing my unit test for my file wrapper to fail; all of my '\n' newlines are converted to '\r\n.' I would greatly appreciate suggestions on how to deal with it as well as using ctypes or adding to _subprocess.c. If you are not subscribed to Python-Dev, you can view the community's discussion on whether ctypes or C code would be a better solution at this link; http://mail.python.org/pipermail/python-dev/2009-July/090720.html

C You Later

I have abandoned modifying _subprocess.c in favor of a path that will be much easier than attempting to pull out the parts of Mark Hammond's C++ code and converting it to C: Python ctypes. After importing ctypes, I can call "ctypes.windll.kernel32.PeekNamedPipe" to access the PeekNamedPipe function which is used in the modifications to subprocess.py. In C/C++ multiple return values are generally handled by passing variable by reference. In ctypes, this functionality is duplicated so I must pass the variables that I will need to get data from using the ctypes.byref() function and then parse said data into a tuple.

MSVC++ Runtime Library

When examining the full build output from Microsoft Visual C++ Express, it was discovered there was an error that was along the lines of "MSC_DLL_LIB" and the program make_versioninfo.exe. Executed alone, make_versioninfo.exe complained of missing MSVC++ DLL runtime libraries and after re-install Microsoft Visual C++ Express, Python now builds successfully.

I adjusted Mark Hammond's code to be integrated into the _subprocess.c file and now that I can compile Python on Windows, I will be testing it to make sure it functions as expected and integrating the other two routines from PyWin32 that I need for my modification to subprocess.py.

Not DEP

I have modified one of the three functions from Mark Hammond's PyWin32 library to be integrated into the _subprocess.c file of the Python 3.1 source code. When I first attempted to compile the Python source code in Windows yesterday, I encountered a linking error: "LINK : fatal error LNK1181: cannot open input file '.\python31_d.lib'". Unable to compile the code, I went to the Python-Dev list for assistance and a few suggestions were made but none seemed to work. I disabled Windows Data Execution Protection and restarted my machine and downloaded the Python 3.1 source code from the SVN library. That code compiled fine and I considered the issue resolved.

As the old saying goes "correlation does not imply causation." I have, again, encountered the same linking error and can no longer compile the Pytohn source. I downloaded both the Python 3.2 code from the Subversion repository again; did not compile. I downloaded the official Python 3.1 code Gzipped and Bzipped tarbills; neither compiled.

Luckily, I thought ahead when I woke up and prepared to compile the _subprocess.c file with Mark Hammond's code integrated into it. I downloaded the Python 3.1 tar ball and attempted to compile taking screenshots at each step.

I downloaded the Python 3.1 source tar ball from python.org.

Using WinRAR, I extracted the source code.

I tried double clicking the pcbuild.sln file and right clicking and selecting open but neither of those actions did anything. The cursor changed to an hour glass briefly but Visual C++ did not open.

I then opened Visual C++ and went to "File," then "Open."


I browsed for the pcbuild.sln file and selected it to open.
A modal error dialog about a limitation in my version of Visual C++ popped up.

The project opens after hitting "OK" so I right click on the "python" portion of the project and hit "Build."

It proceeds to build but eventually errors at the end of the process.
Build log output: http://pastebin.com/m616681cc