Not Far From The Tree

Now that my implementation of subprocess.Popen is complete and unit tested, I will be moving on to moving my changes to Python 3.1. I just compiled Python 3.1 on Debian, all modules included, and pdb is still mal-functioning so if the code conversion isn't as smooth as I anticipate, I will be using print statements to debug which is a bit difficult due to the nature of subprocess.Popen.

The license of my project has been changed from GPL to Apache License 2.0 so that there is a chance of it being integrated into the Pytohn core. There are still some issues that I may have handled incorrectly as far as my subprocess.Popen file wrapper goes, mainly handling the "mode" argument that is used when one opens a file. Right now, it is mostly ignored, the exception being universal newline support.

Updates

Though it has been a while since I have made a post, things are going well right now. Since my last post, I have created a Google Code Repository located at http://code.google.com/p/subprocdev/. I have integrated, modified and written tests for the code I got from Josiah Carlson. Two of the functions he left outside of his Popen patch that were moved into subprocess.Popen. I made some aesthetic and functional changes to his recv_some and send_all functions.

Eventually, I got my code and Josiah's code just about fully unit tested in Python 2.7 and from there, proceeded to see if I could convert it over to 3.0. After I did this, I realized that it still ran, for the most part, in Python 2.7. With the following code, I attempted to get around the only thing that was causing me grief:

import sys
if sys.version_info[0] == 3:
from io import BufferedWriter as buffer
else:
class str(object):
def __init__(self,a,b=None):
self = a.__str__()
def isinstance(a,b):
if b == str:
return hasattr(a,'strip')
try:
return a == b(a)
except:
return False

In Python3.0, the str function can also accept character encoding as an argument so I tried to overload the Python 2.7 str class to make it 3.0 compliant to no avail so I will be producing a separate branch for Python 2.7 and 3.0 development in my Google Code Repository but any suggestions on getting around the str issue are greatly appreciated.

After contacting Josiah Carlson, author of the activestate patch for asynchronous I/O in subprocess.Popen, he was more than happy to release the code into the public domain so my license conflicts with his patch are now resolved. Some of the tests I wrote for his patch have failed and he agreed to take a look at what I wrote to help me get the tests running properly.

Upon running my tests, he realized that pywin32 is not part of the standard library which complicates my project a bit as far as Windows is concerned so I am now looking for alternatives to Josiah Carlson's asynchronous I/O patch for subprocess.Popen on Windows. If I cannot find an alternative, I will be using cTypes with the parts of Mark Hammond's code that I need, license permitting. Any suggestions are greatly appreciated.

My Original Proposal

My proposal was never posted so I am putting it up now.

Proposal

* A long running issue with the subprocess module is the handling of asynchronous io (http://bugs.python.org/issue1191964, http://ivory.idyll.org/blog/feb-07/problems-with-subprocess). Several patches have been proposed and several alternative solutions suggested. The patches and the alternative solutions will be implemented initially for 2.7 and then for 3.0. The subprocess test suite will be augmented to fully test this new functionality.

* Reimplement the commands module in terms of subprocess functions (http://ivory.idyll.org/blog/mar-07/replacing-commands-with-subprocess). Since the commands module will soon deprecated, this will provide a platform independent transition to those still using the commands module.


The modifications to the subprocess module will be presented initially for code review on Rietveld. A message will be added to the above referenced issues so that all interested parties can comment.

Schedule

Start of program:

* Implement command modules using subprocess
* Extend test suite
* Post code review



Midterm evaluation:

* Address issues from code review and post patch
* Synthesize the patches from the above links
* Extend test suite
* Post code review


Final evaluation:

* Address issues from code review and post patch

Who will police the police?

It appears the recent release of Python 3.0 have a buggy pdb. When I first ran into the issue, I thought it might have had something to do with my unit tests or possible incompatibilities between the subprocess.Popen module and pdb. I wrestled with it before finally throwing a bit of "Google-Fu" at it and discovered that I was not the only person having this issue. (http://bugs.python.org/issue6126)

Python 2.7 is supposed to be a 2.X compatible backport of the features and syntax found in Python 3.0 so any code added to the 2.7 branch must be either a bug fix or a backport of a Python 3.0 feature. Before I learned about that, my proposal was directed towards Python 2.7. The pdb module in Python 2.7 works flawlessly and did not give me any trouble while I was debugging my code with it.

Until the issues with the Python 3.0 pdb module are resolved, I will be taking advantage of the forward compatiblity and coding in Python 2.7 so that I will have a functional code debugger.

Google Code and License Conflicts

Until fairly recently, Google Code only supported Subversion code control publicly and Mercurial had to be specifically requested but now, Mercurial or Subversion can be used. Since I am using Mercurial to manage my branch of subprocess, this will make getting feedback on my code from the community much easier thanks to the large number of review features available.

As I was filling out the form to create a code repository, I ran into the issue of code licensing: Python has it's own unique license, the Python License, which is considered compatible with GPL according to the Free Software Foundation. Most of the code I am using comes from activestate.com and as such falls under an MIT license. After contacting the Python-Dev list, I am still at a bit of loss on selecting a license for the Google Code project and what impact the different licenses will have on being able to get my code pushed into the Python core.