Python Compiler Sessions

We had a birds of a feather (BOF) session about the new Python bytecode compiler. We had a large crowd -- as many as 50 people. The discussion was open-ended and we didn't make many specific decisions.

One result was to come up with a list of tasks that should be completed to make the AST useful for Python programmers:

• We need generic API to child nodes in AST. This ai would have a function to get all the child nodes. The current API exposes only named attributes for specific children.
• It would be great if the AST could be used in conjunction with tools that do source-to-source transformations. This use requires an association between the AST and raw tokens (and comments, too). John Ehresman suggested adding a token range as an annotation on the AST.
• Users who tried the old compiler package found that tree walking pure Python was too slow. We should write a generic walker in C that can be extended with Python functions to call for specific nodes, presumably with pre-order, post-order, and in-order traversals. Performance is important in practice.

We collected several concrete use cases for the AST. People who attended the BOF wanted to:

• Restrict the scope of Python programs, along the lines of the compiler for Zope's "Python Scripts"
• Parsing source to generate documentation.
• Generate SQL, Verilog statements from Python code. Generally, I think, things along the lines of LINQ.
• Optimizers
• Branch coverage analysis

The compiler package has always been hard to maintain. In principle, I like having a pure Python compiler. In practice, it is tiresome to maintain two different compilers when one is used infrequently. Several key problems for the 2.5 release are:

• The AST is uses is different than the builtin AST.
• There are bugs in the 2.4 implementation, particularly in namespace resolution. (Why didn't someone assign the bug to me?)
• The compiler package doesn't have a good test suite.

There was some question about whether to label the compiler package as experimental or deprecated, but we did not decide.

Earlier in the morning, I gave a talk on the new bytecode compiler -- a whirlwind tour intended to help new developers get up to speed. I hadn't prepared adequately, so I went through some of the material too quickly. I got a number of good questoins. Brett and Steve suggest it went pretty well.