Sunday, February 26, 2006

Python Compiler Sessions

We had a birds of a feather (BOF) session about the new Python bytecode compiler. We had a large crowd -- as many as 50 people. The discussion was open-ended and we didn't make many specific decisions.

One result was to come up with a list of tasks that should be completed to make the AST useful for Python programmers:
  • We need generic API to child nodes in AST. This ai would have a function to get all the child nodes. The current API exposes only named attributes for specific children.
  • It would be great if the AST could be used in conjunction with tools that do source-to-source transformations. This use requires an association between the AST and raw tokens (and comments, too). John Ehresman suggested adding a token range as an annotation on the AST.
  • Users who tried the old compiler package found that tree walking pure Python was too slow. We should write a generic walker in C that can be extended with Python functions to call for specific nodes, presumably with pre-order, post-order, and in-order traversals. Performance is important in practice.
We collected several concrete use cases for the AST. People who attended the BOF wanted to:
  • Restrict the scope of Python programs, along the lines of the compiler for Zope's "Python Scripts"
  • Parsing source to generate documentation.
  • Generate SQL, Verilog statements from Python code. Generally, I think, things along the lines of LINQ.
  • Optimizers
  • Branch coverage analysis
The compiler package has always been hard to maintain. In principle, I like having a pure Python compiler. In practice, it is tiresome to maintain two different compilers when one is used infrequently. Several key problems for the 2.5 release are:
  • The AST is uses is different than the builtin AST.
  • There are bugs in the 2.4 implementation, particularly in namespace resolution. (Why didn't someone assign the bug to me?)
  • The compiler package doesn't have a good test suite.
There was some question about whether to label the compiler package as experimental or deprecated, but we did not decide.

Earlier in the morning, I gave a talk on the new bytecode compiler -- a whirlwind tour intended to help new developers get up to speed. I hadn't prepared adequately, so I went through some of the material too quickly. I got a number of good questoins. Brett and Steve suggest it went pretty well.

No comments: