Pex
Pex is a preprocessor for Cython developed at PnyLab LLC.
Numerics
Pex provides important experience and Cython will provide almost identical capabilities, but through a more general mechanism: Supporting the buffer PEP natively.
Auto-pickling and dictionary coercion
Automatically generating __reduce__ and __setstate__ is a Cython goal and is already done by Pex.
Dictionary coercion is a nice idea that Cython should copy:
- Pointers cannot be represented into dicts. Pex has a concept of a "modest class" which doesn't have pointers or structs, which can then be represented as dicts. In Cython we may opt for simply ignoring classes that cannot be represented in a dict, or present a run-time error, or provide a mechanism for classes to present pickleable versions of the attributes (e.g. via the property interface).
- structs are not represented in dicts in Pex, although we could in time do this in Cython by creating wrapper objects around the structs on fly
Pragmas
Pex has a pragma syntax that Cython will probably not adopt; although the feature seems to be very valuable. One can either polish the syntax a bit, or fine some subset of module header comments, decorators and with statements. Command-line pragmas should of course be supported. (Config files? Isn't that what setup.py is?) Perhaps syntax similar to specifying the encoding PEP 263
Useful pragmas for safer mode:
- Bounds checking on buffer interface
- Check for uninitialized variable when doing C attribute/method access
- (Cycle detection?)
For speed:
- Drop type-checking when accessing typed Python object buffers
Exception handling
Dan made a very good point that exceptions should always be propagated, at least by default, rather than strange behaviour being the default. We'll look into this.
Faster serialization
Pex is able to automatically create fast dump/load methods for simple classes. While useful, this seems to be more in the application domain than language domain because it is so tightly coupled with trade-offs determined by the type of data, type of inter-object references etc.. At the same time, it's a very useful feature. Ways this can be handled:
- FastIO (from Pex) can perhaps be ported as a Cython plugin module
- Metaprogramming would solve this in a generic way, allowing one to have multiple kinds of such serialization
- One could look into existing C libraries for this and provide hooks at the right API level
pxd generation
Pex has a simpler module concept: All files as "px" files which serves the role both as pxds and pyxs for a Cython module. In Cython we'd like to tweak that into this:
- If no pxd file is present, we auto-generate one (in memory, not written out) from the pyx file
- (If this is not efficient, the user can always continue to provide one, and this can be disabled from a flag)
- One could for instance only export symbols declared "public" or "api" into this auto-pxd
- Optional import = cimport
cimport vs. import
We would phase out cimport, letting cimport automatically be an import if the pxd (or, as above, pyx) file can be located.
Automatic str, cmp and eq overloads
Cython didn't reach a conclusion; it's the kind of things that are nice to have while they don't fit into emulating Python. Metaprogramming would solve this too ("compile-time metaclass support"?) or a method generator of sorts
Build system
Pex has system that mimics the Python import statement; one can simply invoke pex on a px sourcefile (or use the pex module from Python to import a px file) and pex will automatically rebuild the appropriate files and import them.
There could be potential for direct code reuse in this area.
- The modules gets hooks added to do sanity checks for .so file time stamps, so that Python import won't coredump if there's a build problem.
- The Python pex module does magic stack inspection in order to do the right import, might want to look at that
Build path control
Sourcefiles in pex can carry information about their include paths. Cython may provide something similar but in a much more declarative syntax.