CEP 201 - Distutils Preprocessing
Status: Draft
Implementation status: Much has been implemented in Sage's setup.py
Discussion:
Concept
Distutils is the standard way to build Cython extension modules for large projects. Typically one invokes setup(...) which then indirectly invokes Cython when a .pyx source file is encountered. This limits the control we have over the pyx to c compilation process. This proposal is to make a standard Cython method that "preparses" the list of Extension modules and converts all .pyx source files to .c(pp) ones before passing them off to distutils. Doing this step ourselves, all at once, would have several advantages including:
- Cython-aware dependancy tracking
- in-file directives for language, libraries, compiler flags, etc. (which could be transitive)
- efficient multi-file compilation (especially with shared deep cimports)
sharing common utility code in a single .h file
- shared cimported type declaration code (?)
- whole-program type inference or other analysis
This will resolve other nuisances such as the difficulty in supporting parallel Pyrex and Cython installs, or relying on distutils to know about Cython at all. Disentangling the Cythonization phase from the compile process and using in-file directives will allow easier support for other build systems as well.
Example
Rather than writing a setup.py that looks like
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
ext_modules=[
Extension("primes", ["primes.pyx"]),
Extension("spam", ["spam.pyx"]),
...
]
setup(
name = 'MyProject',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules,
)one would write
from distutils.core import setup
from distutils.extension import Extension
from Cython.build import cythonize # names subject to change
ext_modules=[
Extension("primes", ["primes.pyx"]),
Extension("spam", ["spam.pyx"]),
...
]
setup(
name = 'MyProject',
ext_modules = cythonize(ext_modules),
)or possibly even
ext_modules=[
Extension("*", ["*.pyx"]),
]and
ext_modules=["primes.pyx", "spam.pyx", ...]
Issues
* We would still have to read and understand sys.argv to know whether or not to invoke Cython.