The Mutator Interface
Author: FabrizioM
- Status: Prototype/Unpublished
This document describes the steps involved in implementing the with_stmt using the new Cython AST.
First I describe the Mutator interface, then an example of its usage for the with_stmt
Mutator Interface
each AST Node have a method "mutator" that accepts a visitor as argument. i.e for the Stmt Node representing a list of statements:
#in ast.py
class Stmt(Node):
#...
def mutate(self, visitor):
visitor._mutate_list(self.nodes)
return visitor.visitStmt(self)
#...Stmt allows the visitor object to modify his sub-nodes calling visitor._mutate_list(self.node) Where the _mutate_list is implemented as follows:
1 class ASTVisitor(object):
2 #...
3 def _mutate_list(self, lst):
4 i = 0
5 while i < len(lst):
6 item = lst[i].mutate(self)
7 if item is not None:
8 lst[i] = item
9 i += 1
10 else:
11 del lst[i]
12
13 #...
Reading the above code is clear that you can cut an entire branch of the tree simply by returning None in an AstVisitor. for example to remove all the Discard Nodes:
class AstVisitorRemoveDiscardNode:
def visitDiscard(self, node):
return None== A simple Example === Let's look at a concrete example: the With statement introduced in the Python2.5 Version PEP0343
with EXPR as VAR:
BLOCKWe know from the pep that the above code is equal to:
1 # withstmt.txt
2 mgr = (EXPR)
3 exit = mgr.__exit__ # Not calling it yet
4 value = mgr.__enter__()
5 exc = True
6 try:
7 try:
8 VAR = value # Only if "as VAR" is present
9 BLOCK
10 except:
11 # The exceptional case is handled here
12 exc = False
13 if not exit(*sys.exc_info()):
14 raise
15 # The exception is swallowed if exit() returns true
16 finally:
17 # The normal and non-local-goto cases are handled here
18 if exc:
19 exit(None, None, None)
since we use the same ast nodes of the existing Python 2.5 we can generate the Ast from the above doing:
1 >>>import compiler
2 >>>compiler.parse( open('withstmt.txt').read() )
3 Module(None,
4 Stmt([Assign([AssName('mgr', 'OP_ASSIGN')], Name('EXPR')),
5 Assign([AssName('exit', 'OP_ASSIGN')], Getattr(Name('mgr'), '__exit__')),
6 Assign([AssName('value', 'OP_ASSIGN')], CallFunc(Getattr(Name('mgr'), '__enter__'), [], None, None)),
7 Assign([AssName('exc', 'OP_ASSIGN')], Name('True')),
8 TryFinally(
9 Stmt([
10 TryExcept(
11 Stmt([Assign([AssName('VAR', 'OP_ASSIGN')], Name('value')),
12 Discard(Name('BLOCK'))]), [(None, None, Stmt([Assign([AssName('exc', 'OP_ASSIGN')], Name('False')),
13 If([(Not(CallFunc(Name('exit'), [], CallFunc(Getattr(Name('sys'), 'exc_info'), [], None, None), None)),
14 Stmt([Raise(None, None, None)]))], None)]))], None)]),
15 Stmt([If([(Name('exc'), Stmt([Discard(CallFunc(Name('exit'), [Name('None'), Name('None'), Name('None')], None, None))]))], None)]))]))
Since all the above is valid code we can copy and paste that tree in our mutator
1 class With(Node):
2 def __init__(self, expr, vars, body, lineno=None):
3 self.expr = expr
4 self.vars = vars
5 self.body = body
6 self.lineno = lineno
7
8 #...
9 class WithMutator(AstVisitor):
10
11 def visitWith(self, node):
12 #since With is also a stmt we can return a stmt
13 WITH_REMOD = Stmt([Assign([AssName('mgr', 'OP_ASSIGN')], node.expr), # NOTE here we must provide the actual with expr
14 ..
15 Assign([AssName('value', 'OP_ASSIGN')], CallFunc(Getattr(Name('mgr'), '__enter__'), [], None, None)),
16 Assign([AssName('exc', 'OP_ASSIGN')], Name('True')),
17 TryFinally(Stmt([TryExcept(
18
19 Stmt([Assign([AssName('VAR', 'OP_ASSIGN')], Name('value')),
20
21 node.body]), # NOTE substitute block
22
23 [(None, None, Stmt([Assign([AssName('exc', 'OP_ASSIGN')], Name('False')),
24 If([(Not(CallFunc(Name('exit'), [], CallFunc(Getattr(Name('sys'), 'exc_info'), [], None, None), None)),
25 Stmt([Raise(None, None, None)]))], None)]))], None)]),
26 Stmt([If([(Name('exc'), Stmt([Discard(CallFunc(Name('exit'),
27 [Name('None'), Name('None'), Name('None')], None, None))]))], None)]))]))
28
29 # .. update linenumber from node.lineno
30 # .. other checks
31 return WITH_REMOD; # return the new subtree
ERROR: EOF in multi-line statement
and pass it to the ast
1 ast.mutate (WithMutator())
now using the esisting Cython Visitor you just do:
1 ast.visit(CythonVisitor())
and that's it!
Summary
We have seen the proposed mutator interface for the new Ast, how it works and how we could handle the With Stmt .
Note: the mutator interface and the ast code is largely inspired from the Pypy project
Comments
Looks good! A couple of minor comments.
CarlWitty: 1) I don't like the _mutate_list interface, because it feels like the list is an internal property of the AST that shouldn't be exposed to the visitor. I would prefer something like
self.nodes = visitor._mutate_list(self.nodes[:])
Of course, in this case, the only effect of my proposed change is to make things a little bit slower, so feel free to ignore me...
FabrizioM You are right. An internal property of the AST must not be exposed to the visitor. But this is a mutator, has a different contract with the Node interface. Visitor and Mutator shares only the interface.
CarlWitty: 2) A real implementation would need to come up with new names, instead of always using 'value', 'exc', 'exit'. I'm curious how a slightly more realistic example incorporating this requirement would look.
Fabrizio: Well since the WITH_REMOD is itself an ast tree you can do something like:
1 class AstMutatorRenamer(AstVisitor):
2 def __init__(self, name_space, *args):
3 self.name_space = name_space
4 #...
5 def rename (self, name):
6 while name in self.name_space:
7 name = '_' + name
8 return name
9
10 def visitName(self, node):
11 node.name = self.reaname(node.name)
12 return node
13
14 class WithMutator(AstVisitor):
15
16 def visitWith(self, node):
17 #...
18 WITH_REMOD.mutate (AstMutatorRenamer())
19 #...