*********************************************************** Objects, Classes and Metaclasses - Above and Under the Hood *********************************************************** :Author: Beni Cherniavsky :Presented: Python-IL, 2007-03-28 .. class:: handout .. contents:: Agenda Part I: Pure-Python World ========================= .. figure:: Escher/Mobius_Strip_I.jpg :width: 60% :align: center Mobius Strip I, M.C.Escher instance — Class ---------------- `instance.__class__` is `type(instance)` is `Class`. * ``instance(...)`` * ``Class.__call__(instance, ...)`` * ``instance.attr`` * ``Class.__getattribute__(instance, 'attr')`` * ``object.__getattribute__(instance, 'attr')`` * ``Class.attr.__get__(instance, Class)`` * ``instance.__dict__['attr']`` * ``Class.attr`` * ``Class.__getattr__(instance, 'attr')`` A new-style instance is a puppet — and the Class pulls the strings. Class — metaclass (`type`) -------------------------- `Class.__class__` is `type(Class)` is `type`. * ``Class(...)`` * ``type.__call__(Class, ...)`` * ``Class.__new__(Class, ...)`` * ``object.__new__(Class, ...)`` * ``Class.__init__(self, ...)`` * ``Class.attr`` * ``type.__getattribute__(Class, 'attr')`` * ``Class.attr.__get__(None, Class)`` * ``Class.__dict__['attr']`` * ``Class.__base__.attr`` # actually MRO lookup A new-style Class is basically just a puppet factory. Example: time property ---------------------- A `property` returns self on class access: >>> import time >>> time_prop = property(lambda self: time.time()) >>> class Class(object): ... time = time_prop ... >>> Class.time but calls the getter function on instance access: >>> instance = Class() >>> instance.time 1175029167.593503 >>> instance.time 1175029168.477463 `type` instantiation -------------------- A ``class`` statement simply instantiates `type`: * ``class C(object): x = 1`` * ``Class = type('Class', (object,), {'x': 1})`` We can do it ourselves: >>> Class = type('Class', (object,), dict(time=time_prop)) >>> Class().time 1175029698.4495051 Example: class decorator ------------------------ This allows us to write a "class decorator":: def class_decorator(C): return type(C.__name__, C.__bases__, dict(C.__dict__, time=time_prop)) >>> class Class(object): ... "will magically tells the time" >>> Class = class_decorator(Class) >>> Class().time 1175029821.2147801 This gives some power, but ``Class.__class__`` is still `type`. Subclassing `type` ------------------ If we instantiate a subclass of `type`, it becomes the metaclass: >>> class TimedType(type): ... time = time_prop ... >>> TimedClass = TimedType('TimedClass', (object,), {'x': 1}) >>> TimedClass.__class__ >>> TimedType.time >>> TimedClass.time 1175030904.9948061 >>> TimedClass().time Traceback (most recent call last): TimedClass().time AttributeError: 'TimedClass' object has no attribute 'time' __class__ / __base__ relations ------------------------------ @@@ Move below? .. figure:: Escher/Liberation.jpg :width: 30% :align: right Liberation, M.C.Escher ``class`` statement: explicit metaclass --------------------------------------- The member `__metaclass__` (if given) is called to construct the class:: class TimedClass(object): def __metaclass__(name, bases, members): return type(C.__name__, C.__bases__, dict(C.__dict__, time=time_prop)) But usually it will be a class: class TimedClass(object): __metaclass__ = TimedType or directly: class TimedClass(object): class __metaclass__(type): time = time_prop ``class`` statement: inherited metaclass ---------------------------------------- * ``class Class(object):`` instantiates `type` because `object.__class__` is `type`. * When there are no base classes, the global `__metaclass__` is tried. So ``__metaclass__ = type`` at the beginning of a module makes all classes newstyle. * The last fallback is `types.ClassType` (the classic types metaclass). This will change to `object` in Python 3000. * If `TimedClass.__class__` is `TimedType`: >>> class SubClass(TimedClass): ... pass ... >>> SubClass.__class__ >>> SubClass.time 1175057391.6200049 * Multiple inheritance works but with limitations but leads to the dark side... * Usually, the most convenient interface for a metaclass is to inherit from a base class using it. Metaclass uses -------------- .. image:: Escher/Tower_of_Babel.jpg :width: 30% :align: right .. Tower of Babel, M.C.Escher * What: * Validation * Registration * Wrapping / Adaptation * Order-stamping trick * Other deep magic * Why: * Hiding bugs (replacing them by very subtle ones?) * Classes that are also objects * Declarative programming * Domain specific languages * Don't Repeat Yourself! Use in moderation! "A day without objects is a day without job security." Example: Enum ------------- :: class Enum(int): class __metaclass__(type): def __new__(metaclass, name, bases, members): Class = type.__new__(metaclass, name, bases, members) Class._names = {} for k, v in members.items(): if isinstance(v, (int, long)): setattr(Class, k, Class(v)) Class._names[v] = k return Class def __str__(self): return '%s.%s' % (self.__class__.__name__, self._names.get(self, int(self))) >>> class FooBar(Enum): ... FOO = 0 ... BAR = 1 >>> print FooBar(0), FooBar.BAR FooBar.FOO FooBar.BAR Example: inheriting dictionaries -------------------------------- :: class Dict(object): class __metaclass__(type): # , UserDict.DictMixin __getitem__ = type.__getattribute__ __setitem__ = type.__setattr__ >>> class A(Dict): ... foo = 1 >>> class B(A): ... bar = 2 >>> B['bar'] 2 >>> B['foo'] 1 >>> A['foo'] = 0 >>> B['foo'] 0 Python 3000 ----------- :PEP:`3115` * ``class Class(*bases, metaclass=MetaClass, **kwargs): BODY`` 1. *Before* the class body is executed: * ``locals_dict = MetaClass.__prepare__('Class', bases, **kwargs)`` * ``locals_dict = {}`` 2. ``exec '''BODY''', globals(), locals_dict`` 3. ``Class = MetaClass('Class', bases, locals_dict)`` as before The main use case is a ``locals_dict`` that remembers the order of assignments. Questions: and what does the turtle stand on? --------------------------------------------- "You are very clever, young man, very clever. But it is turtles all the way down!" -- An old lady unconvinced by Bertrand Russell's astronomy lecture. * ``instance.__class__`` * ``instance.__class__.__getattribute__('__class__')``? * ...? * ``instance.__dict__['attr']`` * ``Class.__getattribute__('__dict__')``? * ...? * ``instance.attr`` * ``Class.__getattribute__(instance, 'attr')`` * ``type.__getattribute__(Class, '__getattribute__')``? * ...? Part II: Wake up, Neo! ====================== .. figure:: Escher/Dream.jpg :width: 40% :align: center Dream, M.C.Escher The Pytrix has you! ------------------- * All the lookup rules discussed above (``__getattribute__`` &c.) are an illusion maintained inside the "Python Matrix". * Outside the Pytrix, every Python object is represented by an object of the underlying reality: * `struct PyObject`` and related APIs in CPython. * `org.python.core.PyObject` in Jython. * "Object space" Python code simulating "Application space" Python code in PyPy. * ... * A capsule with nourishing liquid in far-future ZeroOneThon ?-; We will focus on Jython_ in this lecture. .. _Jython: www.jython.org The illusion of free will ------------------------- .. image:: Escher/High_and_Low.jpg :width: 30% :align: right * Every operation inside the Pytrix is a method call on the implemeting object. * This method is written and called in Java and is found directly, and cannot be hooked from inside the Pytrix. * The method is called on the object simulating the instance, not the class. Here we are dealing with a Pytrix — implementation relation, and don't yet have an instance — Class relation. * This method *MAY* call hooks inside the Pytrix. * `PyObject` happens to call ``Class.__getattribute__`` inside the Pytrix, creating the illusion inside the Pytrix that every attribute access works this way. Autonomous ajents ----------------- "It's a turtle, for heaven's sake. It swims. That's what turtles are for." -- `Small Gods`:title-reference:, Terry Pratchett * Some objects are native: operations on them are implemented in Java and don't call back into the Pytrix. * Functions implement __call__ natively. * `object` and `type` (or to be precise, `Py.ObjectType` and `Py.TypeType`) implement attribute access natively. * Subclasses of them do call back into the Pytrix - but what we called ``.__class__`` and ``.__dict__`` are always native ``.getType()`` and ``.getDict()`` calls. * This is what stops the recursion. * For consistency, such objects expose their native method as __magic__ methods inside the Pytrix. * See also :PEP:`252` and :PEP:`253` for the CPython specification and implementation overview. instance_PyObject — Class_PyType -------------------------------- `instance_PyObject.getType()` is `Class_PyType`. * Inside the Pytrix: ``instance(...)`` * ``instance_PyObject.__call__(...)`` * ``Class_PyType.__findattr__('__call__').__call__()`` * ``instance.attr`` * ``instance_PyObject.__findattr__('attr')`` * ``Class_PyType.__findattr__('attr').__findattr__('__get__')`` * ``instance_PyObject.getDict().__finditem__('attr')`` * ``Class_PyType.lookup('attr')`` * ``Class_PyType.__findattr__('__getattr__').__call__(instance_PyObject, 'attr')`` More about Jython ----------------- * The last "stable" release was 2.1 years ago. * Use 2.2b1 or SVN head! * Compiles to Java bytecode. * The beauty is that Java supports introspection ("reflection"), so you can just import any Java module from Jython and call it right away. * You can subclass in Python from Java classes! * Not sure about the other way around, * You can implement Java protocols from Python, which effective means you can supply Python callbacks to Java modules. Java Reflection --------------- >>> from org.python.core import PyObject, PyType .. figure:: Escher/Hand_with_Reflecting_Globe.jpg :width: 30% :align: right Hand with Reflecting Globe, M.C.Escher Escaping the Pytrix ------------------- .. figure:: Escher/Print_Gallery.jpg :width: 60% :align: center Print Gallery, M.C.Escher Questions? ---------- .. figure:: Escher/Snakes.jpg :width: 60% :align: center Snakes, M.C.Escher Credits ------- .. image:: GEB/GEBcover.jpg :width: 30% :align: right * Once I was asked "why metaclasses don't work in Jython 2.1?". * For two weeks I stared at a page of code and then added 2 lines. Perhaps the most profound debugging experience I ever had. * The same 2 lines have been added during the same weeks in Jython's CVS :-). * Moshe Zadka's PyPy lecture at August Penguin 2003. * `Gödel, Escher, Bach - an Ethernal Golden Braid`:title-reference: by Douglas R. Hofstader, 1979 * "In a way, this book is a statement of my religion" * #1 on the Jargon File bibliography since 1990 ;-) * This lecture was composed in parallel to reading GEB between patrols round an army base. If this lecture had any style, it came from the book.