Functools – Tools for working with functions

7 min


For future students of the course “Python Developer. Basic” prepared a translation of a useful article.


Module functools provides tools for working with functions and other callables in order to adapt or extend them for other purposes without completely rewriting them.

Decorators

The main tool that the module provides functools is the class partialwhich can be used to “wrap” callable objects with default arguments. The resulting object can also be called or treated like the original function. It accepts the same arguments as the original function and can be called with additional positional or named arguments.

partial

This example shows two simple partial object for function myfunc()… Note that the function show_details() outputs attributes func, args and partial keywords object.

import functools

def myfunc(a, b=2):
    """Docstring for myfunc()."""
    print 'tcalled myfunc with:', (a, b)
    return

def show_details(name, f, is_partial=False):
    """Show details of a callable object."""
    print '%s:' % name
    print 'tobject:', f
    if not is_partial:
        print 't__name__:', f.__name__
    print 't__doc__', repr(f.__doc__)
    if is_partial:
        print 'tfunc:', f.func
        print 'targs:', f.args
        print 'tkeywords:', f.keywords
    return

show_details('myfunc', myfunc)
myfunc('a', 3)
print

p1 = functools.partial(myfunc, b=4)
show_details('partial with named default', p1, True)
p1('default a')
p1('override b', b=5)
print

p2 = functools.partial(myfunc, 'default a', b=99)
show_details('partial with defaults', p2, True)
p2()
p2(b='override b')
print

print 'Insufficient arguments:'
p1()

At the end of the example, the first created partial called without passing a value to aby throwing an exception.

$ python functools_partial.py

myfunc:
        object: <function myfunc at 0x100468c08>
        __name__: myfunc
        __doc__ 'Docstring for myfunc().'
        called myfunc with: ('a', 3)

partial with named default:
        object: <functools.partial object at 0x10046b050>
        __doc__ 'partial(func, *args, **keywords) - new function with partial
 applicationn    of the given arguments and keywords.n'
        func: <function myfunc at 0x100468c08>
        args: ()
        keywords: {'b': 4}
        called myfunc with: ('default a', 4)
        called myfunc with: ('override b', 5)

partial with defaults:
        object: <functools.partial object at 0x10046b0a8>
        __doc__ 'partial(func, *args, **keywords) - new function with partial
 applicationn    of the given arguments and keywords.n'
        func: <function myfunc at 0x100468c08>
        args: ('default a',)
        keywords: {'b': 99}
        called myfunc with: ('default a', 99)
        called myfunc with: ('default a', 'override b')

Insufficient arguments:
Traceback (most recent call last):
  File "functools_partial.py", line 49, in <module>
    p1()
TypeError: myfunc() takes at least 1 argument (1 given)

update_wrapper

The partial object has no attributes _name_ or _doc_ by default, decorated functions are harder to debug without these attributes. Through update_wrapper() you can copy and add attributes from the original function to the partial object.

import functools

def myfunc(a, b=2):
    """Docstring for myfunc()."""
    print 'tcalled myfunc with:', (a, b)
    return

def show_details(name, f):
    """Show details of a callable object."""
    print '%s:' % name
    print 'tobject:', f
    print 't__name__:', 
    try:
        print f.__name__
    except AttributeError:
        print '(no __name__)'
    print 't__doc__', repr(f.__doc__)
    print
    return

show_details('myfunc', myfunc)

p1 = functools.partial(myfunc, b=4)
show_details('raw wrapper', p1)

print 'Updating wrapper:'
print 'tassign:', functools.WRAPPER_ASSIGNMENTS
print 'tupdate:', functools.WRAPPER_UPDATES
print

functools.update_wrapper(p1, myfunc)
show_details('updated wrapper', p1)

The attributes added to the wrapper are defined in functools.WRAPPER_ASSIGNMENTS, whereas functools.WRAPPER_UPDATES lists the values ​​to change.

$ python functools_update_wrapper.py

myfunc:
        object: <function myfunc at 0x100468c80>
        __name__: myfunc
        __doc__ 'Docstring for myfunc().'

raw wrapper:
        object: <functools.partial object at 0x10046c0a8>
        __name__: (no __name__)
        __doc__ 'partial(func, *args, **keywords) - new function with partial
 applicationn    of the given arguments and keywords.n'

Updating wrapper:
        assign: ('__module__', '__name__', '__doc__')
        update: ('__dict__',)

updated wrapper:
        object: <functools.partial object at 0x10046c0a8>
        __name__: myfunc
        __doc__ 'Docstring for myfunc().'

Other callable objects

Partial work with any object that can be called, not just individual functions.

import functools

class MyClass(object):
    """Demonstration class for functools"""
    
    def meth1(self, a, b=2):
        """Docstring for meth1()."""
        print 'tcalled meth1 with:', (self, a, b)
        return
    
    def meth2(self, c, d=5):
        """Docstring for meth2"""
        print 'tcalled meth2 with:', (self, c, d)
        return
    wrapped_meth2 = functools.partial(meth2, 'wrapped c')
    functools.update_wrapper(wrapped_meth2, meth2)
    
    def __call__(self, e, f=6):
        """Docstring for MyClass.__call__"""
        print 'tcalled object with:', (self, e, f)
        return

def show_details(name, f):
    """Show details of a callable object."""
    print '%s:' % name
    print 'tobject:', f
    print 't__name__:', 
    try:
        print f.__name__
    except AttributeError:
        print '(no __name__)'
    print 't__doc__', repr(f.__doc__)
    return
    
o = MyClass()

show_details('meth1 straight', o.meth1)
o.meth1('no default for a', b=3)
print

p1 = functools.partial(o.meth1, b=4)
functools.update_wrapper(p1, o.meth1)
show_details('meth1 wrapper', p1)
p1('a goes here')
print

show_details('meth2', o.meth2)
o.meth2('no default for c', d=6)
print

show_details('wrapped meth2', o.wrapped_meth2)
o.wrapped_meth2('no default for c', d=6)
print

show_details('instance', o)
o('no default for e')
print

p2 = functools.partial(o, f=7)
show_details('instance wrapper', p2)
p2('e goes here')

This example creates partial from an instance and instance methods.

$ python functools_method.py

meth1 straight:
        object: <bound method MyClass.meth1 of <__main__.MyClass object at
0x10046a3d0>>
        __name__: meth1
        __doc__ 'Docstring for meth1().'
        called meth1 with: (<__main__.MyClass object at 0x10046a3d0>, 'no d
efault for a', 3)

meth1 wrapper:
        object: <functools.partial object at 0x10046c158>
        __name__: meth1
        __doc__ 'Docstring for meth1().'
        called meth1 with: (<__main__.MyClass object at 0x10046a3d0>, 'a go
es here', 4)

meth2:
        object: <bound method MyClass.meth2 of <__main__.MyClass object at
0x10046a3d0>>
        __name__: meth2
        __doc__ 'Docstring for meth2'
        called meth2 with: (<__main__.MyClass object at 0x10046a3d0>, 'no d
efault for c', 6)

wrapped meth2:
        object: <functools.partial object at 0x10046c0a8>
        __name__: meth2
        __doc__ 'Docstring for meth2'
        called meth2 with: ('wrapped c', 'no default for c', 6)

instance:
        object: <__main__.MyClass object at 0x10046a3d0>
        __name__: (no __name__)
        __doc__ 'Demonstration class for functools'
        called object with: (<__main__.MyClass object at 0x10046a3d0>, 'no
default for e', 6)

instance wrapper:
        object: <functools.partial object at 0x10046c1b0>
        __name__: (no __name__)
        __doc__ 'partial(func, *args, **keywords) - new function with parti
al applicationn    of the given arguments and keywords.n'
        called object with: (<__main__.MyClass object at 0x10046a3d0>, 'e g
oes here', 7)

wraps

Updating the properties of the wrapped callable is especially useful when used in a decorator, because the converted function ends up with the properties of the original naked function.

import functools

def show_details(name, f):
    """Show details of a callable object."""
    print '%s:' % name
    print 'tobject:', f
    print 't__name__:', 
    try:
        print f.__name__
    except AttributeError:
        print '(no __name__)'
    print 't__doc__', repr(f.__doc__)
    print
    return

def simple_decorator(f):
    @functools.wraps(f)
    def decorated(a="decorated defaults", b=1):
        print 'tdecorated:', (a, b)
        print 't',
        f(a, b=b)
        return
    return decorated

def myfunc(a, b=2):
    print 'tmyfunc:', (a,b)
    return

show_details('myfunc', myfunc)
myfunc('unwrapped, default b')
myfunc('unwrapped, passing b', 3)
print

wrapped_myfunc = simple_decorator(myfunc)
show_details('wrapped_myfunc', wrapped_myfunc)
wrapped_myfunc()
wrapped_myfunc('args to decorated', 4)

Functools provides a decorator wraps()which applies update_wrapper() to the decorated function.

Comparison

Before Python 2, classes contained a method _cmp_()which returned -1, 0 or 1 depending on whether the object is less, equal, or greater than the one with which the comparison is made. Python 2.1 introduces a richer interface for comparison methods, _lt_(), _le_(), _eq_(), _ne_(), _gt_() and _ge_(), each of which performs one comparison operation and returns a boolean value. Removed in Python 3 cmp() in favor of these new methods, therefore functools provides tools to make it easier to write Python 2 classes that meet the new comparison requirements in Python 3.

Rich comparison

A rich comparison interface is implemented so that classes with complex comparisons can pass each test most efficiently. However, for classes where comparison is relatively straightforward, there is no point in manually creating each of the extended set methods. Class decorator total_ordering() takes a class that provides some of the methods and adds the missing ones.

import functools
import inspect
from pprint import pprint

@functools.total_ordering
class MyObject(object):
    def __init__(self, val):
        self.val = val
    def __eq__(self, other):
        print '  testing __eq__(%s, %s)' % (self.val, other.val)
        return self.val == other.val
    def __gt__(self, other):
        print '  testing __gt__(%s, %s)' % (self.val, other.val)
        return self.val > other.val

print 'Methods:n'
pprint(inspect.getmembers(MyObject, inspect.ismethod))

a = MyObject(1)
b = MyObject(2)

print 'nComparisons:'
for expr in [ 'a < b', 'a <= b', 'a == b', 'a >= b', 'a > b' ]:
    print 'n%-6s:' % expr
    result = eval(expr)
    print '  result of %s: %s' % (expr, result)

The class must provide implementation _eq_() and any other comparison method from the extended set. The decorator adds implementations of other methods that work using the provided comparisons.

$ python functools_total_ordering.py

Methods:

[('__eq__', <unbound method MyObject.__eq__>),
 ('__ge__', <unbound method MyObject.__ge__>),
 ('__gt__', <unbound method MyObject.__gt__>),
 ('__init__', <unbound method MyObject.__init__>),
 ('__le__', <unbound method MyObject.__le__>),
 ('__lt__', <unbound method MyObject.__lt__>)]

Comparisons:

a < b :
  testing __gt__(1, 2)
  testing __eq__(1, 2)
  result of a < b: True

a <= b:
  testing __gt__(1, 2)
  result of a <= b: True

a == b:
  testing __eq__(1, 2)
  result of a == b: False

a >= b:
  testing __gt__(1, 2)
  testing __eq__(1, 2)
  result of a >= b: False

a > b :
  testing __gt__(1, 2)
  result of a > b: False

The sort order

Since the old comparison function is no longer used in Python 3, the argument cmp is no longer supported by features such as sort()… Python 2 programs that use comparison functions can use cmp_to_key() to convert them to a function that returns the sort key, which is used to determine the position in the final sequence.

import functools

class MyObject(object):
    def __init__(self, val):
        self.val = val
    def __str__(self):
        return 'MyObject(%s)' % self.val

def compare_obj(a, b):
    """Old-style comparison function.
    """
    print 'comparing %s and %s' % (a, b)
    return cmp(a.val, b.val)

# Make a key function using cmp_to_key()
get_key = functools.cmp_to_key(compare_obj)

def get_key_wrapper(o):
    """Wrapper function for get_key to allow for print statements.
    """
    new_key = get_key(o)
    print 'key_wrapper(%s) -> %s' % (o, new_key)
    return new_key

objs = [ MyObject(x) for x in xrange(5, 0, -1) ]

for o in sorted(objs, key=get_key_wrapper):
    print o

Note: usually, cmp_to_key() is used directly, but in this example an additional function wrapper is needed in order to display more detailed information about how the key function is called.

The output shows that sorted() starts with a call get_key_wrapper() for each item in the sequence to get the key. Keys returned cmp_to_key(), are instances of the class defined in functoolswhich implements the extended comparison interface based on the return value of the old-style compare function. After all the keys are obtained, the sequence is sorted by comparing the keys.

$ python functools_cmp_to_key.py

key_wrapper(MyObject(5)) -> <functools.K object at 0x100466558>
key_wrapper(MyObject(4)) -> <functools.K object at 0x100466590>
key_wrapper(MyObject(3)) -> <functools.K object at 0x1004665c8>
key_wrapper(MyObject(2)) -> <functools.K object at 0x100466600>
key_wrapper(MyObject(1)) -> <functools.K object at 0x100466638>
comparing MyObject(4) and MyObject(5)
comparing MyObject(3) and MyObject(4)
comparing MyObject(2) and MyObject(3)
comparing MyObject(1) and MyObject(2)
MyObject(1)
MyObject(2)
MyObject(3)
MyObject(4)
MyObject(5)

More details about the course “Python Developer. Basic”. You can watch an open lesson on the topic “Pytest: An Introduction to Autotests” here.


0 Comments

Leave a Reply