Functools – Tools for working with functions
For future students of the course “Python Developer. Basic” prepared a translation of a useful article.
Module functools provides tools for working with functions and other callables in order to adapt or extend them for other purposes without completely rewriting them.
Decorators
The main tool that the module provides functools is the class partial
which can be used to “wrap” callable objects with default arguments. The resulting object can also be called or treated like the original function. It accepts the same arguments as the original function and can be called with additional positional or named arguments.
partial
This example shows two simple partial object for function myfunc()
… Note that the function show_details()
outputs attributes func
, args
and partial keywords object.
import functools
def myfunc(a, b=2):
"""Docstring for myfunc()."""
print 'tcalled myfunc with:', (a, b)
return
def show_details(name, f, is_partial=False):
"""Show details of a callable object."""
print '%s:' % name
print 'tobject:', f
if not is_partial:
print 't__name__:', f.__name__
print 't__doc__', repr(f.__doc__)
if is_partial:
print 'tfunc:', f.func
print 'targs:', f.args
print 'tkeywords:', f.keywords
return
show_details('myfunc', myfunc)
myfunc('a', 3)
print
p1 = functools.partial(myfunc, b=4)
show_details('partial with named default', p1, True)
p1('default a')
p1('override b', b=5)
print
p2 = functools.partial(myfunc, 'default a', b=99)
show_details('partial with defaults', p2, True)
p2()
p2(b='override b')
print
print 'Insufficient arguments:'
p1()
At the end of the example, the first created partial
called without passing a value to a
by throwing an exception.
$ python functools_partial.py
myfunc:
object: <function myfunc at 0x100468c08>
__name__: myfunc
__doc__ 'Docstring for myfunc().'
called myfunc with: ('a', 3)
partial with named default:
object: <functools.partial object at 0x10046b050>
__doc__ 'partial(func, *args, **keywords) - new function with partial
applicationn of the given arguments and keywords.n'
func: <function myfunc at 0x100468c08>
args: ()
keywords: {'b': 4}
called myfunc with: ('default a', 4)
called myfunc with: ('override b', 5)
partial with defaults:
object: <functools.partial object at 0x10046b0a8>
__doc__ 'partial(func, *args, **keywords) - new function with partial
applicationn of the given arguments and keywords.n'
func: <function myfunc at 0x100468c08>
args: ('default a',)
keywords: {'b': 99}
called myfunc with: ('default a', 99)
called myfunc with: ('default a', 'override b')
Insufficient arguments:
Traceback (most recent call last):
File "functools_partial.py", line 49, in <module>
p1()
TypeError: myfunc() takes at least 1 argument (1 given)
update_wrapper
The partial object has no attributes _name_
or _doc_
by default, decorated functions are harder to debug without these attributes. Through update_wrapper()
you can copy and add attributes from the original function to the partial object.
import functools
def myfunc(a, b=2):
"""Docstring for myfunc()."""
print 'tcalled myfunc with:', (a, b)
return
def show_details(name, f):
"""Show details of a callable object."""
print '%s:' % name
print 'tobject:', f
print 't__name__:',
try:
print f.__name__
except AttributeError:
print '(no __name__)'
print 't__doc__', repr(f.__doc__)
print
return
show_details('myfunc', myfunc)
p1 = functools.partial(myfunc, b=4)
show_details('raw wrapper', p1)
print 'Updating wrapper:'
print 'tassign:', functools.WRAPPER_ASSIGNMENTS
print 'tupdate:', functools.WRAPPER_UPDATES
print
functools.update_wrapper(p1, myfunc)
show_details('updated wrapper', p1)
The attributes added to the wrapper are defined in functools.WRAPPER_ASSIGNMENTS
, whereas functools.WRAPPER_UPDATES
lists the values to change.
$ python functools_update_wrapper.py
myfunc:
object: <function myfunc at 0x100468c80>
__name__: myfunc
__doc__ 'Docstring for myfunc().'
raw wrapper:
object: <functools.partial object at 0x10046c0a8>
__name__: (no __name__)
__doc__ 'partial(func, *args, **keywords) - new function with partial
applicationn of the given arguments and keywords.n'
Updating wrapper:
assign: ('__module__', '__name__', '__doc__')
update: ('__dict__',)
updated wrapper:
object: <functools.partial object at 0x10046c0a8>
__name__: myfunc
__doc__ 'Docstring for myfunc().'
Other callable objects
Partial work with any object that can be called, not just individual functions.
import functools
class MyClass(object):
"""Demonstration class for functools"""
def meth1(self, a, b=2):
"""Docstring for meth1()."""
print 'tcalled meth1 with:', (self, a, b)
return
def meth2(self, c, d=5):
"""Docstring for meth2"""
print 'tcalled meth2 with:', (self, c, d)
return
wrapped_meth2 = functools.partial(meth2, 'wrapped c')
functools.update_wrapper(wrapped_meth2, meth2)
def __call__(self, e, f=6):
"""Docstring for MyClass.__call__"""
print 'tcalled object with:', (self, e, f)
return
def show_details(name, f):
"""Show details of a callable object."""
print '%s:' % name
print 'tobject:', f
print 't__name__:',
try:
print f.__name__
except AttributeError:
print '(no __name__)'
print 't__doc__', repr(f.__doc__)
return
o = MyClass()
show_details('meth1 straight', o.meth1)
o.meth1('no default for a', b=3)
print
p1 = functools.partial(o.meth1, b=4)
functools.update_wrapper(p1, o.meth1)
show_details('meth1 wrapper', p1)
p1('a goes here')
print
show_details('meth2', o.meth2)
o.meth2('no default for c', d=6)
print
show_details('wrapped meth2', o.wrapped_meth2)
o.wrapped_meth2('no default for c', d=6)
print
show_details('instance', o)
o('no default for e')
print
p2 = functools.partial(o, f=7)
show_details('instance wrapper', p2)
p2('e goes here')
This example creates partial from an instance and instance methods.
$ python functools_method.py
meth1 straight:
object: <bound method MyClass.meth1 of <__main__.MyClass object at
0x10046a3d0>>
__name__: meth1
__doc__ 'Docstring for meth1().'
called meth1 with: (<__main__.MyClass object at 0x10046a3d0>, 'no d
efault for a', 3)
meth1 wrapper:
object: <functools.partial object at 0x10046c158>
__name__: meth1
__doc__ 'Docstring for meth1().'
called meth1 with: (<__main__.MyClass object at 0x10046a3d0>, 'a go
es here', 4)
meth2:
object: <bound method MyClass.meth2 of <__main__.MyClass object at
0x10046a3d0>>
__name__: meth2
__doc__ 'Docstring for meth2'
called meth2 with: (<__main__.MyClass object at 0x10046a3d0>, 'no d
efault for c', 6)
wrapped meth2:
object: <functools.partial object at 0x10046c0a8>
__name__: meth2
__doc__ 'Docstring for meth2'
called meth2 with: ('wrapped c', 'no default for c', 6)
instance:
object: <__main__.MyClass object at 0x10046a3d0>
__name__: (no __name__)
__doc__ 'Demonstration class for functools'
called object with: (<__main__.MyClass object at 0x10046a3d0>, 'no
default for e', 6)
instance wrapper:
object: <functools.partial object at 0x10046c1b0>
__name__: (no __name__)
__doc__ 'partial(func, *args, **keywords) - new function with parti
al applicationn of the given arguments and keywords.n'
called object with: (<__main__.MyClass object at 0x10046a3d0>, 'e g
oes here', 7)
wraps
Updating the properties of the wrapped callable is especially useful when used in a decorator, because the converted function ends up with the properties of the original naked function.
import functools
def show_details(name, f):
"""Show details of a callable object."""
print '%s:' % name
print 'tobject:', f
print 't__name__:',
try:
print f.__name__
except AttributeError:
print '(no __name__)'
print 't__doc__', repr(f.__doc__)
print
return
def simple_decorator(f):
@functools.wraps(f)
def decorated(a="decorated defaults", b=1):
print 'tdecorated:', (a, b)
print 't',
f(a, b=b)
return
return decorated
def myfunc(a, b=2):
print 'tmyfunc:', (a,b)
return
show_details('myfunc', myfunc)
myfunc('unwrapped, default b')
myfunc('unwrapped, passing b', 3)
print
wrapped_myfunc = simple_decorator(myfunc)
show_details('wrapped_myfunc', wrapped_myfunc)
wrapped_myfunc()
wrapped_myfunc('args to decorated', 4)
Functools
provides a decorator wraps()
which applies update_wrapper()
to the decorated function.
Comparison
Before Python 2, classes contained a method _cmp_()
which returned -1
, 0
or 1
depending on whether the object is less, equal, or greater than the one with which the comparison is made. Python 2.1 introduces a richer interface for comparison methods, _lt_()
, _le_()
, _eq_()
, _ne_()
, _gt_()
and _ge_()
, each of which performs one comparison operation and returns a boolean value. Removed in Python 3 cmp()
in favor of these new methods, therefore functools
provides tools to make it easier to write Python 2 classes that meet the new comparison requirements in Python 3.
Rich comparison
A rich comparison interface is implemented so that classes with complex comparisons can pass each test most efficiently. However, for classes where comparison is relatively straightforward, there is no point in manually creating each of the extended set methods. Class decorator total_ordering()
takes a class that provides some of the methods and adds the missing ones.
import functools
import inspect
from pprint import pprint
@functools.total_ordering
class MyObject(object):
def __init__(self, val):
self.val = val
def __eq__(self, other):
print ' testing __eq__(%s, %s)' % (self.val, other.val)
return self.val == other.val
def __gt__(self, other):
print ' testing __gt__(%s, %s)' % (self.val, other.val)
return self.val > other.val
print 'Methods:n'
pprint(inspect.getmembers(MyObject, inspect.ismethod))
a = MyObject(1)
b = MyObject(2)
print 'nComparisons:'
for expr in [ 'a < b', 'a <= b', 'a == b', 'a >= b', 'a > b' ]:
print 'n%-6s:' % expr
result = eval(expr)
print ' result of %s: %s' % (expr, result)
The class must provide implementation _eq_()
and any other comparison method from the extended set. The decorator adds implementations of other methods that work using the provided comparisons.
$ python functools_total_ordering.py
Methods:
[('__eq__', <unbound method MyObject.__eq__>),
('__ge__', <unbound method MyObject.__ge__>),
('__gt__', <unbound method MyObject.__gt__>),
('__init__', <unbound method MyObject.__init__>),
('__le__', <unbound method MyObject.__le__>),
('__lt__', <unbound method MyObject.__lt__>)]
Comparisons:
a < b :
testing __gt__(1, 2)
testing __eq__(1, 2)
result of a < b: True
a <= b:
testing __gt__(1, 2)
result of a <= b: True
a == b:
testing __eq__(1, 2)
result of a == b: False
a >= b:
testing __gt__(1, 2)
testing __eq__(1, 2)
result of a >= b: False
a > b :
testing __gt__(1, 2)
result of a > b: False
The sort order
Since the old comparison function is no longer used in Python 3, the argument cmp
is no longer supported by features such as sort()
… Python 2 programs that use comparison functions can use cmp_to_key()
to convert them to a function that returns the sort key, which is used to determine the position in the final sequence.
import functools
class MyObject(object):
def __init__(self, val):
self.val = val
def __str__(self):
return 'MyObject(%s)' % self.val
def compare_obj(a, b):
"""Old-style comparison function.
"""
print 'comparing %s and %s' % (a, b)
return cmp(a.val, b.val)
# Make a key function using cmp_to_key()
get_key = functools.cmp_to_key(compare_obj)
def get_key_wrapper(o):
"""Wrapper function for get_key to allow for print statements.
"""
new_key = get_key(o)
print 'key_wrapper(%s) -> %s' % (o, new_key)
return new_key
objs = [ MyObject(x) for x in xrange(5, 0, -1) ]
for o in sorted(objs, key=get_key_wrapper):
print o
Note: usually, cmp_to_key()
is used directly, but in this example an additional function wrapper is needed in order to display more detailed information about how the key function is called.
The output shows that sorted()
starts with a call get_key_wrapper()
for each item in the sequence to get the key. Keys returned cmp_to_key()
, are instances of the class defined in functools
which implements the extended comparison interface based on the return value of the old-style compare function. After all the keys are obtained, the sequence is sorted by comparing the keys.
$ python functools_cmp_to_key.py
key_wrapper(MyObject(5)) -> <functools.K object at 0x100466558>
key_wrapper(MyObject(4)) -> <functools.K object at 0x100466590>
key_wrapper(MyObject(3)) -> <functools.K object at 0x1004665c8>
key_wrapper(MyObject(2)) -> <functools.K object at 0x100466600>
key_wrapper(MyObject(1)) -> <functools.K object at 0x100466638>
comparing MyObject(4) and MyObject(5)
comparing MyObject(3) and MyObject(4)
comparing MyObject(2) and MyObject(3)
comparing MyObject(1) and MyObject(2)
MyObject(1)
MyObject(2)
MyObject(3)
MyObject(4)
MyObject(5)
More details about the course “Python Developer. Basic”. You can watch an open lesson on the topic “Pytest: An Introduction to Autotests” here.