Draw beautiful tracebacks by catching exceptions in Python

We all spend a lot of time debugging, digging into logs or reading tracebacks (traceback, stack trace reports). Any of these cases can be complex and lengthy. This post is about how to make stack tracing and exception handling as simple and efficient as possible.

On the way to this goal, we will learn how to implement and use our own exception hooks (exception hook), which allow you to remove all the “information noise” from tracebacks. We will talk about how to improve the readability of stack trace reports, how to output only what is needed in them to solve problems with Python code and exceptions that occur during its operation. In addition, we’ll take a look at some awesome Python libraries that have ready-to-use, well-made exception handlers. They can be used without having to write your own interceptor code.

Exception hooks

Whenever an exception occurs that is not handled in a block try/exceptthe function assigned sys.excepthook. This function, called the exception handler, is used to print any useful information about the incident to the standard output stream. To do this, she uses the three arguments she receives – type (exception class), value (an exception instance) and traceback (trace object). Let’s analyze a minimal example demonstrating the operation of this mechanism:

import sys

def exception_hook(exc_type, exc_value, tb):
    print('Traceback:')
    filename = tb.tb_frame.f_code.co_filename
    name = tb.tb_frame.f_code.co_name
    line_no = tb.tb_lineno
    print(f"File {filename} line {line_no}, in {name}")
    # Класс и экземпляр исключения
    print(f"{exc_type.name}, Message: {exc_value}")
sys.excepthook = exception_hook

Here we use the above arguments to print out the basic trace data related to the exception. Namely, the trace object (tb) is used to work with stack frame trace information. It contains data describing the location of the exception – the name of the file (f_code.co_filename), function/module name (f_code.co_name) and line number (tb_lineno).

In addition, we display information about the exception itself, using variables exc_type And exc_value.

Now we can call a function that throws some kind of exception. With such an interceptor, we will see the message shown at the bottom of the following code snippet:

def do_stuff():
    # ... сделать что-то такое, что вызывает исключение
    raise ValueError("Some error message")
do_stuff()
Traceback:
File /home/some/path/exception_hooks.py line 22, in <module>
ValueError, Message: Some error message

Here is just some information about the exception. In order to see all the information needed to debug the code, as well as to get a complete picture of where and why the exception occurred, we need to dig deeper into the trace object:

def exception_hook(exc_type, exc_value, tb):
    local_vars = {}
    while tb:
        filename = tb.tb_frame.f_code.co_filename
        name = tb.tb_frame.f_code.co_name
        line_no = tb.tb_lineno
        print(f"File {filename} line {line_no}, in {name}")
        local_vars = tb.tb_frame.f_locals
        tb = tb.tb_next
    print(f"Local variables in top frame: {local_vars}")
...
File /home/some/path/exception_hooks.py line 41, in <module>
File /home/some/path/exception_hooks.py line 7, in do_stuff
Local variables in top frame: {'some_var': 'data'}

As you can see, the trace object (tb) is actually a linked list of exceptions that have occurred – a stacktrace. This fact allows us to traverse this list with tb_next and output information about each stack frame. Moreover, you can use the attribute tb_frame.f_locals, output local variables to the console. This can help us debug the code.

Viewing the contents of the trace object, as we have already seen, makes sense, but the data received is poorly organized, it becomes more and more difficult to perceive it as the scale of the task grows. Therefore, it would be better to use the module tracebackwhich has many helper functions aimed at extracting information about exceptions.

Now that we’ve covered the basics, let’s talk about how to create your own exception handlers that have features that can be useful in real work.

Creating Your Own Exception Hooks

When an exception is caught, outputting data about it to stdout – this is not all that can be done with it. In particular, you can also automatically output the data about the exception to a file:

LOG_FILE_PATH = "./some.log"
FILE = open(LOG_FILE_PATH, mode="w")
def exception_hook(exc_type, exc_value, tb):
    FILE.write("*** Exception: ***\n")
    traceback.print_exc(file=FILE)
    FILE.write("\n*** Traceback: ***\n")
    traceback.print_tb(tb, file=FILE)
    
*** Exception: ***
NoneType: None
# 
*** Traceback: ***
  File "/home/some/path/exception_hooks.py", line 82, in <module>
    do_stuff()
  File "/home/some/path/exception_hooks.py", line 7, in do_stuff
    raise ValueError("Some error message")

This approach can be useful in situations where the programmer wants to store information about uncaught exceptions for further analysis.

Messages about uncaught exceptions, by default, are displayed in stderr, standard error stream. And this may be undesirable if there is a system for logging exceptions, and it is necessary that this system would process what is output to the standard error stream. In order for the logging system to work exactly like this, you can use the following interceptor:

import logging
logging.basicConfig(
    level=logging.CRITICAL,
    format="[%(asctime)s] {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s",
    datefmt="%H:%M:%S",
    stream=sys.stdout
)
def exception_hook(exc_type, exc_value, exc_traceback):
    logging.critical("Uncaught exception:", exc_info=(exc_type, exc_value, exc_traceback))
[17:28:33] {/home/some/path/exception_hooks.py:117} CRITICAL - Uncaught exception:
Traceback (most recent call last):
  File "/home/some/path/exception_hooks.py", line 122, in <module>
    do_stuff()
  File "/home/some/path/exception_hooks.py", line 7, in do_stuff
    raise ValueError("Some error message")
ValueError: Some error message

When trying to improve what is output to the console, the first thing that comes to mind is decorating the text by color highlighting the most important:

# pip install colorama
from colorama import init, Fore
init(autoreset=True)  # Сбросить цвета после каждой операции вывода данных
def exception_hook(exc_type, exc_value, tb):
    local_vars = {}
    while tb:
        filename = tb.tb_frame.f_code.co_filename
        name = tb.tb_frame.f_code.co_name
        line_no = tb.tb_lineno
        # Снабдить строку сведениями о нужном цвете (например - цветом с кодом RED)
        print(f"{Fore.RED}File {filename} line {line_no}, in {name}")
        local_vars = tb.tb_frame.f_locals
        tb = tb.tb_next
    print(f"{Fore.GREEN}Local variables in top frame: {local_vars}")

Of course, there’s a lot more you can do to improve the appearance of what’s printed to the console. For example, you can display local variables for each frame, or even look for variables that were accessed from the line in which the exception occurred. Unsurprisingly, exception handlers have already been created to handle such scenarios. Therefore, if you need it, instead of trying to create something like this completely on your own, I recommend taking a look at the following code, which can inspire you to your own developments:

Finally, I would like to warn you. Whenever you decide to install an exception handler, be aware that the libraries you use may install their own hooks. Make sure you don’t override these interceptors. In such cases, you can do otherwise – use the construction try/except and, in the block exceptdisplay the necessary information, for example, using sys.exc_info().

Existing interceptors worthy of attention

Developing your own exception handler can be a fun little learning project. But a decent number of excellent interceptors have already been created. Therefore, instead of reinventing the wheel, let’s take a better look at what can be used right now.

I’ll start with my favorite tool – with Rich:

# https://rich.readthedocs.io/en/latest/traceback.html
pip install rich
python -m rich.traceback
from rich.traceback import install
install(show_locals=True)
do_stuff()  # Вызывает ValueError
The traceback that Rich outputs
The traceback that Rich outputs

Using this tool is very simple – it all comes down to installing the appropriate library, importing it into the project, and calling the function install, which hooks up the exception handler. If you’re interested just look at the output Richwithout writing Python code, you can resort to the command python -m rich.traceback.

Another popular tool of this kind is the library better_exceptions. It also formats the output data very nicely, but, unlike Richyou need to make a little more effort to configure it:

# https://github.com/Qix-/better-exceptions
pip install better_exceptions
export BETTER_EXCEPTIONS=1
import better_exceptions
better_exceptions.MAX_LENGTH = None
Проверьте, установлена ли переменная TERM в значение xterm, если это не так - настройте следующую переменную,
Смотрите это сообщение о проблеме: https://github.com/Qix-/better-exceptions/issues/8
better_exceptions.SUPPORTS_COLOR = True
better_exceptions.hook()
do_stuff()  # Вызывает ValueError

In addition to installing the library better-exceptions via pipto enable it, we also need to set the environment variable BETTER_EXCEPTIONS=1. After that, the library must be configured using the above Python code. The most important thing in it is the function call hook, which sets the exception handler. In addition, we set SUPPORTS_COLOR into meaning True. The need for this setting depends on the terminal you are using. In particular, it is needed if TERM a value is written that is different from xterm.

A traceback that outputs better-exceptions
A traceback that outputs better-exceptions

The next number of our program will be the library pretty_errors. Of the tools I’m talking about, this one is definitely the easiest to set up. In order to use it, it is enough to import it into the project:

# https://github.com/onelivesleft/PrettyErrors/
pip install pretty_errors
import pretty_errors
если вас устраивают стандартные настройки - можно обойтись без configure
pretty_errors.configure(
    filename_display    = pretty_errors.FILENAME_EXTENDED,
    line_number_first   = True,
    display_link        = True,
    line_color          = pretty_errors.RED + '> ' + pretty_errors.default_config.line_color,
    code_color          = '  ' + pretty_errors.default_config.line_color,
    truncate_code       = True,
    display_locals      = True
)
do_stuff()

In the previous code snippet, in addition to the obligatory construct import, shows library settings that you can do without. This is only a small part of the settings that the library supports. Full list of configuration options pretty_errors can be found here.

A traceback that outputs pretty_errors
A traceback that outputs pretty_errors

Our next library outputs tracebacks in a style that anyone who uses Jupyter Notebook will be familiar with. This is an IPython module. ultratb. It allows you to display error messages and tracebacks that look good and are easy to read:

# https://ipython.readthedocs.io/en/stable/api/generated/IPython.core.ultratb.html
pip install ipython
import IPython.core.ultratb
Ещё - ColorTB, FormattedTB, ListTB, SyntaxTB
sys.excepthook = IPython.core.ultratb.VerboseTB(color_scheme="Linux")  # Другие цветовые схемы: NoColor, LightBG, Neutral
do_stuff()
The traceback that ultratb outputs
The traceback that ultratb outputs

And here is another library, stackprinter, last on my list, but far from the last in terms of its capabilities. It gives clear information about the problems, containing all the necessary debugging information. In order to use it, it is enough to install an exception interceptor:

# https://github.com/cknd/stackprinter
pip install stackprinter
import stackprinter
stackprinter.set_excepthook(style="darkbg2")
do_stuff()
The traceback output by the stackprinter
The traceback output by the stackprinter

Results

In this tutorial, you learned how to create your own exception handlers. But I really wouldn’t recommend doing it. The implementation of such an interceptor may be an interesting programming challenge, but the feasibility of such a development is probably in question. It is better to pick up something from the existing libraries discussed above.

But what I would definitely recommend is to choose one of these libraries and install it in all the projects you work on. It is worth doing this to improve code debugging and for the sake of uniformity of the tools used. The more you use one of the above exception hooks, the more you get used to the data it emits, and as a result, the more you can get out of it.

Given the above, I advise you to consider removing your own exception hooks from production builds. The fact is that the peculiarities of the design of the output data can hide from us some information about errors, which, in certain situations, can be extremely important. Let’s say some of the above examples are missing file paths. This improves the readability of the output when debugging locally, but can make life difficult for the programmer when debugging code running on a remote system.

Oh, and come to work with us? 😏

We are in wunderfund.io doing high-frequency algorithmic trading since 2014. High-frequency trading is a continuous competition between the best programmers and mathematicians around the world. By joining us, you will become part of this exciting fight.

We offer interesting and challenging tasks in data analysis and low latency development for enthusiastic researchers and programmers. Flexible schedule and no bureaucracy, decisions are quickly made and implemented.

Now we are looking for plus developers, pythonists, data engineers and ml-risers.

Join our team.

Similar Posts

Leave a Reply