Error Handling (2024 Edition)

This is the seventh part of the Flask mega tutorial series in which I am going to teach you how to do error handling in a Flask application.

In this chapter, I take a break from writing new features in my microblogging application and instead talk about a few strategies for solving the bugs that invariably appear in every software project. To illustrate this point, I intentionally left out a bug in the code I added to chapter 6. Before you continue reading, see if you can find it!

GitHub links for this chapter: Browse, Zip, Diff.

Error Handling in Flask

What happens when an error occurs in a Flask application? The best way to learn about it is to experience it yourself. Launch the application and make sure you have at least two users registered. Log in as one of the users, open the profile page and click the “Edit” link. In the profile editor, try changing the username to the name of another user who is already registered, and boom! A scary looking “Internal Server Error” page will appear:

If you look in the terminal session where the application is running you will see stack trace errors. Stack traces are extremely useful when debugging errors because they show the sequence of calls on that stack, right down to the line that caused the error:

([2023-04-28 23:59:42,300] ERROR in app: Exception on /edit_profile [POST]
Traceback (most recent call last):
  File "venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 918, in do_execute
    cursor.execute(statement, parameters)
sqlite3.IntegrityError: UNIQUE constraint failed: user.username

A stack trace helps determine where the error is. The application allows the user to change the username, but does not check that the selected new username does not collide with another user already logged in to the system. The error comes from SQLAlchemy which is trying to write a new username to the database, but the database is rejecting it because username column defined using option unique=True.

It's important to note that the error page that is presented to the user does not contain much information about the error, which is good. I definitely don't want users to find out that the crash was caused by a database error, or what database I'm using, or what some of the names of the tables and fields in my database are. All this information must be stored internally.

But there are a few things that are far from ideal. I have an error page that is very ugly and doesn't fit the app's layout. I also have important application stack traces dumped to the terminal that I need to constantly monitor to make sure I don't miss any errors. And of course I need to fix the mistake. I'm going to go over all of these issues, but first let's talk about debug mode in Flask.

Debug mode

The way you saw errors being handled above works great for a system running on a production server. When an error occurs, the user gets a vague error page (although I'm going to make the error page nicer), and the important details about the error will be contained in the output of the server process or in the log file.

But when you're developing your application, you can enable debug mode, a mode in which Flask outputs a really nice debugger right in your browser. To enable debug mode, stop the application, and then set the following environment variable:

(venv) $ export FLASK_DEBUG=1

If you are using Microsoft Windows, be sure to use set instead of export.

After setup FLASK_DEBUG restart the server. The output on your terminal will be slightly different from what you are used to seeing:

(venv) $ flask run
 * Serving Flask app 'microblog.py' (lazy loading)
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 118-204-854

Now crash the application again to see the interactive debugger in your browser:

The debugger allows you to expand each stack frame and view the corresponding source code. You can also open a Python prompt in any of the frames and execute any valid Python expressions, such as checking the values ​​of variables.

It is extremely important that you never run a Flask application in debug mode on a production server. A debugger allows a user to remotely execute code on a server, so it can be a surprise boon to an attacker who wants to break into your application or your server. As an additional security measure, the debugger running in the browser starts locked, and the first time you use it, it asks for a PIN, which you can see in the command output flask run.

Since I touched on the topic of debug mode, I must mention the second important feature that is enabled in debug mode, namely rebooter. This is a very useful development feature that automatically restarts the application when the source file changes. If you run flask run in debug mode, you will be able to work on your application, and every time you save a file, the application will restart to get new code.

Custom error pages

Flask provides a mechanism for your application to set its own error pages so that your users don't have to see the default plain and boring pages. As an example, let's define our error pages for HTTP errors 404 and 500, the two most common ones. Defining pages for other errors works in the same way.

A decorator is used to declare an error handler @errorhandler. I'm going to put my error handlers in a new module app/errors.py.

app/errors.py: Custom error handlers

from flask import render_template
from app import app, db

@app.errorhandler(404)
def not_found_error(error):
    return render_template('404.html'), 404

@app.errorhandler(500)
def internal_error(error):
    db.session.rollback()
    return render_template('500.html'), 500

The error functions work very similarly to the view functions. For these two errors I return the contents of the corresponding templates. Note that both functions return the second value after the pattern, which is the error code number. For all the view functions I've created so far, I didn't need to add a second return value because the default value of 200 (the status code for a successful response) is what I wanted. In this case, these are error pages, so I want the response status code to reflect that.

The error handler for error 500 could have been called after a database error, which was actually the case with the duplicate username above. To ensure that any failed database sessions do not interfere with any template-initiated database calls, I perform a session rollback. This puts the session into a clean state.

Here's a template for a 404 error:

app/templates/404.html: Requested file not found

{% extends "base.html" %}

{% block content %}
    <h1>File Not Found</h1>
    <p><a href="https://habr.com/ru/articles/809743/{{ url_for("index') }}">Back</a></p>
{% endblock %}

And here is the one regarding the 500 error:

app/templates/500.html: Internal Server Error

{% extends "base.html" %}

{% block content %}
    <h1>An unexpected error has occurred</h1>
    <p>The administrator has been notified. Sorry for the inconvenience!</p>
    <p><a href="https://habr.com/ru/articles/809743/{{ url_for("index') }}">Back</a></p>
{% endblock %}

Both templates inherit from template base.htmlso the error page has the same appearance as normal application pages.

To register these error handlers in Flask I need to import a new module app/errors.py after creating an application instance:

app/__init__.py: Importing error handlers

# ...

from app import routes, models, errors

If you install in a terminal session FLASK_DEBUG=0 (or remove the variable FLASK_DEBUG), and then run the duplicate username error again, you'll see a slightly clearer error page.

Sending errors by email

Another problem with the default error handling provided by Flask is the lack of notifications. Stack traces for any errors are printed to the terminal, which means that the output of the server process must be monitored to detect errors. When you run an application during development, this is completely fine, but once the application is deployed to a production server, no one will view the server output in real time, so a more robust solution needs to be implemented.

I think it's very important to take a proactive approach to mistakes. If a bug occurs in a production version of the application, I want to know about it immediately. So my first solution would be to configure Flask to send me an email immediately after an error, with the error stack tracked in the body of the email.

The first step is to add the email server information to the configuration file:

config.py: Email Settings

class Config:
    # ...
    MAIL_SERVER = os.environ.get('MAIL_SERVER')
    MAIL_PORT = int(os.environ.get('MAIL_PORT') or 25)
    MAIL_USE_TLS = os.environ.get('MAIL_USE_TLS') is not None
    MAIL_USERNAME = os.environ.get('MAIL_USERNAME')
    MAIL_PASSWORD = os.environ.get('MAIL_PASSWORD')
    ADMINS = ['your-email@example.com']

Configuration variables for email include server and port, a boolean flag to enable encrypted connections, and an optional username and password. The five configuration variables are derived from their environment variable counterparts. If an email server is not configured in the environment, I will use this as an indication to disable error emailing. The mail server port can also be specified in an environment variable, but if it is not set, the default port 25 is used. Mail server credentials are not used by default, but can be provided if necessary. Configuration Variable ADMINS is a list of email addresses to which error messages will be sent, so your own email address should be on this list.

Flask uses the Python package logging to record your logs, and this package already has the ability to send logs by email. All I need to do to receive emails sent with errors is to add an instance SMTPHandler to the Flask logger object, which is accessible via app.logger:

app/__init__.py: Logging errors via email

import logging
from logging.handlers import SMTPHandler

# ...

if not app.debug:
    if app.config['MAIL_SERVER']:
        auth = None
        if app.config['MAIL_USERNAME'] or app.config['MAIL_PASSWORD']:
            auth = (app.config['MAIL_USERNAME'], app.config['MAIL_PASSWORD'])
        secure = None
        if app.config['MAIL_USE_TLS']:
            secure = ()
        mail_handler = SMTPHandler(
            mailhost=(app.config['MAIL_SERVER'], app.config['MAIL_PORT']),
            fromaddr="no-reply@" + app.config['MAIL_SERVER'],
            toaddrs=app.config['ADMINS'], subject="Microblog Failure",
            credentials=auth, secure=secure)
        mail_handler.setLevel(logging.ERROR)
        app.logger.addHandler(mail_handler)

from app import routes, models, errors

As you can see, I'm only going to enable the email logger when the application is running without debug mode, which is denoted by app.debug not equal Trueand also when the mail server has a configuration.

Setting up an email logger is somewhat tedious due to the need to handle additional security settings that are present on many email servers. But essentially the above code creates an instance SMTPHandlersets its level so that it reports only errors and not warnings, informational or debug messages, and finally attaches it to the object app.logger from Flask.

There are two approaches to testing this feature. The simplest is to use an SMTP debug server. This is a fake mail server that accepts emails, but instead of sending them, outputs them to the console. To start this server, open a second terminal session, activate the virtual environment and install the package aiosmtpd:

(venv) $ pip install aiosmtpd

Then run the following command to start the debug mail server:

(venv) $ aiosmtpd -n -c aiosmtpd.handlers.Debugging -l localhost:8025

This command will not print anything yet, but will wait for clients to connect. Leave the debug SMTP server running and go back to your first terminal and configure your mail server like this:

export MAIL_SERVER=localhost
export MAIL_PORT=8025

As always, use set instead of exportif you are using Microsoft Windows. Make sure that for the variable FLASK_DEBUG set to 0 or not set at all because the application will not send emails in debug mode. Run the application and throw the SQLAlchemy error again to see the terminal session running the fake mail server show an email with the full stack trace of the error.

The second way to test this feature is to set up a real mail server. Below is the configuration to use the mail server of your Gmail account:

export MAIL_SERVER=smtp.googlemail.com
export MAIL_PORT=587
export MAIL_USE_TLS=1
export MAIL_USERNAME=<your-gmail-username>
export MAIL_PASSWORD=<your-gmail-password>

If you are using Microsoft Windows, be sure to use set instead of export in each of the instructions above.

The security features of your Gmail account may prevent an app from sending emails through it unless you explicitly allow “less secure apps” access to your Gmail account. You can read about it Hereand if you're concerned about the security of your account, you can create an additional account that you set up to only send emails, or you can temporarily enable less secure apps to run this test and then revert to the default.

export MAIL_SERVER=smtp.yandex.ru
export MAIL_PORT=587
export MAIL_USE_TLS=1
export MAIL_USERNAME=<your-gmail-username>
export MAIL_PASSWORD=<your-gmail-password>

import logging
from logging.handlers import SMTPHandler

# ...

if not app.debug:
    if app.config['MAIL_SERVER']:
        auth = None
        if app.config['MAIL_USERNAME'] or app.config['MAIL_PASSWORD']:
            auth = (app.config['MAIL_USERNAME'], app.config['MAIL_PASSWORD'])
        secure = None
        if app.config['MAIL_USE_TLS']:
            secure = ()
        mail_handler = SMTPHandler(
            mailhost=(app.config['MAIL_SERVER'], app.config['MAIL_PORT']),
            fromaddr=app.config['MAIL_USERNAME'],
            toaddrs=app.config['ADMINS'], subject="Microblog Failure",
            credentials=auth, secure=secure)
        mail_handler.setLevel(logging.ERROR)
        app.logger.addHandler(mail_handler)

from app import routes, models, errors

Another alternative is to use a dedicated email service such as SendGridwhich allows you to send up to 100 emails per day with a free account.

Write to file

Receiving errors by email is nice, but sometimes it's not enough. There are some failure conditions that do not result in a Python exception and are not a serious problem, but they may still be interesting enough to keep for debugging purposes. For this reason I am also going to keep a log file for the application.

To enable file-based logging, another handler must be attached to the application logger, this time like RotatingFileHandlersimilar to an email handler.

app/__init__.py: Write to file

# ...
from logging.handlers import RotatingFileHandler
import os

# ...

if not app.debug:
    # ...

    if not os.path.exists('logs'):
        os.mkdir('logs')
    file_handler = RotatingFileHandler('logs/microblog.log', maxBytes=10240,
                                       backupCount=10)
    file_handler.setFormatter(logging.Formatter(
        '%(asctime)s %(levelname)s: %(message)s [in %(pathname)s:%(lineno)d]'))
    file_handler.setLevel(logging.INFO)
    app.logger.addHandler(file_handler)

    app.logger.setLevel(logging.INFO)
    app.logger.info('Microblog startup')

I am writing a log file named microblog.log to the catalog logswhich I create if it does not already exist.

Class RotatingFileHandler The good thing is that it cyclically rewrites the logs, ensuring that the log files do not become too large when the application is running for a long time. In this case, I limit the log file size to 10 KB and keep the last ten log files as a backup.

Class logging.Formatter provides custom formatting of log messages. Since these messages are sent to a file, I want them to have as much information as possible. So, I use a format that includes a timestamp, logging level, message, and the source file and line number where the log entry came from.

To make logging more useful, I'm also downgrading the logging level to INFO, both in the application logger and in the file logger handler. In case you are not familiar with the logging categories, they are presented in order of increasing severity DEBUG, INFO, WARNING, ERROR And CRITICAL.

As a first interesting use of the log file, the server writes a line to the logs every time it starts. When this application runs on the production server, these log entries tell you when the server was restarted.

Fixing a bug with duplicate username

I've been using the duplicate username bug for too long. Now that I've shown you how to prepare your application to handle this type of error, I can go ahead and fix it.

If you remember in RegistrationForm Validation of usernames has already been implemented, but the requirements of the editing form are slightly different. During registration, I need to ensure that the username entered into the form does not exist in the database. In the edit profile form, I have to do the same check, but with one exception. If the user leaves the original username untouched, then the check should allow this since that username is already assigned to that user. Below you can see how I implemented username validation for this form:

app/forms.py: Checking the username in the profile editing form.

class EditProfileForm(FlaskForm):
    username = StringField('Username', validators=[DataRequired()])
    about_me = TextAreaField('About me', validators=[Length(min=0, max=140)])
    submit = SubmitField('Submit')

    def __init__(self, original_username, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.original_username = original_username

    def validate_username(self, username):
        if username.data != self.original_username:
            user = db.session.scalar(sa.select(User).where(
                User.username == self.username.data))
            if user is not None:
                raise ValidationError('Please use a different username.')

The implementation is done in a custom validation method, but there is an overloaded constructor that takes the original username as an argument. This username is saved as an instance variable and checked in the method validate_username() . If the username entered into the form is the same as the original username, then there is no reason to check the database for duplicates.

To use this new validation method, I need to add the original username argument to the view function where the form object is created:

app/routes.py: Pass the username to the profile edit form.

@app.route('/edit_profile', methods=['GET', 'POST'])
@login_required
def edit_profile():
    form = EditProfileForm(current_user.username)
    # ...

The bug has now been fixed and duplication in the profile edit form will be prevented in most cases. This is not a perfect solution because it may not work when two or more processes are accessing the database at the same time. In this situation race condition may result in the check being successful, but a moment later when attempting to rename, the database will already have been modified by another process and will not allow the user to be renamed. This is somewhat unlikely except for very busy applications that have a lot of server processes, so I'm not going to worry about it just yet.

At this point, you can try to reproduce the error again to see how the new form validation method prevents it.

Always enable debug mode

Flask's debug mode is so useful that you may want to enable it by default. This can be done by adding an environment variable FLASK_DEBUG to file .flaskenv .

.flaskenv: Environment variables for flask command

FLASK_APP=microblog.py
FLASK_DEBUG=1

With this change, debug mode will be enabled when starting the server using the command flask run.