msgspec. Library for serializing and deserializing anything

In everyday tasks, there are many tools for working with different data formats such as JSON, TOML, YAML and others.

msgspec — is a tool that can work with all these formats and at the same time be fast and easy to use. There is one import for all formats, which is an advantage in terms of working with this library. If you need to parse many different data formats, then this library is definitely for you.

The library contains:

• High-performance encoders/decoders for common protocols: JSON, MessagePack, YAML and TOMI.

• Support for a wide range of data types.

• Fast data validation

• Structures that allow you to represent data (similar to attrs/pydantic, but faster)

First, let's install the library:

 pip3 install "msgspec[toml,yaml]"

Without additional parameters, the library will be installed with json and messagepack support.

Serialization/deserialization

Serialization and deserialization of data require a data format and an input value.

import msgspec

a = {"1": 2, "3": 4}
encoded_json_data = msgspec.json.encode(a)
print(encoded_json_data)  # b'{"1": 2, "3": 4}'

decoded_data = msgspec.json.decode(encoded_json_data)
print(decoded_data)  # {'1': 2, '3': 4}

yaml_data = msgspec.yaml.encode(a)
print(yaml_data)  # b"'1': 2\n'3': 4\n"

It can be concluded that it is enough to write msgspec.{format}.[encode/decode]where format is json, yaml, toml, msgpack.

Validation

For data validation, it is proposed to create classes inheriting from msgspec.Structsimilar to those in attrs/dataclsses/pydantic.

class User(msgspec.Struct):
    name: str
    surname: str
    email: str | None = None


print(msgspec.json.decode(b'{"name":"vasya","surname":"pupkin"}', type=User))
# User(name="vasya", surname="pupkin", email=None)

print(msgspec.json.decode(b'{"name":"vasya","surname":123}', type=User))
# print(msgspec.json.decode(b'{"name":"vasya","surname":123}', type=User))
#       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# msgspec.ValidationError: Expected `str`, got `int` - at `$.surname`

Only the argument is added type which defines the scheme.

Benchmarks

Let's compare msgspec in speed with the most popular serialization libraries:

To do this, we take large files and run them a thousand times for each case, and compare the average time.

tests = [
    # JSON
    {
        "msgspec.json.decode(data)": {"data": json_text},
        "json.loads(data)": {"data": json_text},
    },
    {
        "msgspec.json.encode(data)": {"data": json_data},
        "json.dumps(data)": {"data": json_data},
    },
    {
        "msgspec.json.decode(data)": {"data": json_text},
        "orjson.loads(data)": {"data": json_text},
    },
    {
        "msgspec.json.encode(data)": {"data": json_data},
        "orjson.dumps(data)": {"data": json_data},
    },
    # TOML
    {
        "msgspec.toml.decode(data)": {"data": toml_text},
        "toml.loads(data)": {"data": toml_text},
    },
    {
        "msgspec.toml.decode(data)": {"data": toml_text},
        "tomllib.loads(data)": {"data": toml_text},
    },
    {
        "msgspec.toml.encode(data)": {"data": toml_data},
        "toml.dumps(data)": {"data": toml_data},
    },
    # YAML
    {
        "msgspec.yaml.decode(data)": {"data": yaml_text},
        "yaml.load(data, Loader=yaml.Loader)": {
            "data": yaml_text,
        },
    },
    {
        "msgspec.yaml.decode(data)": {"data": yaml_text},
        "yaml.load(data, Loader=yaml.CLoader)": {
            "data": yaml_text,
        },
    },
    {
        "msgspec.yaml.encode(data)": {"data": yaml_data},
        "yaml.dump(data, Dumper=yaml.Dumper)": {
            "data": yaml_data,
        },
    },
    {
        "msgspec.yaml.encode(data)": {"data": yaml_data},
        "yaml.dump(data, Dumper=yaml.CDumper)": {
            "data": yaml_data,
        },
    },
]

It can be concluded that the library has better parsers under the hood. If you use only json, then you should take a closer look at another library (for example, orjson), but still the values ​​of the libraries do not differ critically. Also, in the example of yaml, msgspec uses standard C implementations.

Besides, There is benchmarks from the library creators that compare the speed of data serialization and validation.

If your code requires working with different data formats, for example, your program accepts a config in yaml, raises an API that returns json and is not tied to any pydantiс, then you should try msgspec. The library also has goodies in the form of json schemes.