Restoring JSON types

For data transport, I almost always pack them in JSON. But here’s the problem: as a rule, parsing libraries return primitive types and objects with arrays – everything that is laid down by the format itself. But what if you want to get entity models?

I sometimes like to make my own, even if someone has already done something similar. I can’t help myself. In my daily work with JavaScript, I am not very closely connected, but I often use it for home projects: I write back and front in it, because a single code base is too attractive. To hell with it – with that performance – if it’s for fun. But what a beauty!

// код упрощён для демонстрации
class Folder {
    constructor(data) {
        this.dir = data.dir;
        this.caption = data.caption;
        this.collectTime = data.collectTime;
        this.metaThumbnail = data.metaThumbnail;
        this.extras = data.extras;
    }
    
    get parentDir() {
        return this.dir.replace(//?[^/]+$/, '');
    }
    
    static fromDirent(dirent) {
        return new this(/* ... */);
    }
    
    // не относящиеся к содержанию статьи методы
}

// признаю, название ужасное
class AContainer {
    constructor(data) {
        this.file = data.file;
        this.files = data.files; // остальные картинки из серии burst
        this.parentDir = data.parentDir;
        this.collectTime = data.collectTime;
        this.metaTime = data.metaTime;
        this.metaLat = data.metaLat;
        this.metaLon = data.metaLon;
        this.metaThumbnail = data.metaThumbnail;
        this.extras = data.extras;
    }
    
    static async checkFile(dirent) {
        throw new Error("Method 'checkFile' is abstract");
    }
}

async function collectDir(dir="~") {
    const contClasses = [
        // order of appearence is important!
        ImageGif,
        ImageBurst,
        ImagePano,
        ImageBasic,
        VideoSlowmo,
        VideoTimelapse,
        VideoBasic,
    ];
    const dirents = await fsPromises.readdir(dir, { withFileTypes: true });
    const folders = [];
    const containers = [];
    for (const dirent of dirents) {
        if (dirent.isDirectory()) {
            folders.push(Folder.fromDirent(dirent));
        }
        else {
            let container;
            // последовательные проверки файла
            // всеми типами контейнеров:
            for (const contClass of contClasses) {
                container = contClass.checkFile(dirent);
                if (container) {
                    containers.push(container);
                    break;
                }
            }
            if (!container) {
                console.warn(`Unable to detect media format`, dirent);
            }
        }
    }
}

async function getFolder(dir) {
    dir = dir.replace(/[/\]+$/, '');
    
    const resFolder = await db.queryOne(`
        SELECT * from Folder where dir = ?
    `, [dir], models.Folder);
    if (!resFolder) {
        throw new Error(`Unknown Folder: ${dir}`);
    }
    
    resFolder.childContainers = await db.query(`
        SELECT * from Container where parentDir = ?
    `, [dir], models.AContainer);
    
    resFolder.childFolders = await db.query(`
        SELECT * from Folder where parentDir = ?
    `, [dir], models.Folder);
    
    return resFolder;
}

We pull the API method from the browser and get raw JS objects from the back. And it would be desirable classes of models. Also, the types of containers are lost. How to be?

Class object in JSON

Just in case, I’ll tell you that the standard JSON serializer in JavaScript can not only call the replacer function for each value, but also search for objects method toJSON. This allows you to process the object beyond recognition before packaging. At the same time, the implementation of such processing will be in place – in the class description (but not, but not always). For some standard types, there are such implementations at the JS engine level (for example, for date).

With the Folder class, everything is simple – nothing special is required there. But for containers, you need to save information about the type. I offer simply:

class AContainer {
    // ...
    
    toJSON() {
        const res = {};
        Object.assign(res, this);
        // и классы-наследники тут всё сделают правильно:
        res.containerType = this.constructor.name;
        return this;
    }
}

If desired, it would even be possible to nest one object in another, so that it would not be sour due to a possible property name conflict containerType. But this is for gourmets.

JSON to an object of the desired class

The preliminaries are over. You need to expand the JSON somehow so that instead of raw objects, you get dry crispy instances of the desired classes. And clumsy manual processing is not our method. I liked approach to question in Golang. And then, since we have already seen that the “magic” methods like toJSON not censured, why not go further.

The contract will be as follows: if a class is described static method fromJSON, then we will call it immediately after converting JSON into an internal representation (raw objects, arrays and primitives), but before issuing the result of processing. If there is no method, then we will simply forward the value as the only argument to the constructor.

Another contract with itself would be this: every class that extends AContainerwill be this AContainer report about its existence (this is within the framework of the original project, which is about photos).

class AContainer {
    // ...
    
    static implementations = {};
    
    static registerImplementation(implementationClass) {
        this.implementations[implementationClass.name] = implementationClass;
    }
    
    static fromJSON(data) {
        // нужно распаковать даты:
        data.collectTime = new Date(data.collectTime);
        data.metaTime = new Date(data.metaTime);
        
        // код курильщика:
        /*
        switch (data.containerType) {
            case 'ImageBasic': return new ImageBasic(data);
            case 'ImageBurst': return new ImageBurst(data);
            // ...
            case 'VideoTimelapse': return new VideoTimelapse(data);
        }
        */
        
        // код вейпера:
        const contClass = this.implementations[data.containerType];
        if (contClass) {
            return new contClass(data);
        }
        
        throw new Error(`Unknown container type: ${data.containerType}`);
    }
}


class ImageBasic extends AContainer { /*...*/ }
AContainer.registerImplementation(ImageBasic);


class ImageBurst extends AContainer { /*...*/ }
AContainer.registerImplementation(ImageBurst);


class VideoTimelapse extends AContainer { /*...*/ }
AContainer.registerImplementation(VideoTimelapse);

Looks good. But how to call this fromJSON? I danced around reviver functions, but did not cook porridge. I wanted to somehow cleverly generate it, transfer it to JSON.parse(data, <сюда>), but it is unrealistic to work with it when it comes to arbitrary objects of variable nesting. Well, then we will parse as is, and then do the post-processing of the result.

It’s time to do something like a module. Let it be called JSONSchema jsonson.

JSON Son

Figured out how I would like to use it:

JSONSon.parse(Folder, '{...}');
    // object<Folder>

JSONSon.parse(AContainer, '{...}');
    // object<? extends AContainer>

JSONSon.parse(Date, '"2022-02-08T21:15:56.180Z"');
    // object<Date>

JSONSon.parse('string', '"2022-02-08T21:15:56.180Z"');
    // "2022-02-08T21:15:56.180Z"

// и остановиться я уже не мог:
JSONSon.parse('number', '"2022-02-08T21:15:56.180Z"');
    // NaN
JSONSon.parse('number', '"2022"');
    // 2022
JSONSon.parse('boolean', '"2022"');
    // true
JSONSon.parse('boolean', '""');
    // false
JSONSon.parse('bigint', '"2022"');
    // 2022n
JSONSon.parse(Number, '"2022"');
    // object<Number> {2022}
JSONSon.parse(['number'], '[1, "2", 3, 4]');
    // [1, 2, 3, 4]
JSONSon.parse(['boolean', 'number', 'string'], '[1, "2", 3, 4]');
    // [true, 2, "3", "4"] - discover tuple in JS!
JSONSon.parse({ foo: Folder, bar: [AContainer] }, '{...}');
    // { foo: object<Folder>, bar: [object<? extends AContainer>] }

I had to write. I did a simple symmetrical bypass of structures with type conversion, a couple of checks – and you’re done. I must say that there is no longer any binding to JSON: the JSON data was at the previous stage – before it was processed, and the necessary function will do the post-processing of the already parsed data. But since the schema is bidirectional (data ↔ JSON) and I’m already bound to the function toJSONthen let her still be connected with him.

Objects with undeclared properties

Let me remind you that the apish method getFolder returns an instance of the class Folderbut with additional extraneous properties:

await serverApi.getFolder('~/Pictures'); -> {
    // объявленные свойства (те, о которых класс знает):
    dir: '~/Pictures',
    caption: 'Pictures',
    collectTime: '2022-02-08T21:15:56.180Z',
    metaThumbnail: 'data:image/jpeg;base64,...',
    extras: null,
    
    // посторонние свойства:
    childContainers: [{...}, {...}],
    childFolders: [{...}, {...}]
}

Of course, it would be correct to simply “declare” them. But I’m doing a whole type-converter parse here, and not some kind of … where did I start from? In short, it’s more interesting to look for some other solution, because whatever one may say, such situations are not rare in JS. You need to somehow wrap the class, and in addition tell what other extraneous properties are expected. And when processing, you should first convert it to an instance of the desired class, and then go through the additional properties separately. Here’s how the application looks like:

JSONSon.parse(JSONSon.mix(Folder, {
    childContainers: [AContainer],
    childFolders: [Folder],
}), jsonData);

Hook for BigInt

Haven’t checked how things are in other clients, but in Chrome 97 you’ll be surprised if you want to pack BigInt in JSON:

JSON.stringify(9007199254740993n);
    -> Error "TypeError: Do not know how to serialize a BigInt"

If you don’t know, we’ll teach you:

BigInt.prototype.toJSON = function () { return this.toString(10); };

let json = JSON.stringify(9007199254740993n); // "9007199254740993"
JSONSon.parse('bigint', json); // 9007199254740993n
JSONSon.parse(BigInt, json); // object<BigInt> {9007199254740993n}

Also, BigInt is not a constructor (unlike Boolean, Number, and String). Therefore, to get an object wrapper for this type, special behavior is made in JSONSon: Object(BigInt(value)). I don’t know why this might be needed, but for the sake of order, it’s done.

Weird: large numbers convert to a string, and convert back just as well. If stored as a regular number, then accuracy will be lost in the process between parsing the JSON string (and I remind you that it is native) and passing it to the type converter.

Application

And the parser itself is not inside. Indeed, the function JSONSon.parse does almost nothing: it runs the standard parser, and the result is forwarded to the truly main function – JSONSon.make. Therefore, it turns out that it is not at all necessary to feed her a JSON string. If I already have some raw data, and I just need to convert types, then I can call JSONSon.make.

This craft can be used in two styles:

1. call static methods of the JSONSon class;

2. make an instance of the class and call its methods.

// static:

JSONSon.parse(Folder, "{...}");
//or
let folder = await serverApi.getFolder('~/Videos'); // JSON parsed inside
JSONSon.make(Folder, folder);


// non-static:
let schema = new JSONSon(Folder);

JSONSon.parse("{...}");
// or
JSONSon.make(folder);

Pass JSONSon to JSON

What I usually do is that the backend provides the API along with auto-description of its methods. The client asks the server: “What methods do you have”? And he gives him a list of all methods with declared parameters, as well as their types (if the language allows). After that, the client, on its side in the wrapper, organizes everything so that the application looks like a simple function call: await serverApi.getFolder('~'). It would be nice if the client also took over the type conversion. And this is possible: you just need to somehow transfer the schema itself to the client – an instance JSONSon. It turns out that it also needs to be correctly converted to JSON. And there are classes completely unsuitable for transformation! How to be? It was decided not to bother too much (hehe): we just define the class name and return it as a string. And the client will do the reverse transformation. But who knows what other side will do the reverse transformation. How to find constructors by their names? In short, I made another static method JSONSon.resolveConstructor with a primitive standard implementation. And it is supposed to be changed in its own way at the place of application. This is what my initialization block looks like on the front:

<script type="module">
import JSONSon from './src/utils/JSONSon.js';
import Folder from './src/Folder.js';
import * as contClasses from './src/containers/index.js';

const supportedConstructorLibs = [
    window,
    {
        Folder,
        ...contClasses,
    },
];
JSONSon.resolveConstructor = (name) => {
    for (const lib of supportedConstructorLibs) {
        if (lib[name]) {
            return lib[name];
        }
    }
    return null;
};

//...
</script>

Bottom line: the server returns all declared API methods, and also the JSONSon schema of the result for each; the client gets it all; at the time of calling each method, the client knows what type the result should be converted to.

It’s funny that the implementations JSONSon.toJSON and JSONSon.fromJSON turned out fatter than the code JSONSon.makeand generally make up almost 40% of the module code.

A little more confusion

Remember that the API method returns Folder with additional extraneous properties? So I thought: is it really not possible to complicate things a little more 🙂 Well, in fact, why should I constantly write like this:

JSONSon.mix(Folder, {
    childContainers: [AContainer],
    childFolders: [Folder],
})

Can this be done in class too? Folder. But not manually in the method fromJSON, but somehow smarter. And yes: you can declare that JSONSon will look for another magic method in classes that will produce a refined schema for types. Well, here it is:

class Folder {
    // ...
    
    static getJSONSonSchema() {
        return JSONSon.mix(
            this,
            {
                childFolders: [this],
                childContainers: [AContainer],
            }
        );
    }
}

// демонстрация
let data = {
    dir: '/home/bars/',
    // ...
    childFolders: [
        { dir: '/home/bars/Videos' },
        { dir: '/home/bars/Images' },
        // ...
    ],
    childContainers: [{
        containerType: 'ImageBasic',
        file: '/home/bars/hello.jpg',
        // ...
    }],
};

// теперь можно прямо так:
let folder = JSONSon.make(Folder, data);

The result will be a folder with a correctly populated array of child folders and a container like ImageBasic.

Sources for crafts

https://github.com/bbars/utils/tree/master/js-json-son