Interactive and documented diagrams for complex systems

My first on-call was not easy. Weeks of training and education did not prepare me for the fact that I would have to search through the Slack channels of various teams for someone who might know something about some part of the system. It turned out that many pages in the corporate Wiki had not been updated for several years. Teams stored their documentation wherever they wanted: some in Wiki, some in Google Docs, some in GitHub, etc. Our on-call was not ideal: 2 people were on duty 24/7. One was responsible for the entire infrastructure (MySql, Cassandra, Kafka, ElasticSearch, Nomad, etc.), the second was Developer on-call and was responsible for all microservices and various legacy systems, which in total gave about 300 different services from 6 -ty commands on a variety of stacks and frameworks (Java, Scala, Node, Go). But what bothered me most was the inability to quickly assess at the highest level how a request from a user is processed and processed. Diagrams for different business parts were also either outdated, or without accompanying documentation, or there was absolutely nothing for some business logic. And then the idea of ​​interactive diagrams appeared, in which you can not only click on any element and get more detailed information about it, but also get links to other diagrams and dynamically load them, and in case of changes, edit and save them there. I wanted to be able to quickly understand an unknown distributed system without having to switch between a diagram and documentation in Google Docs or Wiki. So I started working on my own diagram editing solution, which later evolved into a full-fledged prototype and vector editor. Warning: This article contains a large number of GIFs.

The first thing I started working on was a simple diagram editor with built-in documentation, which I called Schemio. The rendering was implemented in SVG, and Vue.js was chosen as the framework. The initial implementation of Schemio was simple. The idea was that when you click on any element in the diagram, text will appear with documentation of that element. In this way, it would be possible to document individual parts of the diagram, rather than simply attaching the diagram itself to the text of a description of a system, as is usually done. However, over time, new ideas appeared, animations and complex interactive elements were added, and it was decided to create a cloud platform for collaboration. In this article I will try to describe how the interactive diagram editor works and its capabilities.

All in all, Schem.io is an open source platform for creating documented and interactive diagrams, application prototypes, vector graphics, and collaborative educational demonstrations. There are two modes available in Schemio: editor and viewer. In Editor mode, users can create objects, connect them, add documentation, and specify behavior in response to various events (such as mouse hover or mouse click). In view mode, users can interact with chart elements, read documentation for each object, download other charts, and more. By default, all charts open in view mode, thus providing an interactive knowledge base.

Chart editor

Working in Schemio is not much different from other similar editors, but it has its own characteristics. Chart elements can be organized in a hierarchical structure, allowing transformation from the parent element (rotation, translation, scaling) to be applied to them.

You can also give a diagram element a name, give it its own description, and add links and tags. If an object has its own description (the “Description” field), then in viewing mode, if you click on such an element, a window will appear with its name and description. This makes it possible to document the various elements of the diagram. There are two modes for displaying documentation: a pop-up window (tooltip) or a side panel (side-panel). The last option is convenient in case of a voluminous description.

If you add links to any element of the diagram, then in viewing mode, when you click on this element, links with smooth animation will appear.

One of the most important elements of any diagram is, of course, connecting lines. In Schemio, these are called connectors and can be attached to other elements, allowing them to be updated when those elements are moved.

Another feature of Schemio is the presence of a HUD (head-up display). Its peculiarity is that in viewing mode such an object will always be in the same place, regardless of scrolling in the diagram itself. This solution is very convenient for creating interactive menus and pop-up panels.

Components and dynamic loading of diagrams

The main idea behind Schemio was the ability to connect different diagrams together and load them dynamically. In this way, it would be possible to delve into the desired part of a complex system in detail without leaving the diagram itself. To implement this feature, Schemio has so-called components.

Components can be built-in (embedded) or external (external). Built-in components allow you to clone existing complex objects. Any changes to the original are instantly reflected on its copies.

External components load only in view mode and allow you to dynamically load other diagrams with a smooth zoom effect.

Behavior Editor

Interactivity is the key functionality of Schemio and allows you to create all kinds of animations, visual effects, and also implement complex scenarios in a diagram.

The behavior of objects is implemented using events. Any diagram element can determine not only its reaction to any event, but also create cyclic events or forward events to other elements. In this way, it is possible to implement unique behavioral functions.

Let's start with simple events “Mouse In” and “Mouse Out”. We can make it so that when the cursor hovers over an element, an animation of changing transparency is called.

By default, when adding an action, the “self” option is used, which means that the actions are applied to the same element on which the event was raised. But we can call any actions with any other elements. For example, we can add a “Clicked” event on our first element, but in the actions point to some rectangle in the diagram, thus simulating the opening of the panel when the button is clicked.

Showing an element smoothly is no longer a bad thing, but it’s even more interesting to combine it with other animations. Animations in the behavior editor can be launched in parallel with others if you select the “In Background” checkbox. Thus, along with the smooth appearance of the panel, you can also create a motion animation.

It is also important to note the “Init” event, which is called for each element of the diagram when it is loaded. This way, you can leave absolutely all the elements of the diagram in editing mode, but hide some of them when loading, so that the user gradually opens them. However, creating an initialization handler for each diagram element is not very convenient. In the case when there are many such objects and they all need to receive the same behavior, you can use tags. First you need to select these objects and mark them with some tag, for example “hide-on-init”.

Then we add a Dummy object to the scene and create a handler for the “Init” event. Select the “Hide” function and disable its animation, but instead of the “self” object we specify the “hide-on-init” tag. As soon as we enter view mode, this event handler will run the “Hide” function for all objects marked with the “hide-on-init” tag. Dummy objects are not drawn in view mode and are used primarily either to group other objects or to handle their events, as in this example.

It is possible to specify your own events (“Custom event”). When selecting this option, we can specify any event name, for example “Expand”. Such an event can be caused from other events of any objects. We can add the “Clicked” event to some other object and from there call the “Expand” event of the previous object. In the picture below you can see that it appears in the options among the various functions of objects.

This solution greatly simplifies working with complex scenarios in the diagram and allows you to implement various interactions of elements, as in the example of the drop-down panel below.

Dynamic behavior copying

The example above showed how to change the states of multiple objects using tags, but what if we want multiple objects to have the same behavior? For example: what if our diagram has a large number of buttons that change their transparency or color when hovered over? You can, of course, create one such button and simply copy it. But what if we already copied 20 of these buttons, but later decided that we needed to tweak their response to hover? In such cases, you can go another way. Instead of copying the button, we will simply select a tag for all buttons, for example “button”. After that, we create one template button object without a tag and implement the reaction to the mouse cursor in it. Then we create the “Init” event in it, in which we add the “Copy events” action. In “Copy events” itself, in the “Destination items” parameter, select our “button” tag. That's all, as a result, when we switch to view mode, all buttons marked with the “button” tag will copy the behavior of the template button and will react identically to the mouse cursor.

There are two more similar functions called “Copy links” and “Copy description”, which, as you might guess, copy links and descriptions for other objects. This can be useful if you have many different similar objects of the same type, but you don't want to constantly sync changed text or links.

Animations

Actually, in the previous examples you have already become acquainted with some animations. Almost all animation functions have a “Transition” parameter that determines the smoothness of the animation. Available options include: “linear”, “smooth”, “ease-in”, “ease-out”, “ease-in-out”, “bounce”.

Schemio has a ready-made set of animations: particle effect, blinking, outline animation, object morphing, scaling, rotation, etc.

Stop motion animation

In the previous examples we looked at the simplest animation functions, but what if we want to implement something more complex? In this case, we will need frame-by-frame animation, which is implemented using the “Frame Player” object.

Each “Frame player” object stores frames of changes to other objects in the scene and can be used in different ways. For example, we can use it as a button to start an animation.

Or, we can specify so-called animation sections, which will open additional buttons. There is a separate track for this in the animation editor called “Sections”. When creating a frame in such a track, you need to specify the section name. If the “Frame player” object has sections added, then in view mode it receives additional buttons. This way we can either play the entire animation or rewind the animation manually to the next or previous sections.

You can hide the “Frame player” object and trigger the animation from another object. In the script editor, if you select the “Frame player” object, two more functions “Play Frames” and “Stop Frame Player” will appear. The first function starts the animation in this player, allowing you to select the start and end frames. Animation “Stop Frame Player” stops the animation.

The frame-by-frame animation editor itself is a simple matrix of frames and properties of selected objects. In order to start working with some object, just click on the record button and start changing the desired object. Then a new track will appear and on it you can create and change animation frames.

If you need to animate the movement of an object along some path, then there is an additional function “Move along path”. It can be added to the matrix by clicking on the “Add path function” button. Then a new track will appear in which we can indicate, frame by frame, the relative location of the selected object on the contour of another object.

Using frame-by-frame animation you can achieve some very interesting visualizations. Here is an example of an animated ARP request diagram: https://schem.io/projects/site-reliability-engineering-t4kEQtKNxpSvZG0b/docs/arp-packet-0wgKzJg4RUI4z3a7

SchemioScript

Over time, I began to miss Schemio's capabilities. I wanted to implement more complex elements (eg tab bar, slide show, etc.), but it was difficult to do with just the existing animation features. So I started working on my own scripting language, SchemioScript. The language itself is a bit similar to JavaScript, but it is simpler. Although it was a very interesting problem, and I didn’t want to stop, I implemented only the most basic things, enough to program any complex behaviors in the diagram. The SchemioScript language is described in detail in this link. You can start programming in SchemioScript using the “Script” function in the behavior editor.

The script itself can be launched in three modes:

• One time – runs only once when calling the “Script” function

• Animation – runs the script as an animation, in which you can specify the smoothness of the animation and its duration. In this case, a variable is passed to the script into the scope twhich represents the animation progress from 0 to 1. That is, if we want to animate the movement along the X axis from 10 to 100, then it can be implemented like this: setX(10 * (1 - t) + 100 * t). The type of animation smoothness (transition) affects the change in parameter t. This way you don’t have to worry about the smoothness of the animation in the script itself.

• Infinite loop – the script runs in infinite animation mode. In this case there is no longer a variable tbut the variable is passed deltaTime, which is the difference in seconds between the current and previous frames. You can stop such an animation either by calling the “Stop all animations” function in the event editor, or by calling the function stop() inside the script. Both the “Animation” type and the “Infinite Loop” have additional scripts “Init” and “On Finish”, which are called at the beginning and end of the animation, respectively.

Here example scripted animation combined with built-in ability to move (drag-n-drop):

A small example of SchemioScript code:

// Находим объекты на сцене по имени и получаем их значение
mass = findItemByName('mass').getValue()
ropeStrength = findItemByName('rope-strength').getValue()
g = Vector(0, findItemByName('gravity').getValue()*1000)

// получаем позицию текущего объекта, для которого вызван этот скрипт
pos = getWorldPos()

// получаем позицию объекта объявленого в скрипте инициализации
anchorPos = anchor.getWorldPos()

airFrictionConst = findItemByName('air-friction').getValue() / 1000
airFriction = - airFrictionConst * mass * velocity * velocity.length() / 2

// в SchemioScript можно работать с векторными вычислениями
// также как и со скалярными
ropeVector = anchorPos - pos
distance = ropeVector.length()


Frope = ropeVector * ropeStrength * distance

acc = (mass * g + Frope + airFriction) / mass


velocity += acc * deltaTime
if (velocity.x > 10000) {
	velocity.x = 10000
}
if (velocity.x < -10000) {
	velocity.x = -10000
}
if (velocity.y > 10000) {
	velocity.y = 10000
}
if (velocity.y < -10000) {
	velocity.y = -10000
}

pos += velocity * deltaTime

// Изменяем позицию текущего объекта (для которого был вызван скрипт)
// Причем в это случае нам все равно, прикреплен ли объект к другому объекту
// или нет. setWorldPos расположит объект так что он будет соответствовать
// трансформации сцены.
setWorldPos(pos.x, pos.y)

Templates

Since it has now become easier to implement complex object behavior in Schemio, so-called Item templates have been added. For example, there are templates for designing application prototypes, a panel with tabs, a drop-down list, a slide show, etc. When using such a template, the editor allows you to update the generated object, even if you add other custom objects to it.

Collaborative editing in Schem.io

IN https://schem.io The ability to collaborate with diagrams has been implemented. Unfortunately, it does not yet have a real-time mode. Changes are applied in the form of patches, which makes it possible for several users to roll back or work with one document, merging their changes and preventing each other’s work from being overwritten. You can also organize projects and invite people to them with editing or viewing rights. But the most interesting functionality is Patch Requests. I wanted to develop Schemio not just as a diagram editor, but also as an interactive knowledge base accessible to everyone. Unfortunately, you can’t do all this alone, so I added the ability for any registered user to edit other people’s public diagrams and send Patch Requests to the authors. You can also make a fork of someone else’s diagram if the work is expected to take a long time, and then you can create a change request from the fork.

In such a Patch Request, you can highlight deleted, changed or added objects and see in detail the changes to all settings and text. Similar to GitHub, you can discuss a Patch Request in the comments, accept or reject changes.

Running Schemio from a Docker container

The browser part of Schemio, along with an alternative backend, is open source, distributed under the Mozilla Public License and posted on GitHub (https://github.com/ishubin/schemio). You can run the Schemio server from a Docker container. True, in this case the functionality on the backend will be very limited: the diagrams are simply stored on the file system, there is no user management and there is no functionality for resolving document change conflicts. You can start the Schemio container like this:

docker pull binshu/schemio:latest

docker run -v "$(pwd):/opt/schemio" \
    -p 4010:4010 \
    -e FS_ROOT_PATH=/opt/schemio \
    -e SERVER_PORT=4010 \
    binshu/schemio:latest

Export SVG, PNG and GIF

Schemio supports exporting a chart or selection not only in vector or raster format, but also in animated GIF format. To do this, select the options “Export as SVG”, “Export as PNG” or “Export animation” in the upper left menu. If you export an animation, you need to select the duration, size and number of frames per second and then Schemio will go into view mode and begin recording all interactions with the diagram.

Export to HTML player or iframe

It is possible to export the diagram to a completely autonomous HTML player, which you can place on your website. The player itself will be completely cut off from the functionality of the editor and will only contain viewing. To do this, select the “Export as HTML” option from the menu.

Future plans

I continue to develop Schemio and am currently working on expanding the functionality of templates and animations. In addition to the editor itself, I am also working in parallel on creating an interactive knowledge base. So far I started with the most common question in interviews, “What happens when we open a website in a browser”: https://schem.io/projects/site-reliability-engineering-t4kEQtKNxpSvZG0b/docs/what-happens-when-you-enter-a-website-in-a-browser-USqkMRHEY7JZav9t. In the future, I want to start documenting everything that might be interesting in IT topics: System Design, Kubernetes, Mysql, Kafka, algorithms, etc. Unfortunately, most of the time is spent both on the Schemio editor itself and on supporting its cloud platform. If you are interested in this project and the format for presenting information, I suggest sending your patches or creating your projects on https://schem.io.

Links