Nowadays, it has become fashionable to structure large systems in the form of many separate packages. The driving idea underlying this approach is that it’s better not to restrict people to a specific feature (proposed by you) by implementing a feature, but to provide this feature as a separate package that a person can download along with the basic system package.
To do this, in general terms, you will need …
- The ability not to download features that you do not need is especially useful on client systems.
- The ability to replace with another implementation a piece of functionality that does not meet your goals. Thus, the load on the kernel modules is also reduced – it is not required to cover all possible practical cases with their help.
- Checking kernel interfaces in real conditions – by implementing basic features on top of the interface facing the client side, you are forced to make this interface at least so powerful that it can support these features. Thus, you can be sure that it will be possible to build on it similar things written by third-party developers.
- Isolation between parts of the system. Project participants can simply search for the package they are interested in. Packages can be versioned, marked as unwanted, or replaced without affecting the core code.
This approach is usually associated with increased complexity. To make it easier for users to start, you can provide them with an “all inclusive” wrapper package, but sooner or later they will probably have to get rid of this shell and install and configure specific auxiliary packages, which is sometimes more difficult than just switching to another feature in the monolithic library.
In this article, we will try to discuss options for expansion mechanisms that support “large-scale extensibility” and allow you to create new extension points where this was not provided.
What do we want from an extensible system? First of all, of course, it should have the ability to expand its own capabilities using external code.
But this is hardly enough. Let me digress about one stupid problem that I once encountered. I am developing a code editor. In one of the earlier versions of this editor, you could set the style for a specific line in client code. It was great – selective line layout.
Except for the case when attempts to change the appearance of the line are made immediately from two sections of the code – and these attempts begin to butt. The second extension applied to the line overwrites the style of the first extension. Or, when the first code tries to remove the design it has made somewhere in the further part of the code, it overwrites the style introduced by the second code fragment.
We managed to find a solution, providing the ability to add (and delete) the extension, rather than defining it, so that two extensions could interact with the same line without sabotaging each other’s work.
In a more general sense, it is necessary to ensure that the extensions can be combined, even if they are completely unaware of the existence of each other – without causing conflicts between them.
To do this, you need to make sure that any number of actors can affect each expansion point. How exactly multiple effects will be processed depends on the situation. Here are some approaches you might find useful:
- All of them take effect. For example, when adding a CSS class to an element or when displaying a widget, both of these features are added at the same time. Often, they will still have to be sorted in some way: widgets should be displayed in a predictable, well-defined sequence.
- They line up in the form of a conveyor. An example is a handler that can filter changes that are added to a document before they are made. Each change is first fed to a handler, which, in turn, can additionally change it. Ordering in this case is uncritical, but it can make a difference.
- You can apply a first-come-first-serve approach to event handlers. Each handler has a chance to serve the event until one of them says that it has already dealt with it, after which the handlers standing in line for it are no longer interrogated.
- It also happens that you really need to choose a specific value – for example, determine the value of a specific configuration parameter. It may be advisable to use a certain operator (say, logical and, logical or, minimum or maximum) to limit the number of input values for one position. For example, an editor may switch to read-only mode if any extension orders it. Either you can set the maximum value of the document, or the minimum number of values that are reported to this option.
In many such cases, order is important. This means that the priority of the applied effects should be controllable and predictable.
It is on this front that imperative extension systems based on the use of side effects are usually unable to cope. For example, operation
addEventListenerexecuted by the browser DOM model causes the event handlers to be called exactly in the order in which they were registered. This is normal if all calls are controlled by a single system, or if the order of operations is really not important, however, when you have to deal with a lot of software fragments that independently add handlers, it can be very difficult to predict which of them will be called in the first place.
To give you a simple example: I first applied a modular strategy to ProseMirror, a system for editing rich text. The core of this system in itself is, in principle, useless – it relies entirely on additional packages describing the structure of documents, binding keys, leading the history of cancellations. Although it’s really a bit difficult to use this system, it has been adopted in products that require custom text design, which is not available in classic editors.
The extension mechanism used in ProseMirror is relatively straightforward. When creating the editor, the client code indicates a single array of connected objects. Each of these plugins can somehow affect the work of the editor, for example, add pieces of status data or handle interface events.
All these aspects are designed to work with an array of configuration values using the strategies outlined in the previous section. For example, when specifying multiple key assignments, the order in which keymap plugin instances are specified determines their priority. The first keymap that knows how to handle it receives a specific key in processing.
Usually this mechanism is quite powerful, and it is actively used. However, at a certain stage, it becomes complicated and it becomes uncomfortable to work with it.
- If the plugin has many effects, then you can either hope that in that order they will be applied to other plugins, or you will have to break them into smaller plugins so that you can organize them correctly.
- In general, organizing plugins becomes very sensitive, as the end user does not always understand which plugins can affect the operation of other plugins if they get a higher priority. All errors usually appear only at runtime, when using specific functionality – therefore, they are easy to miss.
- Plugins based on other plugins should document this fact – and it remains to be hoped that users will not forget to enable their dependencies (in the correct order).
CodeMirror in version 6 Is rewritten editor of the same name. In the sixth version, I try to develop a modular approach. This requires a more expressive extension system. Let’s look at some of the challenges associated with designing such a system.
It is easy to design a system that provides complete control over the order of extensions. But it is very difficult to design such a system, which at the same time will be pleasant to use and allow you to combine the code of independent extensions without extensive and thorough manual intervention.
When it comes to ordering, it pulls to apply priority values. A similar example is a property
z-index in CSS, which allows you to specify a number that indicates how deep the item will be on the stack.
Since ridiculously large values are sometimes found in style sheets
z-index, obviously, this way of indicating priority is problematic. A particular module individually “does not know” which priority values indicate other modules. Options are just points in an undefined number range. You can specify prohibitively high (or deeply negative values), hoping to get to the ends of this scale, but everything else is a guessing game.
This situation can be improved somewhat by defining a limited set of clearly defined priority categories so that extensions can be classified by the approximate “level” of their priority. But you still have to somehow break the ties within these categories.
Grouping and Deduplication
As I mentioned above, once you start seriously relying on extensions, situations may arise in which some extensions will use others. Mutual dependency management does not scale well, so it would be nice if you could pull a group of extensions at once.
However, not only that, in this case, the problem of ordering will aggravate even more; another problem will arise. Many other extensions can depend on a particular extension at once, and if you represent them as values, then the situation with multiple downloads of the same extension may well arise. In some cases, for example, when assigning keys or handling event handlers, this is normal. In others, for example, when tracking cancel history or working with a tooltip library, such an approach would be a waste of resources with the risk of breaking something.
Therefore, allowing the composition of extensions, we are forced to shift to the extension system part of the complexity associated with managing dependencies. You need to be able to recognize those extensions that should not be duplicated, and download only one instance of each of them.
However, since in most cases extensions can be configured, and all instances of a particular extension will be somewhat different from each other, we cannot just take one instance of the extension and use it – we will have to combine them in some meaningful way (or report an error, when this is not possible).
Here I will describe what has been done in CodeMirror 6. I propose this example as a solution, and not as the only true solution. It is possible that this system will develop further as the library stabilizes.
The main primitive in this approach is called behavior. Behaviors are just those things that you can expand by indicating values. As an example, consider the behavior of a state field, where with the help of extensions you can add new fields, giving a description of each field. Or the behavior of a browser-based event handler, where you can add your own handlers using extensions.
From the point of view of consumer behavior, those behaviors that are configured in a particular instance of the editor give an ordered sequence of values, where values with higher priority come first. Each behavior has a type, and the values for it must match that type.
Behavior It is represented as a value used both to declare an instance of behavior and to refer to the values that the behavior may have. For example, an extension that defines the background of a line number may define behavior that allows another code to add new markers to this background.
Expansion Is a value that can be used when configuring the editor. An array of extensions is reported during initialization. Each extension is allowed in zero or more behaviors.
The simplest type of extension is an instance of behavior. By setting a value for this behavior, we get in response the value of the extension that implements this behavior.
A sequence of extensions can also be grouped into a single extension. For example, in the configuration of the editor for a given programming language, a number of other extensions can be pulled up, in particular, a grammar for parsing and highlighting the language, information on how to indent, and also a source of information about completion that intelligently complements the code in that language. So you get a language extension, which simply collected all the necessary extensions that give the cumulative value.
Describing a simple version of such a system, we could stop at this and simply fit the nested extensions into a single array of extensions for behaviors. Then they could be grouped by type of behavior and get ordered sequences of behavior values.
However, we still have not figured out deduplication, and we need more complete control over ordering.
Values of the third type include unique extensions, this is the mechanism for ensuring deduplication. Extensions that you do not want to instantiate twice in the same editor are just that. To determine such an extension, a spec type is specified, that is, the type of configuration value expected by the extension constructor, and an instantiation function that takes an array of such spec values and returns the extension.
Unique extensions complicate the process of resolving a collection of extensions into a set of behaviors. As long as there are unique extensions in the aligned set of extensions, the resolution mechanism should select the type of unique extension, collect all its instances, call the instantiation function with their spec values and replace them with the result (in one instance).
(There is one more hitch – they must be resolved in a certain order. If you first enable the unique extension X, but then the extension Y resolves to another X, this will be an error, since all instances of X must be combined together. Since instantiating the extensions is a pure operation, the system, faced with it, executes it by trial and error, restarting the process – and recording the clarified information.)
Finally, let’s talk about priority. The basic approach in this case is to maintain the order in which extensions were reported. Compound extensions are aligned and built into this order exactly at the position where they first meet. The result of resolving a unique extension is also inserted at the place where it first occurs.
But extensions can assign some of their subextensions to a category with a different priority. The system determines the types of such categories: rollback (takes effect after other things happen), by default, expand (higher priority than the bulk) and redefine (perhaps should be located at the very top). Actual ordering is carried out first by category, and then by starting position.
So, an extension with a low-priority key assignment and an event handler with the usual priority can give us a compound extension built on the basis of an extension with a key assignment (in this case, you do not need to know what behaviors are included in it, with the priority “rollback” plus an instance of behavior event handler.
The main achievement seems to be that we have acquired the ability to combine extensions, regardless of what is done inside each of them. In the extensions we have modeled so far, among which: two parsing systems with the same syntactic behavior, syntax highlighting, smart indentation service, cancellation history, line number background, parenthesis auto-closing, key assignment and multiple selection – everything works well.