Organizing game data using the example of the game Mind Over Magic
About the game
Mind Over Magic — is a simulation game developed by Sparkypants and published by Klei Publishing.
Design, build, and optimize every aspect of your perfect school of magic! Keep everyone alive and happy as you prepare your students to explore the horrors of the Underschool, a deadly underground trial. Study lost arcana, grow exotic plants, brew potions, and raise the undead. But don't forget to stop the deadly fog from consuming everything in its path. – from the official Steam page.
The game features building, research, exploration, and turn-based combat, making it a great example for discussing game data organization.
Physical file locations and formats
Mind Over Magic is developed on Unityso all the compiled code is in the game folder /MindOverMagic/mindovermagic_Data/Managedand game resources are one level higher. Game data, including tables and settings, are stored in more than three hundred .yaml files in the folder StreamingAssets.
YAML is a text format that is unique among text formats in that all serializer implementations are unique and partially compatible.
The project also uses serialization in JSON by means of Newtonsoft.JSONHowever, most likely, the YAML format was chosen because of its more compact syntax and ability to work with multi-line text.
Loading game data
Static game data consists of entity configurations Config and definitions IDefinition.
For each YAML file there is a special directory class that inherits DefinitionCatalog<T>Where T — is a class that describes the data structure of each entry in the directory. For example, SpellConfigCatalog inherits DefinitionCatalog<SpellConfig> and provides methods to access the collection, and SpellConfig its fields describe the structure of each spell (name, description, effects, mana cost, etc.).
Data from these files is accessible in game code via global singletons. DefinitionCatalog<T>.Instancewhich are initialized by the class ConfigBundlewhich in turn is created IConfigBundleProvider.
Entity configurations are accessible via a global singleton ConfigData.Instance and class Archetype. Archetype defines the components that the ECS entities created according to this configuration will have.
Each file is uploaded in a separate Taskand then all tasks are expected in Task.WaitAllwith code similar to the following:
DefinitionCatalog[] allCatalogs = GetAllCatalog();
Task loadTasks = allCatalogs.ConvertAll(catalog =>
{
catalog.Load(parserType);
});
Task.WaitAll(loadTasks);
// post processing for catalogs
ResearchTechCatalog.Finalize();
There are 68 such definition classes in total, another 95 config classes for ECS components, and 26 validator classes for catching errors in configurations. All these classes are written and supported by programmers.
Or all the same in single-threaded execution. As a parameter is passed parserTypewhich has three variants: Yaml.Net, Fast, Fast Single Thread. The parser variants indicate attempts to find the fastest solution for loading data into the game. Implementation Fast looks like a custom YAML parser, with a tokenizer containing over 1000 lines of code, which highlights the complexity of the YAML format.
Deserialization into objects is also done custom, using reflection without caching metadata fetch calls (e.g. type.GetProperties called for each record) and without optimizations through code generation, Linq.Expressions or MSIL generation. This direct approach to serialization works on all platforms, but is the slowest.
On my PC this process takes 2944.104ms according to the logs and the game's built-in profiler.
Accessing configurations and definitions from code
As mentioned, all configuration directories are accessible through the global instance of the class. DefinitionCatalog<T>.Instance And Simulation.Instance.Configs. The relationship between the handling options is unclear; it could be a split between combat and build mode, or it could just be legacy.
Configurations are retrieved from the catalog by string values directly at the application site. For example, to get the complexity level:
var gamePressure = DefinitionCatalog<GamePressureDefinition>
.Instance
.TryGetDefinitionFromStringKey("Relaxed", out var id);
Relationships between configurations, where one references another, are handled similarly via passing DefId to another DefinitionCatalog<T>.Instance:
DefinitionCatalog<ResearchTechDefinition>.Instance.TryGetDefId("TechTreeRoot", out var techTreeRootId);
var subResearches = DefinitionCatalog<ResearchTechDefinition>
.Instance
.AllDefinitions()
.Where(subResearch => subResearch.ParentTech == techTreeRootId);
How could this be implemented?
By studying how others design, store, and use static game data, I aim to improve my game data editor, Charon (Charon).
In the case of this game, I simulated about 15 definitions using the editor's built-in AI assistant:
First, I converted all the data from YAML to JSON, which yielded about 8MiB of data. The definitions I mocked up earlier capture about 1MiB of the original game data. From the editor Charon I exported all the data into one JSON file and an additional one MessagePack 550KiB file for download speed comparison.
Charon generates C# code with classes for all definitions (called schemas) and built-in formatters for JSON and MessagePack. So everything is packaged inside the generated code, with no external dependencies.
To test the loading speed, I created a simple console application in C# and loaded the game data using the following code:
var jsonGameDataPath = @"\publication.json";
var messagePackGameDataPath = @"\publication.msgpack";
var sw = Stopwatch.StartNew();
var gameData = new GameData(File.OpenRead(jsonGameDataPath), new Formatters.GameDataLoadOptions { Format = Formatters.GameDataFormat.Json });
Console.WriteLine($"Load JSON: {sw.ElapsedMilliseconds:F2}ms");
sw = Stopwatch.StartNew();
gameData = new GameData(File.OpenRead(messagePackGameDataPath), new Formatters.GameDataLoadOptions { Format = Formatters.GameDataFormat.MessagePack });
Console.WriteLine($"Load MesagePack: {sw.ElapsedMilliseconds:F2}ms");
The result was as follows:
Load JSON: 78.00ms
Load MessagePack: 16.00msQuite good. I wrote about how I sped up the JSON formatter in my previous article. If we extrapolate this to all the game configurations and definitions, which take up about 8MB, then they would load in the game in 624ms instead of 2944ms.
One class would be needed to access all game data. GameDatasimilar to the one already in use ConfigBundle.
To retrieve specific documents, you can use generated “Id classes” instead of string literals:
var champion1 = gd.Badges.Get(BadgeId.Champion1);The advantage of such “Id classes” is that when a document is deleted, the code stops compiling, and with a string constant it will simply break quietly in production.
References to other documents in the code generated by Charon will be automatically checked and replaced with the real document instance. The original code required manual resolution of references:
// Original code
DefinitionCatalog<BadgeRewardDefinition>
.Instance
.TryGetDefinitionFromStringKey("Heart_Tier1Reward_1", out var heartTier1);
DefinitionCatalog<BadgeCategoriesDefinition>
.Instance
.TryGetDefinition(heartTier1.CategoryId, out var heartTier1Category);
// Charon generated code
BadgeCategory heartTier1Category = gd.BadgeRewards
.Get(BadgeRewardId.HeartTier1Reward1)
.CategoryId;
Unfortunately, the hot reloading feature of game data into the same objects in memory that is present in the original game code is missing from the code generated by Charon. It is impossible to implement due to the immutability of game data after loading.
Modding support
Although there is no modding information in the original game code, modders will still find a way. Meanwhile, Charon supports loading patches on top of the game data, which allows the game to be modified.
For example, with game data like this:
"Badge": [
{
"Id": "SoloRecreator_1",
"DisplayName": "Recreating like a wild - Bronze",
"CategoryId": "HeartCenter"
}
]Modder #1 can create a patch to change the name of this icon:
"Badge": {
"SoloRecreator_1": {
"DisplayName": "Recreating like a boss - Bronze"
}
}And modder #2 can delete the current icon and add a new one:
"Badge": {
"SoloRecreator_1": null,
"SleekDancer_1": {
"Id": "SleekDancer_1",
"DisplayName": "Dancing like a wild - Bronze",
"CategoryId": "HeartCenter"
}
}It will be possible to download base game data and a list of patches from mods to get modified game data:
var gd = new GameData(File.OpenRead(jsonGameDataPath),
new Formatters.GameDataLoadOptions {
Format = Formatters.GameDataFormat.Json,
Patches = new Stream[] {
File.OpenRead("Patch1.json"),
File.OpenRead("Patch2.json"),
}
});This allows for easy modification of game configurations. The Charon editor itself can be freely distributed along with the game, and creating patches can be a one-button process within the editor, making modding the game a breeze.
Conclusion
The approaches to solving problems in game development have remained unchanged since the earliest days of the industry. Everything that can be reinvented will be reinvented, tools will be ignored, and releases will be delayed. Don't be like that! Learn, develop, and keep making games. Write to me if you want to implement CharonI will help at all stages.
