Sawing the Arcanum engine. Lesson 02. Working with game files, drawing the first sprite
The Arcanum game loads all its resources from the game archives with the .dat extension and the game's own directories.
You probably noticed that the original game, on old hardware, takes a long time to start and hangs on the splash screen. So, first the game loads a list of files from the dat archives, there are quite a lot of them, about 70k records. Then it proceeds to loading records from the archives in the module directory, the module is a new game that uses common resources located in the root directory.
How does a file load in a game?
The root of the game directory data/ is checked
If the file does not exist, load the file from the modules/module/data/ directory
If the file does not exist, loading from dat files modules/module/module.dat
If it is not there, then we load the file from the dat files of the game root.
Based on the logic, we will write the code.
To do this, we create a structure: DateItemthis structure contains fields that are in the dat archive sorted by file name. The archives themselves are gzip files combined into one large file with a header that describes from what offset the records about files and their number come.
class DatItem
{
public:
enum
{
Uncompressed = 0x01,
Compressed = 0x02,
MaxPath = 128,
MaxArchive = 64
};
DatItem();
int PathSize; //Размер пути в байтах
int Unknown1; //I don't now
int Type; // Сжатый или нет
int RealSize; //Размер несжатого файла
int PackedSize; //Размер сжатого файла
int Offset; // Смещение файла в архиве
char Path[MaxPath]; //Наименование файла, его полный путь
char Archive[MaxArchive]; //Путь до dat файла, в котором находится файл
};Class DatReader can open a dat file, read the header and go through each record in the file.
bool DatReader::Open(const std::string& file)
{
_File.open(file.c_str(), std::ios::binary);
if (_File.is_open())
{
int treesubs = 0;
_File.seekg(-0x1Cl, std::ios::end);
_File.seekg(16, std::ios::cur);
_File.seekg(4, std::ios::cur);
_File.seekg(4, std::ios::cur);
_File.read((char*)&treesubs, 0x04);
_File.seekg(-treesubs, std::ios::end);
_File.read((char*)&_TotalFiles, 0x04);
return true;
}
return false;
}Reading the entry looks like this:
bool DatReader::Next(DatItem& item)
{
if (_CurrentFile < _TotalFiles)
{
_File.read((char*)&item.PathSize , 4);
_File.read((char*)&item.Path , item.PathSize);
_File.read((char*)&item.Unknown1 , 4);
_File.read((char*)&item.Type , 4);
_File.read((char*)&item.RealSize , 4);
_File.read((char*)&item.PackedSize, 4);
_File.read((char*)&item.Offset , 4);
_CurrentFile++;
return true;
}
return false;
}Having data about a record, we can save this information, for example, in a table, in my case it is std::map, with open methods to add and get a record by name, since the name for the record is unique.
For this I added a class DatLoaderwhich uses DatReader to read and update DateList. I also add the path to the dat file to each entry, this is necessary so that I don’t physically go through all the dat files when searching for a file, but only access the indexed list of files in DatList.
To handle paths in the game directory and search correctly in modules, a class has been added PathManager
Initialization example:
PathManager("", "data/", "modules/", "Arcanum")This is the path to the game directory, by default it is empty,
This is the name of the directory where the general game files are located.
This is the name of the directory where the modules are located.
The name of the current game module
Let's create another class DateManager
This class, using a list of records, can search and unpack gzip files using zlib. As a result, the GetFile function accesses the Archive field and reads a gzip file from the archive specified in it. After that, a simple Uncompress method unpacks this file into RAM, a previously prepared buffer.
const std::vector<unsigned char>& DatManager::GetFile(const std::string& path)
{
_Result.clear();
DatItem* p = _DatList.Get(path);
if (p != NULL)
{
_File.open(p->Archive, std::ios::binary);
if (_File.is_open())
{
_File.seekg(p->Offset, std::ios::beg);
_Result.resize(p->RealSize);
_Buffer.resize(p->PackedSize);
if (p->Type == DatItem::Uncompressed)
{
_File.read((char*)&_Result[0], p->RealSize);
}
else if (p->Type == DatItem::Compressed)
{
_File.read((char*)&_Buffer[0], p->PackedSize);
if (!_Unpacker.Uncompress((unsigned char*)&_Result[0], p->RealSize, (unsigned char*)&_Buffer[0], p->PackedSize))
{
throw std::runtime_error("Can't uncompress file: " + path);
}
}
_File.close();
}
}
return _Result;
}Then the game works with this buffer placed in RAM for ease of operation, class MemoryReader. The class is very simple, it allows reading data and operating with offset. This is necessary so that other format loaders could have a universal interface to game files.
We also have files left in the game directories. I will do the same with them. Cool FileLoader loads into the buffer, a file from disk and also to generalize the work, created a class FileManager. This is a copy of DatManager, but it only allows you to work with files from the directory.
Now to unify the two file upload types into a uniform way, I created a class ResourceManagerwhich, having dependencies on the previous classes, contains a unified GetFile method, and already searches in available dat files, the root directory and the game module directory.
const std::vector<unsigned char>& ResourceManager::GetFile(const std::string& dir, const std::string& file)
{
const std::vector<unsigned char>& fromDir = _FileManager.GetFile(_PathManager.GetFileFromDir(dir, file));
if (fromDir.size() > 0)
{
return fromDir;
}
else
{
const std::vector<unsigned char>& fromModule = _FileManager.GetFile(_PathManager.GetFileFromModuleDir(dir, file));
if (fromModule.size() > 0)
{
return fromModule;
}
else
{
const std::vector<unsigned char>& fromDat = _DatManager.GetFile(_PathManager.GetFileFromDat(dir, file));
if (fromDat.size() == 0)
{
throw std::runtime_error("Can't found file: " + dir + file);
}
return fromDat;
}
}
}The code shows that I use the SOLID principle at its minimum. More precisely, the first two principles. Each class has minimal functionality and does one action. For example, the DatReader class only reads dat archives, DatLoader is dependent on this class and accepts the dependency through the constructor. And the remaining classes are also developed according to the same principle.
This provides several advantages:
The classes are really small and fit on one screen. I opened the file, looked at the implementation and immediately understood what it does and how it does it.
This is of course the ability to write tests for each class. Writing an engine is not a trivial task at all and is quite complex. At least for me as a backend developer, the area is quite new and therefore very interesting.
I didn't cover each class with an interface, because it just doesn't make sense. I don't need to substitute any fake classes or mock them and complicate the codebase even more. For testing, I use real game data. More about testing below.
The tests are in the Tests directory. And as an example I will give a couple of tests,
A very simple test.
#include <Arcanum/Managers/PathManager.hpp>
#include <Pollux/Common/TestEqual.hpp>
using namespace Arcanum;
int main()
{
PathManager pathManager("C:/Games/", "data/", "modules/", "Arcanum");
POLLUX_TEST(pathManager.GetFileFromDir("art/item/", "P_tesla_gun.ART") == "C:/Games/data/art/item/P_tesla_gun.ART");
POLLUX_TEST(pathManager.GetFileFromDat("art/item/", "P_tesla_gun.ART") == "art/item/P_tesla_gun.ART");
POLLUX_TEST(pathManager.GetFileFromModuleDir("art/item/", "P_tesla_gun.ART") == "C:/Games/data/modules/Arcanum/art/item/P_tesla_gun.ART");
POLLUX_TEST(pathManager.GetDat("arcanum1.dat") == "C:/Games/arcanum1.dat");
POLLUX_TEST(pathManager.GetModules("arcanum.dat") == "C:/Games/modules/arcanum.dat");
POLLUX_TEST(pathManager.GetModule() == "Arcanum");
return 0;
}From the code it is clear that the PathManager class forms paths to the game files in the engine. We pass the starting parameters to the constructors and the class operates with them.
The test is more complex:
#include <Arcanum/Formats/Dat/DatLoader.hpp>
#include <Arcanum/Managers/ResourceManager.hpp>
#include <Pollux/Common/TestEqual.hpp>
using namespace Arcanum;
using namespace Pollux;
int main()
{
std::vector<unsigned char> buffer;
std::vector<unsigned char> result;
DatList datList;
DatReader datReader;
DatLoader datLoader(datReader);
DatManager datManager(buffer, result, datList);
Pollux::FileLoader fileLoader(buffer);
FileManager fileManager(fileLoader);
PathManager pathManager("", "data/", "modules/", "Arcanum/");
ResourceManager resourceManager(pathManager, datManager, fileManager);
datLoader.Load("TestFiles/arcanum4.dat", datList);
MemoryReader* data = resourceManager.GetData("art/item/", "P_tesla_gun.ART");
POLLUX_TEST(data != NULL);
POLLUX_TEST(data->Buffer() != NULL);
POLLUX_TEST(data->Buffer()->size() == 6195);
return 0;
} The test checks reading and unpacking files from directories or dat files of the game. At the beginning, I initialize all dependent classes, as I do in the engine. After that, I simply submit test data and check the output data. In this case, ResourceManager should return an object with a certain size. In order to make sure that there were no errors during search and unpacking.
Tests written once allow me to constantly check whether I have broken the previous code. By commits, you can see the number of commits for code refactoring, there are about three of them and in them I change half of the code base and I can do this only thanks to tests.
I did not use the tests built into cmake. This is necessary to simplify testing on other platforms, for example, where cmake may be absent. For this, I wrote the simplest function for testing:
Pollux::TestEqual tests the assertion and prints an error message if it is not true. The POLLUX_TEST macro simply wraps the function for convenience.
#include <Pollux/Common/TestEqual.hpp>
#include <iostream>
using namespace Pollux;
void Pollux::TestEqual(bool condition, const char* description, const char* file, int line)
{
if (!condition)
{
std::cout << "Test fail: " << description << " File: " << file << " Line: " << line << '\n';
}
}namespace Pollux
{
void TestEqual(bool condition, const char* description, const char* file, int line);
#define POLLUX_TEST(x) Pollux::TestEqual(x, #x, __FILE__, __LINE__)
}After each code change I run tests, if there are no errors, the console remains empty. If there is an error I will be able to see information about it. Example:
I can see in which file and on which line something broke.
Well, we know how to upload files, but what next?
Let's go load the graphics, but first let's learn how to convert the game graphics into an rgba array from which we can create a texture.
Let's create a class ArtReader. It can read graphic game files with the art extension and convert them to an rgb array.
The original was taken as a basis converter art files to bmp from Alex.
Having adapted it for my engine, the result was not the most beautiful, but working code.
void ArtReader::Frame(size_t index, std::vector<unsigned char>& artBuffer, std::vector<unsigned char>& rgbBuffer)
{
size_t offset = _FrameOffset.at(index);
size_t size = _FrameHeader.at(index).size;
size_t width = _FrameHeader.at(index).width;
size_t height = _FrameHeader.at(index).height;
_Reader->Offset(offset);
artBuffer.resize(size);
_Reader->Read(&artBuffer[0], size);
rgbBuffer.resize(width * height * 4);
size_t j = 0;
if ((width * height) == size)
{
for (size_t i = 0; i < size; i++)
{
unsigned char src = artBuffer.at(i);
if (src != 0)
{
rgbBuffer.at(j + 0) = _Pallete[0].colors[src].r;
rgbBuffer.at(j + 1) = _Pallete[0].colors[src].g;
rgbBuffer.at(j + 2) = _Pallete[0].colors[src].b;
rgbBuffer.at(j + 3) = 255;
}
else
{
rgbBuffer.at(j + 0) = 0;
rgbBuffer.at(j + 1) = 0;
rgbBuffer.at(j + 2) = 0;
rgbBuffer.at(j + 3) = 0;
}
j += 4;
}
}
else
{
for (size_t i = 0; i < size; i++)
{
unsigned char ch = artBuffer.at(i);
if (ch & 0x80)
{
int to_copy = ch & (0x7F);
while (to_copy--)
{
i++;
unsigned char src = artBuffer.at(i);
if (src != 0)
{
rgbBuffer.at(j + 0) = _Pallete[0].colors[src].r;
rgbBuffer.at(j + 1) = _Pallete[0].colors[src].g;
rgbBuffer.at(j + 2) = _Pallete[0].colors[src].b;
rgbBuffer.at(j + 3) = 255;
}
else
{
rgbBuffer.at(j + 0) = 0;
rgbBuffer.at(j + 1) = 0;
rgbBuffer.at(j + 2) = 0;
rgbBuffer.at(j + 3) = 0;
}
j += 4;
}
}
else
{
int to_clone = ch & (0x7F);
i++;
unsigned char src = artBuffer.at(i);
while (to_clone--)
{
if (src != 0)
{
rgbBuffer.at(j + 0) = _Pallete[0].colors[src].r;
rgbBuffer.at(j + 1) = _Pallete[0].colors[src].g;
rgbBuffer.at(j + 2) = _Pallete[0].colors[src].b;
rgbBuffer.at(j + 3) = 255;
}
else
{
rgbBuffer.at(j + 0) = 0;
rgbBuffer.at(j + 1) = 0;
rgbBuffer.at(j + 2) = 0;
rgbBuffer.at(j + 3) = 0;
}
j += 4;
}
}
}
}
}Don't rush to throw tomatoes, in the following lessons, I will definitely correct it and refactor it. I think there is no point in me describing the format of art files, this format is perfectly described in the article on Habr.
The code above reads an art file, each frame is converted from palette pixels to 32 bit pixels and stored in a buffer. If there is data in the buffer, then we create a texture from the received data and display it on the screen.
MemoryReader* mem = _ResourceManager.GetData("art/scenery/", "engine.ART");
ArtReader artReader;
artReader.Reset(mem);
if (artReader.Frames() > 0)
{
std::vector<unsigned char> artBuffer;
std::vector<unsigned char> rgbBuffer;
artReader.Frame(0, artBuffer, rgbBuffer);
int w = artReader.Width(0);
int h = artReader.Height(0);
_Texture = new Texture(_Canvas, Point(w, h), 4, &rgbBuffer[0]);
}In further lessons, we will improve the code, add a sprite class that will store textures and related information. A sprite manager that will hide texture creation and sprite formation from us. Add code for displaying the map and objects on it.
As a result, a steam engine sprite will be displayed on the screen.
I will be glad to receive criticism, advice and suggestions. I understand that it is much easier to write code than to describe all this obscenity later:)
I will be glad to chat in the comments. I can, based on your advice on improving the engine code, supplement the article with code improvements from viewers.
