Reverse Engineering LHX Game Assets. Part 4

To new heights

At the end of the last post, it became obvious to me (it became obvious to others much earlier) that the decrypted models need to be converted into something younger than the 90s and more universal than a binary dump with custom commands. In other words, we need to somehow extract the geometry from the custom binary format into something supported by at least one 3D editor. I didn’t go far — I decided to convert the results into something as simple as a stick.obj-format (and it’s from the mid-90s, as it turns out…).

Non-painting of OBJ format

I won't describe it, of course. All details of a simple and well-known format that is already about 30 years old, but I will still only mention the key features that I used:

1. An OBJ file is a text file.

2. Each geometry element as a separate plain-text string.

3. Point (vertex) is described as “v XYZ”, where v is the letter v, and the characters X, Y and Z are the coordinates of the point (obviously (I hope)). The line describing the point can go anywhere in the file.

4. Polygon description format (face) allows you to specify a polygon with an arbitrary number of vertices. The polygon is defined through points (in the form of their numbers, of course). You can optionally specify texture coordinates and even normal coordinates for it (both are also points that need to be defined in the file, only not through v, but through vt and vn). I had neither, so I used a description in the form of “f n1 n2 n3 … nm”. Again, f is exactly the letter f, and n1..nm are the numbers of the points that define the polygon. Important nuances:

   1. All points described in the file BEFORE this polygon description form the actual list of points available for enumeration in this polygon. The point number in the description is the number in the list, not the line number. The count starts from 1.

   2. An unobvious trick. If the number of the specified point is positive, then the N-th point from the beginning of the list is taken. But if it is negative — then the point is counted from the end of the list(!).

   3. The polygon is formed by lines that go around the points in the specified order.

      

      1. Final line of the polygon Always goes from the last listed point to the first one – and therefore is not indicated.

      2. If you don't take into account the order of traversal, there will be trash (the red area in the picture is generally at the mercy of the rendering program, there are no guarantees of the same interpretation by different programs).

      3. OBJ suggeststhat the polygon has only one side. And if you look at it from the other side, it is transparent. In OBJ, it is considered that if you look perpendicularly at the visible side, then the points are listed clockwise.

5. Polygons can not only be covered with textures, but also simply painted. The epithet “simple” is a slight (slight) exaggeration. For painting you will need:

   1. A text file in .mtl format, which specifies the parameters (there are quite a few) of the “paints”. The main ones are the paint name (as a string), color (not in RGB, but in the values ​​of the Phong reflection model parameters) and transparency.

   2. Specifying this MTL file in the OBJ file.

   3. Specifying the paint name in the OBJ before the block describing the geometry to be painted.

6. Since the format is from the 90s, it allows you to specify primitives other than polygons that rely on points. Namely, points and lines. That's all. In fact, splines (which are also lines in some way) and even surfaces, but they write that no one really bothered with implementing support. So lines and points. No spheres. I'll get back to this a bit later.

Re-inventory

Of the supported in OBJ I had:

1. Lists of points

2. Geometry references with references to point numbers in the list:

   1. Polygons

   2. Lines

3. Colors and transparency

Of the unsupported things in OBJ, I had:

1. Geometry references with references to point numbers in the list:

   1. Double-sided polygons

   2. Invisible polygons

   3. Polygons with inverted normal

   4. Lines lying on the polygon

   5. Dots

   6. Spheres

2. There would be animations, if I could figure them out. But fortunately, I slapped my perfectionist on the wrists and didn't figure them out. But most likely, animations are set not in the model itself, but in the code.

I also had a personal requirement – that, in the limit, any model could be viewed in any modern viewer/editor.

The wonderful Kaitai Struct offers the ability to generate a parser listing on a whole bunch of languages from the description on Kaitai Struct DSL. That is, in the case of Windows:

• download the installer,

• you install,

• go to the folder with the composed description of the format in the Kaitai language

• you call the program by specifying the programming language you are interested in – and it gives out a parser program in this language. If you do not specify the desired language – the output will be in all possible ones. I chose C#, entered this command in the command line and kaitai gave me what I wanted:

kaitai-struct-compiler -t csharp lhx_points.ksy

The resulting parser takes the source bytes from the specified file as input, parses them according to the provided kaitai format description, and provides the data laid out on shelves. You connect this parser to your application, and it starts supporting this format.

So, I had everything I needed to write a converter from LHX to OBJ format (naturally, with a bunch of compromises and far-fetched problems).

About the converter

After some time spent on inspired coding, the result was a converter utility (link to github), which does the following:

Step 1. Receives conversion parameters as input + path to source files (there are 2 of them – .pnt with points and .bin with geometry description) without specifying extensions + path to output file (there will be 2 or 4 of them – depending on the number of details in the model – each will get its own obj and mtl file, high resolution will get the suffix -med to the name – because let's be honest, the detail there is medium at most). If something is wrong, it will give help. Example:
/code lhx2obj ‑infile init‑models\apache ‑outfile output‑models\apache

Step 2. Adds all the points to obj (changing the sign of the Y coordinate, because LHX draws models in mirror image – and this fact was quite difficult to simply see in the game, because almost all models are either symmetrical in Y, or chaotic). You can also tell the utility how to interpret the original coordinates – as XYZ or as XZY

Step 3. Translates geometry from bin to obj taking into account the user's preferences. Along the way, in addition to obj, an mtl file is generated with the correct colors and transparency values ​​of the material.

Step 4. Saves the result to output files.

I would like to highlight the features of conversion separately.

Separately highlighted conversion features

Firstly, although OBJ assumes that a polygon is one-sided, many programs that display it do not think so. A simple polygon is often drawn as 2-sided. And 2 polygons with opposite normal directions but the same coordinates are treated as 2 overlapping 2-sided polygons. It doesn't look very good.

Secondly, as I already mentioned, lines as a primitive in OBJ are quite supported and quite ignored by modern renderers. Blender will show them, of course, but will not render them. Because what is there to draw?

Thirdly, there are no spheres at all in OBJ. But in LHX commands and, as a result, in models there are.

For these and a couple of other reasons, I also had to learn the C# command line parameter support library (unpretentiously called CommandLine) – because the above nuances required at least some configuration.

Anything that is even slightly out of line with what is (from my point of view) logical is commented in some detail in the resulting file.

Polygon conversion.

1. If just polygons are specified, then the converter accordingly simply generates a record of a regular polygon (f n1 n2 n3…).

2. Polygons with inverted normal – depending on the parameters:

   1. Or skips it completely. Sometimes in models there were straight and inverted polygons – to make a double-sided one. And if you broadcast both of them – they will overlap each other.

   2. (Default) Or honestly adds it as a regular polygon with the required direction of traversal.

3. Double-sided polygons – depending on the parameters:

   1. Or simply generates one polygon with the traversal order specified in the initial command. Where the normal will look is generally unknown, so it is considered randomly set.

   2. (Default) Or it still generates 2 polygons with opposite traversal orders, but places them at a certain distance (specified in the utility launch parameters) to avoid overlapping and these terrible, terrible artifacts.

4. Invisible polygons – here everything is fair: since they are invisible, they have neither color nor normal. Therefore, they are translated simply as a set of points (here a point is a primitive, not a set of coordinates), on which they are based: “p n1, n2, n3…nm”. The standard allows you to list the coordinates of several graphic primitives “point” in one line.

Line conversion.

Since the “line” primitive is not always drawn, and you always want to see it, you had to make a line from something that is always drawn – polygons. The converter can generate six polygons instead of a line, which form a narrow (well, how narrow – depends on the specified parameters) parallelepiped of square cross-section, resting with its ends on the ends of the line.

I had to tinker with the ends to make them perpendicular to the original line, but everything worked out.

As a result, for both commands from LHX – both “line” and “line lying on the polygon” the converter offers, depending on the parameter, to do the following:

1. (Default) Or simply write a line primitive with the required points in the obj file,

2. Or it will register there a parallelepiped replacement with a thickness according to the specified parameters.

Sphere Conversion

If there was at least some support with the lines from OBJ, then here it is simply a clear and distinct – no. I went downhill and decided that since I had already imitated the line, I should try with the sphere too. But! An ideal hundred and fifty-polygon sphere is, of course, beautiful, but not absolutely not true. Therefore, I decided to support the spirit of the old school and make a sphere that minimally (ironic) eats up polygons, but still gives a decent result. And therefore the converter can generate polygons in place of the sphere that form a regular icosahedron, inscribed into a sphere of the required diameter. I think it turned out pretty.

As a result, if the converter encounters a command to draw a sphere, it, according to the parameters:

1. (Default) Or models an icosahedron of the required diameter with the center at the required point using polygons.

2. Or it specifies in a comment line in the file that they wanted a sphere of such-and-such diameter with a center at such-and-such point. And that's all our powers really are. This is precisely a comment – and such a sphere will never be drawn anywhere.

Conversion of the command to draw exactly a point.

The situation is strange. And OBJ supports a point (primitive point), and the converter can already simply generate an icosahedron of the required size, and no one needs this – I do not remember models that needed to draw points. But to be even, the converter still has the following parameters:

1. (Default). Or it will generate an OBJ primitive point

2. Or an icosahedron of a given size

Results

As a result, I adjusted the parameters and converted original models in as many as 3 variants of models in OBJ format:

1. The closest to the original and the historical spirit (points and lines as OBJ primitives, double-sided polygons are generated both without gaps, spheres are only mentioned). This can only be looked at with deep respect for the roots.

2. The most beautiful looking in modern viewers (points and lines are like spheres and parallelepipeds, double-sided polygons are generated on both sides, but with a gap, spheres are like icosahedra). If you want to do something with the models further (for example, upload them to a slicer), then this, in my opinion, is the most suitable option.

3. Something of a compromise (dots and lines are like spheres and parallelepipeds, one polygon instead of two for double-sided polygons, spheres are like icosahedra). Just because, why not?)

It is also worth considering that I did not figure out some elements – and among them was clearly the order of drawing polygons. That is why on the same Apache there is a polygon displaying part of the hull, and half of its points are tied to the cockpit polygons. And both of these polygons lie in the same plane. And they blink through each other. Such is life.

And another important point about animation. I, again, did not figure it out, but it is in the game – that is why some objects contain all animation stages (except rotation) at the same time. You have to live with it.

And now, after all these efforts, I have finally truly achieved my goal. The long-term gestalt is finally closed, I have obj files with almost all the objects from the game in my hands, and I am still a complete noob in DOS and disassembling – this is a personal triumph!

Cliffhanger 2

But here they are – kaitai, hexeditors, vscode, new knowledge, and here they are – other resource files.

Maybe we should dig around some more?