Working with audio in Unity - tracing and cloud computing

Hello everyone! My name is Ilya, I am from the TinyPlay team. In this article I would like to share how we work with audio. I hope you find this article helpful.

Audio architecture

The project consists of 3 components: sounds (steps, atmosphere, opponents, weapons and much more), music (atmospheric, action music, additional accompaniment) and voice acting (characters, audio diaries, loudspeakers). The audio architecture is divided as follows:

Sounds:

Environment and everything connected with it (foliage, wind, squeaks of trees, gates, etc.);
Opponent sounds (attack, damage, detection, death);
Footstep sounds for each of the surfaces (a total of more than 200 sounds for different physical materials);
Horror sounds (triggered);
Weapons (shooting, reloading, etc.);
Collision sounds for physical objects;
Sounds of effects (explosions, hits, sparks, etc.);
Other sounds;

Music:

Atmospheric background music;
Action music for specific scenes;
Intense music;
Music on the menu;

Voice acting:

Voice acting for characters in different languages;
Voice acting for audio diaries;
Voice acting for switches and other voice acting;

Compression settings for music and sounds are different. Music is streamed, large sounds are loaded and unpacked before loading the scene, small sounds are on the fly.

Working with music and sounds. Mixing

To achieve smooth transitions, different sound areas and work with compression – we use the standard Unity mixer. For smooth transitions – DOTween.

An example of how the music flows smoothly in the game:

isSwitchingMusic = true;
MasterMixer.DOSetFloat("MainMusic", -80f, 2f);
MasterMixer.DOSetFloat("SecondMusic", 0f, 2f);

Mixers also switch through special Volume-zones, which change the active mixer or its settings for the environment (for example, so that there is an echo effect in the basement).

An example of working with terrain material detection:

namespace TinyPlay.Components
{
    using UnityEngine;

    public class TerrainDetector
    {
        private TerrainData terrainData;
        private int alphamapWidth;
        private int alphamapHeight;
        private float[,,] splatmapData;
        private int numTextures;

        public TerrainDetector()
        {
            terrainData = Terrain.activeTerrain.terrainData;
            alphamapWidth = terrainData.alphamapWidth;
            alphamapHeight = terrainData.alphamapHeight;

            splatmapData = terrainData.GetAlphamaps(0, 0, alphamapWidth, alphamapHeight);
            numTextures = splatmapData.Length / (alphamapWidth * alphamapHeight);
        }

        private Vector3 ConvertToSplatMapCoordinate(Vector3 worldPosition)
        {
            Vector3 splatPosition = new Vector3();
            Terrain ter = Terrain.activeTerrain;
            Vector3 terPosition = ter.transform.position;
            splatPosition.x = ((worldPosition.x - terPosition.x) / ter.terrainData.size.x) * ter.terrainData.alphamapWidth;
            splatPosition.z = ((worldPosition.z - terPosition.z) / ter.terrainData.size.z) * ter.terrainData.alphamapHeight;
            return splatPosition;
        }

        public int GetActiveTerrainTextureIdx(Vector3 position)
        {
            Vector3 terrainCord = ConvertToSplatMapCoordinate(position);
            int activeTerrainIndex = 0;
            float largestOpacity = 0f;

            for (int i = 0; i < numTextures; i++)
            {
                if (largestOpacity < splatmapData[(int)terrainCord.z, (int)terrainCord.x, i])
                {
                    activeTerrainIndex = i;
                    largestOpacity = splatmapData[(int)terrainCord.z, (int)terrainCord.x, i];
                }
            }

            return activeTerrainIndex;
        }

    }
}

We use a player based on animation events, but you do it differently. Above is the class for defining terrain material. Using it, you can map the surface to sounds.

Surround sound. Ray tracing and geometry detection

We went through several options for working with surround and, most importantly, physically sound sound. We tried Steam Audio, FMOD, Microsoft Acoustics, and more.

As a result, our choice fell on FMOD + Dolby Atmos + Microsoft Acousitcs… Why not Steam Audio? Well, we were getting unexpected crashes in Windows builds from their library that we couldn’t debug.

What are we using all of this for?

To automatically calculate the propagation of sound in the environment, taking into account its geometry, materials and reflectivity (each object in the game is configured separately);
For spatial processing of audio using HRTF. In conjunction with Dolby Atmos, the sound is as natural as possible.

How it works?

The system takes into account the spatial position of the sound, the source of its reception, angle and dozens of other parameters for correct perception, both in headphones and on 5.1 systems.

In addition, the system uses ray tracing (real-time ray tracing) to cast rays to determine collisions and reflections applicable to sound waves. To reduce the load, these calculations are performed on the GPU + the sound is baked (similar to Light Probes, but for sound).

Cloud Computing for Sound Baking

It takes a long time to bake sound on a stage with thousands of 3D models and millions of polygons, so we use cloud computing from Azure. Fortunately, Microsoft has an excellent tool on virtual machines designed for this – Azure Batch

Using a cloud computing service, it is possible to reduce the time for baking sound from 2 hours to 10 minutes, which significantly saves time at a fairly low price ($ 1.5 per hour). At the same time, there is a free subscription to Azure for 12,500Р, which is suitable for novice developers.

Total

The combination of these technologies allows us to achieve more or less realistic sound, and he, in turn, can greatly affect the atmosphere of the game.

Useful links:

https://valvesoftware.github.io/steam-audio/

https://docs.microsoft.com/ru-ru/gaming/acoustics/

https://www.fmod.com/unity

https://developer.dolby.com/partners/middleware/fabric/