How we wrote a GPU-based Gaussian Splatting viewer in Unreal using Niagara

In this article I want to tell you about how we wrote a fully functional Gaussian Splatting viewer for Unreal Engine 5 from scratch.

Getting started

First of all, let's quickly recap what Gaussian splatting is. In short, it's a process that creates something similar to a point cloud, but instead of points it uses colored elliptical shapes (splats or 3D Gaussians). They change and stretch depending on camera position and perspective, merging into a continuous representation of space. This method allows you to preserve visual information such as reflections and light and shade in the captured virtual model, conveying its details as realistically as possible.

If you are interested in reading more about this method, I recommend that you read my previous articles:

Our first task was to understand which format to use for a 3D Gaussian file and which one is more common in the industry. After conducting in-depth research, we identified two main formats that are currently popular: .ply And .splat.

After some thought, we chose the format .ply as it covers a wider range of applications. This decision was also driven by consideration of other tools such as Super Splatwhich allows you to import 3D Gaussians only as .ply-files (although it also offers the ability to export them to .splat-files).

What is a .PLY file?

To begin with, there are two different types .ply-files:

.ply files with ASCII, in which data is stored in text form.
Binary .ply files with much worse readability.

You can think of .ply as a very flexible format for specifying a set of points and their attributes (in the body of the file), as well as a set of properties defined in its header. With their help, he tells the parser how to interpret the data contained in its body. For reference, Here A very informative guide to the general structure of .ply files.

Below is an example of what a typical .ply file for Gaussian splatting looks like:

ply
format binary_little_endian 1.0
element vertex 1534456
property float x
property float y
property float z
property float nx
property float ny
property float nz
property float f_dc_0
property float f_dc_1
property float f_dc_2
property float f_rest_0
(... f_rest from 1 to  43...)
property float f_rest_44
property float opacity
property float scale_0
property float scale_1
property float scale_2
property float rot_0
property float rot_1
property float rot_2
property float rot_3
end_header

The first line confirms that this is a .ply file.
The second line specifies whether the format of the data stored after the header is ASCII or binary (binary in this example).
The third line tells the parser how many elements the file contains. In our example there are 1534456 elements, that is, a 3D Gaussian.
Starting from the fourth line and up to the “end_header” line, the structure of each element is described as a set of properties, each of which has its own data type and name. In general, most Gaussian splatting .ply files follow the order of these properties. It is worth noting that regardless of the order, an important rule is that all non-optional properties must be defined in the file, and the data, in turn, must correspond to the declared structure.

After the section with the header comes the body of the file with the element data. For correct parsing, each element must strictly adhere to the order declared in the header.

This might give you an idea of what to expect when we want to describe a single Gaussian Splat element loaded from a ply.file:

Position in space as XYZ (x, y, z);
[Опционально] Normal vectors (nx, ny, nz);
Spherical functions of zero order (f_dc_0, f_dc_1, f_dc_2), which determine what color an individual 3D Gaussian should be using a special mathematical formula that calculates the resulting RGB value for rendering;
[Опционально] Spherical functions of higher order (from f_rest_0 to f_rest_44), which determine how the color of the 3D Gaussian should change depending on the camera position. This is necessary to improve the realism of the reflection or illumination information recorded in the 3D Gaussian. It's worth noting that this information is optional, and the files it is written to will weigh much more than files containing only zero-order functions;
Opacity (opacity), which determines the transparency of the 3D Gaussian;
Scale as XYZ (scale_0, scale_1, scale_2);
Orientation in space in the form of quaternions WXYZ (rot_0, rot_1, rot_2, rot_3).

All this information has its own coordinate system, which, after loading, must be converted to the Unreal Engine system. We'll talk about this in more detail a little later.

Now that you are familiar with the data we have to work with, we are ready to move on to the next step.

Parsing a .PLY file in Unreal

In our implementation, we wanted to support both ASCII and binary ply.files, so we needed a way to quickly parse and store their data appropriately. Luckily, .ply files are nothing new. They have been used for 3D models for a long time, even before Gaussian splatting became popular. Therefore, there are several .ply parsers on GitHub that we could use for this purpose. We decided to adapt the implementation Happily — a universal header-only parser ply written in C++ with open source code (many thanks to the author Nicholas Sharp).

Taking the parser implementation as a basis Happywe adapted its functionality to the Unreal standard and transferred it to the game engine, not forgetting about custom garbage collection and data types expected by Unreal. We then adapted our parsing code to the above Gaussian splatting framework.

The next logical step, after we figured out what the data looks like and how to read it from a file, was storing it. This meant that we needed a class or structure that could store all this data in the engine. It's time to dig into C++ code!

How can we define a 3D Gaussian in Unreal?

The simplest way to store 3D Gaussian data in Unreal was to define a custom USTRUCT, optionally available through Blueprints, that looks like this:

/**
 * Представляет собой данные, полученные в результате парсинга 3D-гауссианы, загруженные из PLY-файла.
 */
USTRUCT(BlueprintType)
struct FGaussianSplatData
{

GENERATED_BODY()

// Положение 3D-гауссианы (x, y, z)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
FVector Position;

// Векторы нормали [опционально] (nx, ny, nz)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
FVector Normal;

// Ориентация 3D-гауссианы в виде wxyz из PLY (rot_0, rot_1, rot_2, rot_3)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
FQuat Orientation;

// Масштаб 3D-гауссианы (scale_0, scale_1, scale_2)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
FVector Scale;

// Непрозрачность 3D-гауссианы (opacity)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
float Opacity;

// Коэффициенты сферических функций - нулевого порядка (f_dc_0, f_dc_1, f_dc_2)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
FVector ZeroOrderHarmonicsCoefficients;

// Коэффициенты сферических функций - высшего порядка (f_rest_0, ..., f_rest_44)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
TArray<FVector> HighOrderHarmonicsCoefficients;

FGaussianSplatData()
: Position(FVector::ZeroVector)
	, Normal(FVector::ZeroVector)
	, Orientation(FQuat::Identity)
	, Scale(FVector::OneVector)
	, Opacity(0)
	{
	}
};

Thus, at the parsing stage, we generate an instance of this structure for each 3D Gaussian and add them to the TArray array in order to use this data for visualization in the next stages.

Now that we understand the data, let's move on to the hardest and most interesting part: transferring the data to the GPU so that the Niagara system can read it!

Why Niagara?

Niagara is an ideal candidate for representing particles in Unreal. Specifically, the Niagara system consists of one or more emitters that are responsible for spawning particles and updating their state every frame.

In our specific case, we will use a single Niagara emitter to create the base implementation. For clarity, let's call it “GaussianSplatViewer“.

Now that we have our nice new emitter, we need a way to “pass” the 3D Gaussian data into it so that for each one we can spawn a corresponding point in space to represent it. You might be wondering if Unreal has any solution for this that we could use out of the box? The answer is yes, and it's called “Niagara Data Interface (NDI).”

What is Niagara Data Interface (NDI) and how do I use it?

Imagine you want to tell the Niagara emitter: “I'm reading a bunch of points from a file and I want to display them as particles. How do I make it clear to you what position each point should be in?” Niagara will respond: “Give me the correct NDI so I can understand your data and then extract the position for each particle from it.”

At this point, you may be wondering how to write NDI and where to even look for documentation for it? The answer is simple: the lion's share of Unreal Engine source code uses NDI for custom particle systems, and these in turn are a great source of inspiration for creating your own! We were most inspired by “UNiagaraDataInterfaceAudioOscilloscope“.

We needed to structure our custom NDI so that each 3D Gaussian would be “understood” by Niagara when passed along. Remember that this class will store the list of Gaussians that we loaded from the .ply file so that we can access their data from it and convert it into Niagara-compatible data types that will be used inside the particles.

First, we want our NDI class to inherit from UNiagaraDataInterface — the interface that Niagara needs to work with custom data types via NDI. To fully implement this interface, we need to override a few functions, which we'll talk about below.

Overriding GetFunctions

By overriding this function, we're telling Niagara, “I want you to see a list of the functions I'm defining so I can use them in your modules.” This way the system knows what inputs and outputs each of these functions should have, their names, and whether it is static or non-static.

// Определяем функции, которые мы хотим передать системе Niagara из
// нашего NDI. Например, мы определяем функцию для извлечения позиций из
// данных с гауссовым сплеэтингом.
virtual void GetFunctions(TArray<FNiagaraFunctionSignature>& OutFunctions) override;

This is what the implementation will look like GetFunctionswhich defines the function GetSplatPosition for the Niagara system. We want you to GetSplatPosition there were exactly 2 parameters and 1 result:

A parameter that refers to the NDI in which the 3D Gaussian array is stored (needed to access Gaussian data through this NDI from the Niagara system scratchpad module);
An integer parameter that specifies the position of which Gaussian we are requesting (it will match the ID of the particle from the Niagara emitter, so that each particle receives the position of its corresponding 3D Gaussian);
A Vector3 result that returns the XYZ position of a particular 3D Gaussian obtained at the provided index.

void UGaussianSplatNiagaraDataInterface::GetFunctions(
    TArray<FNiagaraFunctionSignature>& OutFunctions)
{   
   // Получаем позицию частицы, считывая ее из нашего массива гауссиан по индексу
   FNiagaraFunctionSignature Sig;
   Sig.Name = TEXT("GetSplatPosition");
   Sig.Inputs.Add(FNiagaraVariable(FNiagaraTypeDefinition(GetClass()),
       TEXT("GaussianSplatNDI")));
   Sig.Inputs.Add(FNiagaraVariable(FNiagaraTypeDefinition::GetIntDef(),
       TEXT("Index")));
   Sig.Outputs.Add(FNiagaraVariable(FNiagaraTypeDefinition::GetVec3Def(),
       TEXT("Position")));
   Sig.bMemberFunction = true;
   Sig.bRequiresContext = false;
   OutFunctions.Add(Sig);
}

In a similar way we define in GetFunctions and other functions, to get the scale, orientation, opacity, spherical functions and particle count of our 3D Gaussians. The particles will use this information to change shape, color, and position in space.

Overriding GetVMExternalFunction

This override is necessary so that Niagara can use the functions we declared in GetFunctions. This will make them available in Niagara graphs and scratchpad modules. Unreal has a macro designed for this purpose DEFINE_NDI_DIRECT_FUNC_BINDERwhich we will also use. Below is an example of a function definition GetSplatPosition.

// Мы биндим эту функцию, чтобы ее можно было использовать в графе Niagara
DEFINE_NDI_DIRECT_FUNC_BINDER(UGaussianSplatNiagaraDataInterface, GetSplatPosition);


void UGaussianSplatNiagaraDataInterface::GetVMExternalFunction(const FVMExternalFunctionBindingInfo& BindingInfo, void* InstanceData, FVMExternalFunction& OutFunc)
{
   if(BindingInfo.Name == *GetPositionFunctionName)
   {
       NDI_FUNC_BINDER(UGaussianSplatNiagaraDataInterface,
         GetSplatPosition)::Bind(this, OutFunc);
   }
}


// Функция, определенная под работу на CPU, понятная Niagara
void UGaussianSplatNiagaraDataInterface::GetSplatPosition(
  FVectorVMExternalFunctionContext& Context) const
{
   // Входные параметры - NDI и индекс частицы
   VectorVM::FUserPtrHandler<UGaussianSplatNiagaraDataInterface> 
     InstData(Context);


   FNDIInputParam<int32> IndexParam(Context);
  
   // Результат с положением
   FNDIOutputParam<float> OutPosX(Context);
   FNDIOutputParam<float> OutPosY(Context);
   FNDIOutputParam<float> OutPosZ(Context);


   const auto InstancesCount = Context.GetNumInstances();


   for(int32 i = 0; i < InstancesCount; ++i)
   {
       const int32 Index = IndexParam.GetAndAdvance();


       if(Splats.IsValidIndex(Index))
       {
           const auto& Splat = Splats[Index];
           OutPosX.SetAndAdvance(Splat.Position.X);
           OutPosY.SetAndAdvance(Splat.Position.Y);
           OutPosZ.SetAndAdvance(Splat.Position.Z);
       }
       else
       {
           OutPosX.SetAndAdvance(0.0f);
           OutPosY.SetAndAdvance(0.0f);
           OutPosZ.SetAndAdvance(0.0f);
       }
   }
}

note that GetSplatPosition defined this way for NDI compatibility with the CPU.

Overriding copy and equality

We also need to override these functions so that Niagara understands how to copy or compare the NDIs that our class uses. Specifically, we instruct the engine to copy the list of 3D Gaussians when copying a given NDI to a new one, and to determine whether two NDIs are the same based on the data for the Gaussians.

virtual bool CopyToInternal(UNiagaraDataInterface* Destination) const override;
virtual bool Equals(const UNiagaraDataInterface* Other) const override;

This function is necessary so that the Niagara system understands where our NDI functions should be executed – on the CPU or on the GPU. In this case, we originally wanted it to run on the CPU (for debugging), but for the release version we will change it so that it runs on the GPU. I will explain the reason for this choice later.

virtual bool CanExecuteOnTarget(ENiagaraSimTarget Target) const override { return Target == ENiagaraSimTarget::GPUComputeSim; }

Additional overrides needed to make our NDI work on GPUs as well

In order for us to tell Niagara how our data will be stored on the GPU and how the functions we declare will be translated for the GPU using HLSL shader code (more on this later), we need to override the following functions:

// Определения HLSL для GPU
virtual void GetParameterDefinitionHLSL(const FNiagaraDataInterfaceGPUParamInfo& ParamInfo, FString& OutHLSL) override;


virtual bool GetFunctionHLSL(const FNiagaraDataInterfaceGPUParamInfo& ParamInfo, const FNiagaraDataInterfaceGeneratedFunction& FunctionInfo, int FunctionInstanceIndex, FString& OutHLSL) override;


virtual bool UseLegacyShaderBindings() const override { return false; }


virtual void BuildShaderParameters(FNiagaraShaderParametersBuilder& ShaderParametersBuilder) const override;


virtual void SetShaderParameters(const FNiagaraDataInterfaceSetShaderParametersContext& Context) const override;

Niagara CPU and GPU based systems

Each Niagara particle system emitter can run on either a CPU or GPU. It is very important to immediately decide which of these two options we will ultimately choose, because each of them has its own side effects.

In the initial implementation, we used a CPU-based Niagara emitter. This was necessary to ensure that the 3D Gaussian data and coordinates were correctly reproduced in terms of position, orientation and scale in the Niagara system.

However, CPU-based emitters have a number of important limitations:

They cannot spawn more than 100,000 particles;
They rely solely on the CPU, which means each frame can take extra time away from executing other scripts, resulting in lower frame rates, especially when working with the maximum number of supported particles;
GPUs can handle extremely parallel tasks much better than CPU. This makes the GPU the best choice for handling large volumes of particles.

Although for debugging it is possible to tolerate a processor limit of 100 thousand particles, this is definitely not the figure that is needed for our scale. We need to support large files that can contain millions of particles.

For the second iteration, we decided to move to a GPU-based emitter. Not only does it rely entirely on the GPU without affecting the CPU, but it can also support up to 2 million spawned particles, which is 20 times more than the CPU.

A side effect of running on the GPU is that we also need to take care of allocating and managing GPU resources, which requires us to work with HLSL shader code and convert data between the CPU and GPU.

How to do this? As you might have guessed by extending our wonderful custom NDI.

From .PLY file to GPU via NDI

With our custom NDI, we have full control over how our data is stored in memory and how it is converted to Niagara-compatible form. Now the challenge is to implement this in code. We will break this task into two parts:

Let's allocate memory on the GPU to store Gaussian splatting data coming from the CPU.
Let's transfer the Gaussian splatting data from the CPU to the prepared GPU memory.

Preparing GPU memory for storing Gaussian splatting data

The first thing to know is that we cannot use Unreal data types such as TArray (which stores a list of 3D Gaussians in our NDI) when defining data on the GPU. This is because TArray is CPU-specific and is stored in RAM, which only the CPU can access. The GPU, on the other hand, has its own separate memory (VRAM) and requires special types of data structures to optimize access, speed and efficiency.

To store data collections on the GPU, we needed to use GPU buffers. There are different types of buffers:

Vertex Buffers: store vertex parameters such as position, normals and texture coordinates;
Index Buffers: used to tell the GPU the order in which vertices should be processed to form primitives;
Constant Buffers: Store values such as transformation matrices and material properties that remain constant across many frame rendering operations;
Structured Buffers and Shader Storage Buffers: More flexible because they can store a wide range of data types, needed for complex operations.

In our case, I decided to use a simple implementation in which each type of 3D Gaussian information is stored in a specific buffer (position buffer, scale buffer, orientation buffer, spherical function and opacity buffer).

Note that both buffers and textures are equally valid structures to consider for Gaussian data on a GPU. We chose buffers because we felt that their implementation was more readable and also avoided the problem with the texture-based approach where the last row of pixels would often end up not being completely filled.

To declare these buffers in Unreal, we need to add a definition for “Shader parameter struct“, which uses an Unreal Engine macro to tell the engine that this is a data structure supported by HLSL shaders (hence supported by GPU operations):

BEGIN_SHADER_PARAMETER_STRUCT(FGaussianSplatShaderParameters, )
   SHADER_PARAMETER(int, SplatsCount)
   SHADER_PARAMETER(FVector3f, GlobalTint)
   SHADER_PARAMETER_SRV(Buffer<float4>, Positions)
   SHADER_PARAMETER_SRV(Buffer<float4>, Scales)
   SHADER_PARAMETER_SRV(Buffer<float4>, Orientations)
   SHADER_PARAMETER_SRV(Buffer<float4>, SHZeroCoeffsAndOpacity)
END_SHADER_PARAMETER_STRUCT()

It's worth noting that these buffers can be optimized since the W coordinate is left unused in positioning and scaling (they only need XYZ). To reduce the memory they occupy, we could use channel packing techniques, but this is beyond the scope of this article. You can also use half precision instead of full floating point for optimization purposes.

Before the buffers we also define an integer to keep track of the number of Gaussians we need to process (SplatsCount), and vector GlobalTintrepresenting an RGB value that we can use to change the hue of our Gaussians. This definition is placed in the header file of our NDI class.

We also need to implement custom shader code for the GPU that will declare our buffers so they can be referenced later and used in custom shader functions. We communicate this to Niagara via an override GetParameterDefinitionHLSL:

void UGaussianSplatNiagaraDataInterface::GetParameterDefinitionHLSL(
  const FNiagaraDataInterfaceGPUParamInfo& ParamInfo, FString& OutHLSL)
{
  Super::GetParameterDefinitionHLSL(ParamInfo, OutHLSL);


  OutHLSL.Appendf(TEXT("int %s%s;\n"), 
    *ParamInfo.DataInterfaceHLSLSymbol, *SplatsCountParamName);
  OutHLSL.Appendf(TEXT("float3 %s%s;\n"),
    *ParamInfo.DataInterfaceHLSLSymbol, *GlobalTintParamName);
  OutHLSL.Appendf(TEXT("Buffer<float4> %s%s;\n"),
    *ParamInfo.DataInterfaceHLSLSymbol, *PositionsBufferName);
  OutHLSL.Appendf(TEXT("Buffer<float4> %s%s;\n"),
    *ParamInfo.DataInterfaceHLSLSymbol, *ScalesBufferName);
  OutHLSL.Appendf(TEXT("Buffer<float4> %s%s;\n"),
    *ParamInfo.DataInterfaceHLSLSymbol, *OrientationsBufferName);
  OutHLSL.Appendf(TEXT("Buffer<float4> %s%s;\n"),
    *ParamInfo.DataInterfaceHLSLSymbol, *SHZeroCoeffsBufferName);

Essentially this means that a Niagara system using our custom NDI will have this shader code under the hood. This will allow us to reference these GPU buffers in our HLSL shader code in later steps. To make the code more maintainable, we defined the parameter names as FString.

Transferring Gaussian splatting data from CPU to GPU

Now comes the tricky part: we need to “populate” the GPU buffers using C++ code as a bridge between CPU memory and GPU memory, defining how this data is transferred.

To do this, we decided to implement a custom “Niagara data interface proxy” – data structure used as “bridge» between CPU and GPU. This proxy helped us transfer data from CPU-side buffers to buffers declared as shader parameters for the GPU. To do this, we defined buffers and functions for their initialization and updating in the proxy.

I understand that this all seems very complicated, but from a logical point of view it is quite simple. The whole concept fits into this diagram:

Now that we have a complete understanding of our system, in order for it to work to its full potential, there are only a couple of small details left to be finalized.

We have already obtained buffer definitions for the GPU in the form of HLSL code using the function GetParameterDefinitionHLSL. Now we need to do the same for the functions we previously defined in GetFunctions. This is necessary so that the GPU understands how to translate them into HLSL shader code.

Let's take for example the function GetSplatPosition. We saw earlier how it was defined for use on the CPU. Now we need to expand its definition so that it is declared for GPUs as well. We can do this by overriding GetFunctionHLSL in our custom NDI:

bool UGaussianSplatNiagaraDataInterface::GetFunctionHLSL(
  const FNiagaraDataInterfaceGPUParamInfo& ParamInfo, const
  FNiagaraDataInterfaceGeneratedFunction& FunctionInfo, 
  int FunctionInstanceIndex, FString& OutHLSL)
{
   if(Super::GetFunctionHLSL(ParamInfo, FunctionInfo,
     FunctionInstanceIndex, OutHLSL))
  {
    // Если функция уже определена в родительском классе, 
    // то ее определение не нужно дублировать.
    return true;
  }
  
  if(FunctionInfo.DefinitionName == *GetPositionFunctionName)
  {
    static const TCHAR *FormatBounds = TEXT(R"(
      void {FunctionName}(int Index, out float3 OutPosition)
      {
        OutPosition = {PositionsBuffer}[Index].xyz;
      }
    )");
    const TMap<FString, FStringFormatArg> ArgsBounds =
    {
     {TEXT("FunctionName"), FStringFormatArg(FunctionInfo.InstanceName)},
     {TEXT("PositionsBuffer"),
       FStringFormatArg(ParamInfo.DataInterfaceHLSLSymbol + 
         PositionsBufferName)},
    };
    OutHLSL += FString::Format(FormatBounds, ArgsBounds);
  }
  else
  {
    // Возвращаем false, если имя функции не совпадает ни с одной из ожидаемых.
    return false;
  }
  return true;
}

As you can see, this part of the code simply adds to the line OutHLSL HLSL shader code that implements our function GetSplatPosition for GPU. Whenever Niagara is executed on the GPU and the function GetSplatPosition called by the Niagara graph, this shader code will be executed.

For the sake of brevity, I did not include code for other HLSL shaders for scaling, orientation, spherical, and opacity getter functions. However the idea is the same, we'll just add them inside GetFunctionHLSL.

Finally, the actual code for transferring data from the CPU to the GPU via DIProxy is handled by an override SetShaderParameters:

void UGaussianSplatNiagaraDataInterface::SetShaderParameters(
  const FNiagaraDataInterfaceSetShaderParametersContext& Context) const
{
  // Инициализируем параметры шейдера, чтобы они были одинаковыми с 
  // нашими буферами в прокси
  FGaussianSplatShaderParameters* ShaderParameters =
    Context.GetParameterNestedStruct<FGaussianSplatShaderParameters>();
  if(ShaderParameters)
  {
    FNDIGaussianSplatProxy& DIProxy = 
      Context.GetProxy<FNDIGaussianSplatProxy>();


      if(!DIProxy.PositionsBuffer.Buffer.IsValid())
      {
        // Инициализация буферов 
        DIProxy.InitializeBuffers(Splats.Num());
      }


      // Константы
      ShaderParameters->GlobalTint = DIProxy.GlobalTint;
      ShaderParameters->SplatsCount = DIProxy.SplatsCount;
      // Назначаем инициализированные буферы параметрам шейдера
      ShaderParameters->Positions = DIProxy.PositionsBuffer.SRV;
      ShaderParameters->Scales = DIProxy.ScalesBuffer.SRV;
      ShaderParameters->Orientations = DIProxy.OrientationsBuffer.SRV;
      ShaderParameters->SHZeroCoeffsAndOpacity =
        DIProxy.SHZeroCoeffsAndOpacityBuffer.SRV;
  }
}

In particular, buffer data is transferred from the NDI proxy (DIProxy) to the corresponding HLSL shader parameters controlled by the structure FGaussianSplatShaderParameters.

That's quite a lot of code! If you made it all the way here, congratulations! Now you're almost done with the low-level implementation. Let's go back one level and add some leftovers to complete the viewer!

Registering our custom NDI and NDI proxies in Niagara

And the last thing required to access our custom NDI inside Niagara property types is to register it with registry FNiagaraTypeRegistry. For convenience, we decided to do this in PostInitProperties our NDI, where we will also create an NDI proxy that will transfer data from the CPU to the GPU.

void UGaussianSplatNiagaraDataInterface::PostInitProperties()
{


  Super::PostInitProperties();


  // Создаем прокси, который мы будем использовать для передачи данных между CPU и GPU
  // (требуется для поддержки системы Niagara на базе GPU).
  Proxy = MakeUnique<FNDIGaussianSplatProxy>();
 
  if(HasAnyFlags(RF_ClassDefaultObject))
  {
    ENiagaraTypeRegistryFlags DIFlags =
      ENiagaraTypeRegistryFlags::AllowAnyVariable |
      ENiagaraTypeRegistryFlags::AllowParameter;


    FNiagaraTypeRegistry::Register(FNiagaraTypeDefinition(GetClass()), DIFlags);
  }


  MarkRenderDataDirty();
}

Here's a screenshot of our shiny new Niagara system using our custom NDI and getter functions!

Great difficulty in converting coordinates from PLY to Unreal

Currently, there is virtually no documentation on the Internet that explicitly specifies the transformations required to convert the data coming from a PLY file into Unreal Engine.

Here are some fun, painful trial failures we had to go through before we found the conversions we were looking for.

After much trial and error, we were finally able to establish the correct conversion. Below is a list of operations that need to be performed:

Position (x, y, z) from PLY 
Position in UE = (x, -z, -y) * 100.0f

Scale (x, y, z) from PLY
Scale in UE = (1/1+exp(-x), 1/1+exp(-y), 1/1+exp(-z)) * 100.0f

Orientation (w, x, y, z) from PLY
Orientation in UE = normalized(x, y, z, w)

Opacity (x) from PLY
Opacity in UE = 1 / 1 + exp(-x)

To maintain optimal performance, these transformations are performed at load rather than at runtime, so that once the 3D Gaussians are in the scene, frame-by-frame updating is not required.

This is how the resulting Gaussian Splatting viewer will show biker ply file from Will Eastcott after calculations as a result of the process I described in this article.

There are a few more code snippets for further geometric transformations and cropping, but they are already beyond the scope of this article.

Conclusion and Feedback

It was a very long journey, resulting in a very long article. But I hope it has inspired you to better understand how Niagara in Unreal can be configured to interpret your user data; how you can optimize its performance using GPU-based HLSL shader code input from your custom Niagara Data Interface and Niagara Data Interface Proxy; and finally seeing Gaussian splatting in the viewport after all that hard work!

Hurry up to join the open lesson on the topic “Creating a real-time strategy on Unreal Engine 5. Implementing control of AI characters and their behavior,” which will begin on October 17 at 20:00. You can sign up on the course page “Unreal Engine Game Developer. Basic”.

How we wrote a GPU-based Gaussian Splatting viewer in Unreal using Niagara

Getting started

What is a .PLY file?

Parsing a .PLY file in Unreal

Why Niagara?

What is Niagara Data Interface (NDI) and how do I use it?

Overriding GetFunctions

Overriding GetVMExternalFunction

Overriding copy and equality

Additional overrides needed to make our NDI work on GPUs as well

Niagara CPU and GPU based systems

From .PLY file to GPU via NDI

Preparing GPU memory for storing Gaussian splatting data

Transferring Gaussian splatting data from CPU to GPU

Registering our custom NDI and NDI proxies in Niagara

Great difficulty in converting coordinates from PLY to Unreal

Conclusion and Feedback

high-performance NLP pipelines in Rust

Object Detection in an Image Using YOLOv5 and YOLOv8 Models

How to increase software development efficiency. 5 major destinations

What does the security of modern applications consist of?

Can Artificial Intelligence Replace a Teacher in the Classroom?

What Vector DBMS Can Do – Open Source Tools & Cloud

Leave a Reply Cancel reply

Getting started

What is a .PLY file?

Parsing a .PLY file in Unreal

Why Niagara?

What is Niagara Data Interface (NDI) and how do I use it?

Overriding GetFunctions

Overriding GetVMExternalFunction

Overriding copy and equality

Additional overrides needed to make our NDI work on GPUs as well

Niagara CPU and GPU based systems

From .PLY file to GPU via NDI

Preparing GPU memory for storing Gaussian splatting data

Transferring Gaussian splatting data from CPU to GPU

Registering our custom NDI and NDI proxies in Niagara

Great difficulty in converting coordinates from PLY to Unreal

Conclusion and Feedback

Similar Posts

Leave a Reply Cancel reply