The magic of controlling the game via GigaChat (or any other model)

just watch the video, it has no sound

Just to try this for yourself – koiiiey/generative_ai_unity (gitverse.ru)

for those who don't want to watch the video

Hello everyone, my name is Pavel, and today I will show you the magic of (in the future obviously voice) control and input based on large language models.

The general idea was that in order to describe some action, theoretically it should be added to the prompt, then code should be written for it, all this should somehow be connected with each other, etc.
It sounds long and complicated, so I did it for you.

So, let's begin.

To do this, we will use an analogue of Function Calling in the form of a prompt (the prompt has already been driven into the project on Unity, be sure to download and have a look) and a drop of reflection.

Let's say that we want to make a bunch of bots that we can order to do something that we would previously have ordered to do with a mouse/keyboard/other input means, so let our bots now be able to be ordered to go to a point, recolor and change their size and all this separately or simultaneously.

[AI("Двигает куб на точку")]
public void MoveToPoint([AI("Номер точки")] int pointNumber)
{
    this.transform.position = points[pointNumber].transform.position;
}

[AI("Меняет размер")]
public void Resize([AI("от 0.001 до 2")] int size)
{
    this.transform.localScale = new Vector3(size, size, size);
}

[AI("Меняет цвет")]
public void Colorize(int r, int g, int b)
{
    GetComponent<Renderer>().material.color = new Color(r, g, b);
}

So, above is all the code you need to make it work!

Prompt just in case 🙂

You are a system for changing JSON state, perform JSON Method Calling to change state.

Just answer like this:
{“methods_calls”:[{“callOn”:{“name”:”имя объекта”,”goInstanceId”:”goInstanceId объекта”},”method”:”название метода”,”arguments”:[{“argumentName”:”имя параметра”,”value”:”значение”}]}]}

Use only those methods and only on those objects that are being discussed right now, that are needed for change and that the user has asked for

Don't call example methods – it will actually change the JSON!

If the list of available methods does not contain what you need, do not invent or propose a replacement, but simply refuse to make such a change.

Change only those objects in question, do not change those that do not ask.
Use only available methods and parameters.

Make sure the JSON is valid.
Make sure that the ID/parameter, etc. is valid.

JSON state (objects and available actions):
{{#STATE#}}

And so, after we launch our project, through reflection all objects with components that have an AI tag will be found, they will be collected in a cache based on the instanceId of game objects and all this fun will turn into a state in the form of JSON, which we will feed to our text model.

Well, then we just add user input – like “move the cube to the left”, “paint everything green”, “sell all goods cheaper than 200”, “don’t shoot, let’s be friends”.

That's all, let's add voice control 🙂