ChatGPT Deep integration

I consider myself an indie developer, my most serious work is MMO Project zone 2d online (online version of the game “STALKER” in 2D). When I learned about ChatGPT, I decided that I definitely needed to make a game using it. The first thing that comes to mind when it comes to ChatGPT and games is working with text: dialogues, quests, plot. But this is banal, boring and not interesting. I want the AI ​​to be able to control literally every aspect of the gameplay.
Now I am at the initial stage, and nevertheless I managed to find a way in which everything I had planned could be realized. The fundamental thing in every game is the movement of objects, so that's where I decided to start.

What's under the hood?

The main idea consists of three steps.

• We send a request to ChatGPT and receive a response in the form of a code.

• We execute this code and get information.

• We use the information received for our own purposes.

As you can see, everything is quite simple, but nevertheless, after searching a little on the Internet, I did not find anything like this. This method can be implemented in any programming language, and is suitable for a wide range of tasks.

My program is written in java language. To connect to the ChatGPT API I used the library openai-java by Theo Kanning. To display the interface and graphics, the LibGDX framework was used. Also, I used the free version of ChatGPT “gpt-3.5-turbo-0613”.

Read more.

Initially, I planned to receive ready-made data from the AI. But this gives much less freedom, and does not work at all. ChatGPT can't handle some problems if you ask it not to write code, but it will solve the same problem correctly using code.

Java is a compiled programming language. This means that all code must go through the compilation stage before running the program. However, when we receive the code from ChatGPT, the program is already running. Therefore, it is required to compile the code on the fly. The compiler library is responsible for this JavaCompiler , which is built into java by default. Here is a useful article on this topic. Here's an example from my project: It compiles a class called Main and runs the run() method.

JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
    DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<>();
    StandardJavaFileManager fileManager = compiler.getStandardFileManager(diagnostics, null, null);
    JavaFileObject fileObject = new JavaSourceFromString("Main", code);
    Iterable<? extends JavaFileObject> compilationUnits = Collections.singletonList(fileObject);
    JavaCompiler.CompilationTask task = compiler.getTask(null, fileManager, diagnostics, null, null, compilationUnits);
    boolean success = task.call();
    if (success) {
        System.out.println("Compilation succesful");
        URL[] urls = new URL[]{new File("").toURI().toURL()};
        URLClassLoader classLoader = new URLClassLoader(urls);
        Class<?> cls = classLoader.loadClass("Main");
        Object obj = cls.newInstance();
        cls.getMethod("run").invoke(obj);
        System.out.println("start function run");
    } else {
        for (Diagnostic<? extends JavaFileObject> diagnostic : diagnostics.getDiagnostics()) {
            System.out.println(diagnostic.getCode());
            System.out.format("Error on line %d in %s%n", diagnostic.getLineNumber(), diagnostic.getSource().toUri());
        }
    }
    fileManager.close();

Our code knows nothing about the code sent from ChatGPT, and vice versa, the code sent from ChatGPT knows nothing about our code. Therefore, they cannot interact with each other. All that remains is to get some result after executing the new code and then use this result in your project. Specifically, in my case, game objects change the values ​​of their coordinates in a loop. So I asked ChatGPT to write the coordinates of each object to a test file each iteration of the loop. Each class received from ChatGPT contains code something like this:

FileWriter writer = new FileWriter("coordinates.txt", false);
            for (int cycle = 0; cycle < numCycles; cycle++) {
                for (Object object : objects) {
                    object.move(width, height);
                    writer.write(object.toString());
                }
            }

This is the connecting link between ChatGPT and our code.

Now we have a file coordinates.txt in which all coordinates of each object are stored in the format – Object “object number” at (“x”,”y”). Next, we parse this file and write the received data into the “x” and “y” arrays. After which, in essence, we reproduce the pre-recorded route. By the way, ChatGPT wrote a parser for the coordinates.txt file for me, I could have done it myself, but it would have taken some time.

Setting up ChatGPT.

To interact with the ChatGPT API, we need to obtain a key from OpenAI. You can get it at OpenAI API in your personal account. Next, I advise you to save this key as an environment variable (windows) and call it in code using the line:

System.getenv("OPENAI_TOKEN");

Before working with ChatGPT, it can be “configured” using a system message. My system message looks something like this:

ChatGPT does not store chat history. When you write something in the chat, you always send it all past messages. And every time it analyzes the entire story to give you the correct answer. Knowing this, I made another very important thing. After sending the system message, I write two more messages to the correspondence array. First on behalf of the user:

Second on behalf of ChatGPT with an example of the correct response to the query above:

import java.io.FileWriter;
import java.io.IOException;
import java.util.Random;

public class Main {
    public static void run() {
        int width = 1920;
        int height = 1025;
        int objectSize = 50;
        int numObjects = 7;
        int numCycles = 1000;

        Random random = new Random();
        Object[] objects = new Object[numObjects];

        for (int i = 0; i < numObjects; i++) {
            int x = random.nextInt(width - objectSize);
            int y = random.nextInt(height - objectSize);
            objects[i] = new Object(i, x, y);
        }

        try {
            FileWriter writer = new FileWriter("coordinates.txt", false);
            for (int cycle = 0; cycle < numCycles; cycle++) {
                for (Object object : objects) {
                    object.move(width, height);

                    writer.write(object.toString());
                }
            }
            writer.close();
        } catch (IOException e) {
            System.out.println("Ошибка записи в файл.");
        }
    }

    public static void main(String[] args) {
        run();
    }
}

class Object {
    private int number;
    private int x;
    private int y;
    private int direction;

    public Object(int number, int x, int y) {
        this.number = number;
        this.x = x;
        this.y = y;
        this.direction = 1;
    }

    public void move(int maxX, int maxY) {
        if (x + direction * 5 >= maxX - 50 || x + direction * 5 <= 0) {
            direction *= -1;
        }
        x += direction * 5;
    }

    @Override
    public String toString() {
        return "Object " + number + " at (" + x + ", " + y + ")\n";
    }
}

Thus, before each request to ChatGPT, we will already have the “correct” correspondence history, after analyzing which ChatGPT will give the “correct” answer.

ChatGPT can call your methods (functions).

Until I started understanding the library for interacting with the ChatGPT API, I didn’t even know about such functionality. This is quite an interesting feature. It was used to change the color of text in chat. I also wanted to use this function to accept code, but there were problems with this, which I will describe a little further.

Here is the code for getting the color to change the text in the chat:

    public static class СhangeСolorText {
        @JsonPropertyDescription("Измени цвет текста для чата,число r, значение должно быть от 0 до 0.999")
        public float r;
        @JsonPropertyDescription("Измени цвет текста для чата,число g, значение должно быть от 0 до 0.999")
        public float g;
        @JsonPropertyDescription("Измени цвет текста для чата, число b, значение должно быть от 0 до 0.999")
        public float b;
    }

    public static class СhangeСolorTextResponse {
        public float r;
        public float g;
        public float b;

        public СhangeСolorTextResponse(float r, float g, float b, MyGdx mygdx) {
            this.r=r;
            this.g=g;
            this.b=b;
            mygdx.chat.changeTextColor(r,g,b);
        }
    }

FunctionExecutor functionExecutor = new FunctionExecutor(Collections.singletonList(ChatFunction.builder()
                .name("get_color")
                .description("Измени цвет текста для чата, мне требуется ответ из трех чеслел r g b, значение каждой переменной " +
                        "должно варьироваться от 0 до 0.999")
                .executor(СhangeСolorText.class, w -> new СhangeСolorTextResponse(w.r,w.g, w.b,mygdx))
                .build()));

ChatFunctionCall functionCall = responseMessage.getFunctionCall();
        if (functionCall != null) {
            System.out.println("Trying to execute " + functionCall.getName() + "...");
            functionExecutor.execute(functionCall);
        }

Initially, ChatGPT sent RGB variable values ​​from 0 to 256. I thought that I would have to write a method to convert the values, since LibGDX uses the range from 0 to 1.0. But I just asked ChatGPT to use the correct range, and it understood everything.

I had some difficulties calling the method to get the java code. A method was written in the parameters to which the code should come from ChatGPT. When a method call is required, ChatGPT sends a response in a JSON wrapper. In my case, the answer was java code. Java code wrapped in a json wrapper… When parsing the response, in 50% of cases everything was fine, in the remaining 50% I received the following errors:

This happened because there was an extra slash somewhere. I struggled with this problem for two days but still couldn't fix it. I simply gave up and removed the functionality associated with calling methods for java code, and began to cut the code out of regular messages.

This is done quite simply. Sometimes ChatGPT sends the code without any comments. In this case, the message begins with importing the required classes. Specifically, with the keyword “import” and if it comes first, then this is java code.

 String simport=sCode.substring(0,6);
 if(simport.equals("import"))code=sCode;

And sometimes ChatGPT gives some comments to the code (even though I asked it not to do this). In this case, the code is enclosed in quotes.

int startPosition=sCode.indexOf("```java")+7;
int endPosition=sCode.lastIndexOf("```");
code=sCode.substring(startPosition,endPosition);

If none of the above is found, then this is a normal message.

About ChatGPT and games.

In fact, coming up with a game concept where AI would fit organically is not an easy task. And I don’t even have specific ideas about this. If anyone has interesting thoughts on this matter, I will be glad to read them in the comments. The next step I want to take is generating tilemaps on demand. I think there won't be any particular difficulties with this. Also, ChatGPT could generate quests, not text for quests, but create the very structure of the tasks. This is completely doable.

There is a joke that in an ideal game development engine there should be only one button – “Make a cool game”. ChatGPT has brought us seriously closer to this. I am sure that soon a game engine will appear in which it will be possible to make a game simply by describing everything in ordinary sentences. Perhaps I will do this somehow, the only problem that can prevent this is the limitation in the number of tokens.

Some lyrics

AI is a very promising technology that will penetrate deeper and deeper into our lives over time. The creation of AGI can be compared to the invention of electricity or the Internet. With the development of AI, the line between human languages ​​and programming languages ​​will become blurred. The latter will be used only by highly specialized specialists or will be completely forgotten – it’s a matter of time. My program is a small proof of this, we can program the movement of objects without using code.

A few final words.

I just started to understand ChatGPT, so I think some things could have been done much better, the project code is also not ideal. But I hope my article was useful. Perhaps it will give someone an interesting idea. Link to the project – github.