CI - Prog.World

Having tried different Source Controls in conjunction with UE (Gitlab, SVN, Perforce) at the moment, for an indie project I chose the most convenient solution, although not the easiest to install:

Gitlab – To use it on your server, you need to have a Linux machine (I use mint), some time and patience to install the server and that’s it, we are happy owners of a free gitlab server. Next, we set up a connection via SourceTree and use;

Another charm of Gitlab, I would call the built-in tool for CI / CD, which is very easy to configure and allows one click of a button: Build a light, pack a game, upload it to steam, send a message to discord, etc., you just need a server with the UE installed. I’ll tell you about setting up CI under Windows.

Let’s skip the moment when to install the Gitlab / Source tree, create a new project and other items in it. Let’s assume that all this is already there and you need to configure the runner along with the CI.

Installing Gitlab-Runner

Create a folder for it, for example: C: Gitlab-Runner, download the binary: 64-bit or 32-bit, drop it into the folder created earlier and change the name to: gitlab-runner.exe (more details). The main thing is to make sure that the folder has write access and you can run files from it. Run cmd to register and install Gitlab-runner.

First of all, we install and start the runner, below is an example, taking into account the cases when a username and password are needed to start a Windows service.

cd C:Gitlab-Runner
.gitlab-runner.exe install --user ENTER-YOUR-USERNAME --password ENTER-YOUR-PASSWORD
.gitlab-runner.exe start

Registering a new Gitlab-runner.

We go through a web browser to the Gitlab server, go to our new project. Further Settings-> CI / CD-> Runners and expand the tab, which will indicate the settings for the new server.

We open a command line on the server with Gitlab-runner installed and register a new runner for registering and fill in the requirements point by point, specifying the server URL, a unique token (it can be regenerated if necessary), name and tags as server identifiers. An important detail is with which executor we will process the yml file (I use shell) more details…

cd c:Gitlab-runner
.Gitlab-runner.exe register

That’s it, the new Runner is registered, by refreshing the page we can see all the available servers. An important point for further work is to go to the server settings and install, without this checkbox, CI hangs in pending mode:

I will also add that in “General pipelines” you can change the timeout (by default 2 hours) to a large value (I have 16h), after this time the CI process is disabled.

In such a very simple way, we prepared our server for work, as a build machine for CI / CD.

Setting up yml and first launch

For CI / CD, you need to create a file (.gitlab-ci.yml) in the root of the project and upload it to the repository, thereby giving instructions for work. Gitlab will check the given script and if something goes wrong it will issue a yaml invalid error in the CI / CD. In order not to push dozens of broken files, right on the site in the CI / CD-> Pipelines tab there is a tool: “CI Lint”, with which you can check the script for correctness.

Since we previously indicated that we are using the Shell executor, the yml file is created according to the powershell principle (for other executors it may differ). Below I gave an example, assembling a client in two forms (dev / spike) and a server, as well as uploading solutions to Steam. As a bonus, sending a message about a successful build to the discord.

Sample script for .gitlab-ci.yml

# last update 10.04.2019 12:58PM
# BUILD_CONFIG - Shipping / Development / DebugGame / Debug

variables:
    GIT_STRATEGY: none        # we disable fetch, clone or checkout for every job.
    GIT_CHECKOUT: "false"     # as we only want to checkout and fetch in the preperation stage.
    CI_DEBUG_TRACE: "false" 
    CLIENT_BUILD_CONFIG: Shipping
    DEV_CLIENT_BUILD_CONFIG: Development
    CLIENT_PLATFORM: Win64
    SERVER_BUILD_CONFIG: Development
    SERVER_PLATFORM: Win64
    BUILD_DIR: D:SoulhaimBuilds
    CLIENT_DIR: D:SoulhaimBuildsclientez-client    
    CLIENT_STEAM_PATH: $CLIENT_DIRWindowsNoEditor*
    CLIENT_ARCH_NAME: $CLIENT_BUILD_CONFIG-$CI_COMMIT_SHA
    DEV_CLIENT_DIR: $BUILD_DIRdev_clientez-client
    DEV_CLIENT_STEAM_PATH: $BUILD_DIRdev_clientez-clientWindowsNoEditor*
    DEV_CLIENT_ARCH_NAME: $DEV_CLIENT_BUILD_CONFIG-$CI_COMMIT_SHA
    SERVER_DIR: $BUILD_DIRserverez-server
    SERVER_ARCH_NAME: $SERVER_BUILD_CONFIG-$CI_COMMIT_SHA
    LOCAL_ARCH_PATH: D:BuildsArchives
    STEAM_PATH: D:steamworks_sdk_148asdktoolsContentBuilder
    UE4_ROOT: D:UnrealEngine
    
stages:
    - preperations
    - build_path
    - build_reflection
    - build_light
    - build_client    
    - load_to_steam
    - upload_to_steam
    - pack_client
    - upload_client
    - clean_steam_client
    - dev_build_client
    - dev_load_to_steam
    - dev_upload_to_steam
    - dev_pack_client
    - dev_upload_client    
    - dev_clean_client
    - build_server
    - pack_server
    - upload_server   
    - clean_server
    - clean
                 
job_preperations:
    stage: preperations
    variables:
        GIT_STRATEGY: fetch
        GIT_CHECKOUT: "true"  
    script:
        - echo "$CLIENT_STEAM_PATH"
        - echo "$STEAM_PATHcontentSoulhaim"
        - (Remove-Item -Path $BUILD_DIR -Force -Recurse -ErrorAction Ignore)
        - echo "Hello"
        - echo "$UE4_ROOT"    
        - echo "$UE4_ROOTEngineBuildBatchFilesRunUAT.bat"
        - echo "$CLIENT_DIR"
        - echo "$CLIENT_BUILD_CONFIG"
        - echo "$CLIENT_ARCH_NAME"
        - echo "$DEV_CLIENT_DIR"
        - echo "$DEV_CLIENT_BUILD_CONFIG"
        - echo "$DEV_CLIENT_ARCH_NAME"
        - echo "$SERVER_DIR"
        - echo "$SERVER_ARCH_NAME"
    only:
        - web
        
job_build_path:
    stage: build_path
    script:
        - Start-Process -NoNewWindow -Wait "D:Soulhaim_build_path.bat"
    only:
        - web     

job_build_reflection:
    stage: build_reflection
    script:
        - Start-Process -NoNewWindow -Wait "D:Soulhaim_build_Reflection.bat"
    only:
        - web  
        
job_build_light:
    stage: build_light
    script:
        - Start-Process -NoNewWindow -Wait "D:Soulhaim_build_light.bat" 
    only:
        - web
        
job_build_client:
    stage: build_client
    script:
    #-ForceDebugInfo -CrashReporter - for debug files
    - Start-Process "$UE4_ROOTEngineBuildBatchFilesRunUAT.bat" -NoNewWindow -Wait -ArgumentList "BuildCookRun", "-project=$CI_PROJECT_DIRSoulhaim.uproject", "-nop4", "-build", "-cook", "-compressed", "-stage", "-ForceDebugInfo", "-allowcommandletrendering", "-platform=$CLIENT_PLATFORM", "-clientconfig=$CLIENT_BUILD_CONFIG", "-pak", "-archive", "-archivedirectory=$CLIENT_DIR", "-utf8output"
    only:
        - web

job_Load_To_Steam:
    stage: load_to_steam
    script:
        cp -r "$CLIENT_STEAM_PATH" "$STEAM_PATHcontentSoulhaim"
    only:
        - web

job_Upload_To_Steam:
    stage: upload_to_steam
    script:
        - Start-Process "D:steamworks_sdk_148asdktoolsContentBuilderrun_build.bat" -NoNewWindow -Wait
    only:
        - web

job1_pack_client:
    stage: pack_client
    script:
        - Start-Process -NoNewWindow -Wait "C:Program Files7-Zip7z.exe" -ArgumentList "a", "$CLIENT_DIR.zip", "$CLIENT_DIR"
    only:
        - web
        
job2_copy_to_local_client:
    stage: upload_client
    script:
        - cp "$CLIENT_DIR.zip" "$LOCAL_ARCH_PATHclient-$CLIENT_ARCH_NAME.zip"
    only:
        - web  

job3_clean_Steam_Folder:
    stage: clean_steam_client
    script:
        (Remove-Item -Path "$STEAM_PATHcontentSoulhaim*" -Force -Recurse -ErrorAction Ignore)   
    only:
        - web

dev_job_build_client:
    stage: dev_build_client
    script:
    #-ForceDebugInfo -CrashReporter - for debug files
        - Start-Process "$UE4_ROOTEngineBuildBatchFilesRunUAT.bat" -NoNewWindow -Wait -ArgumentList "BuildCookRun", "-project=$CI_PROJECT_DIRSoulhaim.uproject", "-nop4", "-build", "-cook", "-compressed", "-stage", "-ForceDebugInfo", "-CrashReporter", "-debug", "-allowcommandletrendering", "-platform=$CLIENT_PLATFORM", "-clientconfig=$DEV_CLIENT_BUILD_CONFIG", "-pak", "-archive", "-archivedirectory=$DEV_CLIENT_DIR", "-utf8output"
    only:
        - web

job_dev_Load_To_Steam:
    stage: dev_load_to_steam
    script:
        cp -r "$DEV_CLIENT_STEAM_PATH" "$STEAM_PATHcontentSoulhaim"
    only:
        - web

job_dev_Upload_To_Steam:
    stage: dev_upload_to_steam
    script:
        - Start-Process "D:steamworks_sdk_148asdktoolsContentBuilderrun_build.bat" -NoNewWindow -Wait
    only:
        - web

job1_dev_pack_client:
    stage: dev_pack_client
    script:
        - Start-Process -NoNewWindow -Wait "C:Program Files7-Zip7z.exe" -ArgumentList "a", "$DEV_CLIENT_DIR.zip", "$DEV_CLIENT_DIR"
    only:
        - web
        
job2_dev_copy_to_local_client:
    stage: dev_upload_client
    script:
        - cp "$DEV_CLIENT_DIR.zip" "$LOCAL_ARCH_PATHclient-$DEV_CLIENT_ARCH_NAME.zip"
    only:
        - web  

job_clean_client:       
    stage: dev_clean_client
    script:
       (Remove-Item -Path $BUILD_DIR -Force -Recurse -ErrorAction Ignore)
    only:
        - web    
        
job_build_server:
    stage: build_server
    script:
        - Start-Process "$UE4_ROOTEngineBuildBatchFilesRunUAT.bat" -NoNewWindow -Wait -ArgumentList "BuildCookRun", "-nocompileeditor", "-project=$CI_PROJECT_DIRSoulhaim.uproject", "-nop4", "-build", "-cook", "-compressed", "-stage", "-noclient", "-server", "-serverplatform=$SERVER_PLATFORM", "-serverconfig=$SERVER_BUILD_CONFIG", "-pak", "-archive", "-archivedirectory=$SERVER_DIR", "-utf8output"
    only:
        - web               
        
job1_pack_server:
    stage: pack_server
    script:
        - Start-Process -NoNewWindow -Wait "C:Program Files7-Zip7z.exe" -ArgumentList "a", "$SERVER_DIR.zip", "$SERVER_DIR"
    only:
        - web
        
job2_copy_to_local_server:
    stage: upload_server
    script:
        - cp "$SERVER_DIR.zip" "$LOCAL_ARCH_PATHserver-$SERVER_ARCH_NAME.zip"
    only:
        - web
     
job3_unpack_local_server:
    stage: upload_server
    script:
        - Start-Process -NoNewWindow -Wait "C:Program Files7-Zip7z.exe" -ArgumentList "x", "$LOCAL_ARCH_PATHserver-$SERVER_ARCH_NAME.zip", "-oD:"
    only:
        - web
        
job4_send_notify_server_to_slack:
    stage: upload_server
    script:
        - Start-Process "D:Soulhaim_Message_Discord.bat" "New Awesome build is ready" -NoNewWindow -Wait
    only:
        - web
    
job_clean_server:
    stage: clean_server
    script:
        (Remove-Item -Path $BUILD_DIR -Force -Recurse -ErrorAction Ignore)        
    only:
        - web
        
job_clean:
    stage: clean
    script:
        (Remove-Item -Path "$STEAM_PATHcontentSoulhaim*" -Force -Recurse -ErrorAction Ignore)   
    only:
        - web

At the first stage (preperation), we clear the folder from the old build, and also take all the latest changes from the repository. It is advisable to make only this stage as the first launch, since the next steps are related to the project and we need a path to it.

The next steps are to calculate navmesh light reflection, the scripts of which are located in regular * .bat files. An example for rendering light on two maps: Through the “+” you can specify a list of maps.

Separately, it is necessary to add that if you are working with a binary version, then it is advisable to calculate navmesh light reflection in the same way on the binary version, otherwise, on the version from the source, the engine will require you to rebuild the solution, and it will get stuck in the process (perhaps there is a solution , but unfortunately I didn’t find it).

"C:Program FilesEpic GamesUE_4.26EngineBinariesWin64UE4Editor-Cmd.exe" "D:GitLab-RunnerbuildsComtV2y9SoulhaimSoulhaimSoulhaim.uproject" -nop4 -run=resavepackages -buildlighting -quality=Preview -allowcommandletrendering -map=LobbyMap+GameMap

Next, step by step, the assembly will take place Shipping client-> Steam, Development client-> Steam, Server, archive all this and place it in a separate folder in the form of client / server-Dev / ship-hash.zip, if there are a lot of builds, then the time from time to time it will be necessary to clean the folder with archives, or simply remove these steps from CI.

An important point, since the process is automatic, under Steam you need to create a separate user with the rights to upload the build, and without any two-factor protection, otherwise the process will hang waiting for a confirmation code that will have nowhere to drive.

Launch

Launching is extremely simple, go to Ci / CD-> Pipelines and press the big green RunPipeline button, select which branch we want to build from and start the process. Then, if all is well, it remains to sit back in a comfortable chair with tea and wait until the process reaches the end (or fails). Any stage can be opened and problems can be investigated, if any.

Bonus for sending messages to discord

First, go to the discord channel settings-> integration-> Webhooks, create a new one and copy its URL, and then just paste the message and link into the bat:

curl -d "content=Build is ready: Soulhaim" -X POST https://вебхук вашего канала в дискорде

Conclusion

That’s actually the whole process of setting up CI CD in Gitlab, you can read more on the official website in the documentation, but I tried to describe the process in detail with examples. I hope the article will be useful to you, and I did not forget to indicate anything important in it.

Thank you all for your attention and may the power of knowledge be with you.