neural interface and talking inhabitants

Would you like to try yourself as a mage, whose spell power depends on concentration in the real world? Freely talk to all NPCs in natural language and get intelligible answers? Or maybe just play Skyrim VR with improved textures, interfaces and physics?

This article is a tutorial that covers the following topics:

• What you need to play Skyrim VR comfortably.

• How to install FUS assembly and debug any problems that arise.

• How to use neural interface in Skyrim VR, why and what it is.

• How to set up communication with NPCs so that LLM models speak for them.

The article is quite large. If you just need instructions, go down to the section “How to install Skyrim VR with mods”. Difficulties arise – the article has a small list of popular problems and solutions.

About the pros and cons of Skyrim VR

Before deciding whether or not to bother with setting up mods in Skyrim VR, it's a good idea to decide whether it's worth it at all.

Pros of Skyrim VR

• Without a doubt, the most content-rich AAA open-world VR game.

• Mods allow you to do anything you want with the game: enhance magic with concentration from the real world, talk to NPCs with your voice, create magic with gestures, etc.

• The beauty is incredible, if you bother with mods, settings and hardware.

Cons of Skyrim VR

• The game requires some familiarity with mods and quality optimization, which isn't exactly trivial.

• I measured the calorie burn from playing Skyrim VR – it turned out to be on average 2.5 times less than in Assassin's Creed: Nexus and Blade & Sorcery.

• The community still hasn't managed to bring the physics up to the level of Blade & Sorcery.

Why install mods

Why not just play “vanilla”? I tried, it's unbearable:

• Bad interface. You constantly have to go into the menu to change your weapon to a spell or drink an elixir. There is no way to draw your sword from its sheath with a natural hand movement.

• You can't get close to anything. The enemy hits from two meters away, without even touching you.

• Running up the stairs feels like your head is on a jackhammer.

• The graphics are pixelated and the game freezes, making for a nauseating experience even if you're trained to play VR.

• You can't pick up objects with your hands, you have to hold the button and they hang in the air.

And much, much more. There are no problems with mods of all the above.

Why FUS-modlist

This article will be a tutorial based on FUS. The assembly is considered one of, if not the most stable. It adds a minimum number of mods so that you can play comfortably, and on top of this you can put something of your own, practically without bothering with compatibility issues.

For starters, I would recommend just trying Skyrim VR with FUS – play a little, have fun, and only then consider other mods, such as mods for adding a neural interface and natural speech.

What is a neural interface and how does it work?

Let's start with the fact that all of this is needed so that in the game you can boost magic and increase the speed of mana regeneration when your mind is focused, and lose mana and even take damage when trying to use a spell with unfocused attention.

Neurointerface (I think) muse) reads EEG signals and pulse, and mod (real virtual magic) translates the received information into concentration. How he does it, you can see in class ixrdashboard.py software that we will use in conjunction with the mod.

Spell Power:

self.power_metrics = [np.float32(self.engagement + (1 - head_movement) * self.head_impact)]

Here we see that power is engagement, and head movement takes away power. If we discard all the calibrations and noise corrections, then the essence of engagement is in this formula:

engagement_idx += (beta / (theta + alpha)) / gamma 
# divided by gamma to reduce power also during strong muscle activity

Beta, theta, alpha, gamma — types of brain waves, more details there is an article on the muse website.

Gamma really does grow with physical activity – I checked.

There was no analysis of why the formula works, but I decided to trust the author of the mod – a neurobiologist from the Brandenburg Institute of Technology.

Myths about neural interface

The neural interface does not recognize signals well.

No, it works like a clock. There may be problems for the first 3-5 minutes, but then everything will stabilize. About the accuracy of the definition see research on the muse website.

The neural interface cannot recognize concentration well

I didn't believe it myself until I tried it. For example, if you just close your eyes, all the graphs will go down. And if you try to solve some puzzle, Gamma will jump up.

There is a problem with Bluetooth – the signal from the computer to the neural interface is poorly transmitted

Right here (youtube) the author of the mod says that there are similar problems. I have muse 2, Bluetooth 2.0 on the PC, which is in another room behind the wall. According to the signal, everything is fine.

Headache and/or motion sickness when playing Skyrim

That's true, but the problem is solvable. As for the graphics, see the “Problems and Solutions” section, there's a lot to dig into. As for the fact that muse 2 is pressing under the helmet, you'll have to look for solutions: change the helmet holder (for example, this one for Quest 3) or buy a more expensive and fragile muse s.

Why do you need a LLM-based chatterbox?

With mod mantella you can talk to any NPC in the game, and the NPC knows the context – everything that it is supposed to know. You can't initiate a quest by voice (I haven't found a mod for this), although NPCs are happy to tell you about quests. There is a mischievous opportunity to make the NPC attack you, then the guards won't mind if you use force (this option is enabled in the mod settings).

I don't use natural speech in the game very often, but there are options for using it. I'll give you 3 examples when this mod “saved” me:

1. When joining the assassins' guild, you had to choose which of the three to kill. I have a hard time making decisions like that even in the game, and the available dialogue options were limited. I had to talk to everyone.

2. The quest to expose a vampire at court involves pickpocketing and stealth, skills I hadn't developed at all in my Orc Warrior playthrough. Using natural speech, I got the vampire to attack me, and the northern traditions of Skryrim allow self-defense.

3. I didn't want to start quests for the thieves' guild, but I needed to find old man Esbern. I talked to several residents, and they told me where to look for him and explained the way.

Some examples of conversations in different contexts:

How much does all this cost?

How long it will take you depends on many factors, but the prices I give are approximate.

Budget option with neural interface and LLM

We buy everything from hands, for example, on Avito.

• Quest 2 ≈ 20k rubles.

• PC with minimum specifications (see below “What is needed from the material world”) ≈ 40k rubles from hands.

• We'll buy Skyrim VR from Steam when there are discounts, or we'll consider the possibility of “borrowing”, more on that below.

• Neuro hoop Muse 2 ≈ 23k rubles from hands.

We don't pay for mod subscriptions, we download everything manually. If you want to save money on a helmet, losing the reliability of the product, turn to Chinese helmets on AliExpress – at your own risk.

Mods and programs for speech recognition and generation are downloaded for free. If PC resources allow, then instead of OpenAI API you can install LLM locally or use free analogs (OpenRouter: Toppy model). You will have to tinker more, but we will save money.

Total: less than 90k rubles.

Premium version with neural interface and LLM

• MetaQuest 3, Valve Index, HTC VIVE Pro, VIVE XR Elite — from 55k to 220k.

• Helmet accessories for convenience (for example, Bobovr mount for MetaQuest 3 and additional battery) ≈ 7k rubles.

• PC from 150k.

• Muse s second generation neuro hoop – 65-70k (I saw such prices for new hoops on Avito).

• OpenAI 4o — 1-3k rubles per month with moderate chatting and playing every day.

• Text-to-speach, paid version XTTS ≈ 1k per month.

• Wi-Fi 6e — from 10k.

Total: more than 220 thousand rubles.

What you need to play with all mods and settings

Below, optional items will be marked with an asterisk.

From the material world

1. Any VR headset compatible with Windows PC: MetaQuest 2, 3, pro; Pico 4; Valve Index, HTC VIVE, etc.

2. PC. Minimum according to Steam: Intel Core i5-6600K or AMD Ryzen 5, 8gb ram, Nvidia GeForce GTX 970 / AMD RX 480.

3. Neurointerface. Muse 1, 2, s, s second generation.

4. Bluetooth adapter on PC, if there is no built-in one, for connection with the neural interface.

5. * Router, preferably Wi-Fi 6 or 6e – if there are many networks nearby and you want to play wirelessly.

6. * Lenses for a VR headset, if you realize that playing with glasses and a neural interface is inconvenient and if the headset supports it. For example, muse 1 and 2 are practically incompatible with glasses, but you can buy lenses for MetaQeust 3.

What software do you need?

1. Skyrim VR.

2. Steam and SteamVR.

3. Program for connecting the helmet to the PC. Meta quest link in the case of Meta Quest 2, 3. Virtual Desktop is suitable for any helmets.

4. FUS — a minimal set of mods for beginners.

5. IXR Suite to transmit a signal from the neural interface to the PC.

6. Additional mods and software for communication with LLM, there will be a section below about this.

Which helmet should I take?

For beginners, I would recommend MetaQeust 3, as there will be options to play both standalone games with acceptable graphics and PCVR. Difficulties will still arise:

• To receive the update you will need to share Wi-Fi with a VPN.

• install Meta Quest Link on your PC, connect it, select performance parameters to match the characteristics of your PC, router, and helmet.

Which neural interface to choose and where to buy it?

I bought mine on Avito, you can order it from Aliexpress. I decided to try it with Muse 2, because the mod developer wrote that Muse S is much less durable and you will have to constantly buy additional sensors. Here is the original message.

Muse S is much more pleasant and has better signal through larger touch points. However, it might not fit under all VR headsets because it is larger on the forehead, and after extended use the fabric sensors can break. New sensors cost around 60 bucks I think. My old Muse 1 and Muse 2 are still going strong, but I needed to replace the Muse's fabric strap of two different ones already.

That is, to choose from:

1. Muse 2: Sturdy and uncomfortable, can be tight under a helmet, and is virtually incompatible with glasses.

2. Muse S: ​​holds the battery twice as long (10 hours), sits more comfortably, does not conflict with glasses, but is fragile, and where to find breakable sensors – I don’t know.

How to Install Skyrim VR with Mods

Be prepared that everything listed below will take about an hour if you are doing it for the first time.

First we buy game on steam.

Considering that you can’t pay with a Russian card, and not everyone can get a foreign one, it’s possible (purely theoretically) to borrow the game.

FUS – the most popular mod preset for beginners, very stable and great for starting.

The project's Readme contains link to tutorial 2023, still relevant. If you're lucky, you'll only have to click “next” and “ok” a few times (and buy a subscription to nexusmods.comto download faster, or press “Slow Download“ 300 times) and everything will stop. You will launch ModOrganizerwhich will launch the game, and that's it.

For the Wabbajack problem, you can start with FAQ And Troubleshooting on the FUS page, and also look at the list of problems with free installation from the spoiler above.

How to install a “chatterbox” based on LLM models

I chose Mantellawalked along instructionsand everything started and worked the first time without any problems, except that it took about 2 hours to read the instructions and download everything necessary.

I will duplicate here a short instruction in Russian for FUS users

Speech generation

We put xVASynth from Steam, the program is free.

Inside the program, find Skyrim, refresh the list, click “Download all” if you have purchased a subscription to nexusmods. If not, then either download torrentor voice models from nexusmods (in files you can download everything that starts with “Voice model”). You will need to unzip and put the models in the folder <путь к xVASynth>/resources/app/models/skyrim/. As a result, the contents of the folder should be as follows:

Install the plugin .lip in xVASynth: you just need to move the downloaded folder lip_fuz V <путь к xVASynth>/resources/app/plugins. And download for .lib additionally FaceFXWrapper:

Optional: to play in English, set DeepMoji Pluginwhich will make speech less robotic.

Mods that are not in FUS and are needed

List of required mods:

• Mantella – Bring NPCs to Life with AI, file: Mantella Spell.

• No NPC Greetings (Special Edition) ESL.

• World Encounter Hostility Fix, file: Performance version.

Open ModOrganizer, click on the top left file -> visit nexuswe find ourselves in nexusmods. There we look for the necessary mod, go to files -> mode manager download. We agree – let's go back to ModOrganizer. On the right in the Downloads section you need to find the downloaded mods and install them (right-click -> install).

Next, you need to activate the checkboxes for the installed mods in the left panel. The mods will appear at the very bottom.

A program for managing all of the above

Download Mantella Softwareput the folder somewhere near the FUS mods. Before launching, it is important to correct the settings in config.ini. It already has everything documented in detail, the main thing is to replace the variables specified below. I give you how it is for me (with paths to my game folders, FUS and xVASynth), only the important lines (example 1):

[Game]
game = SkyrimVR
; skyrimvr_folder = E:\vr\mods_new\FUS\overwrite\root
skyrimvr_folder = E:\SteamLibrary\steamapps\common\SkyrimVR
skyrimvr_mod_folder = E:\vr\mods_new\FUS\mods\Mantella - Bring NPCs to Life with AI
xvasynth_folder = E:\SteamLibrary\steamapps\common\xVASynth
language = en
microphone_enabled = 1
audio_threshold = auto
model = gpt-4o
; model = gpt-3.5-turbo

; — comment symbol (i.e. the program ignores this line).

I have a VPN set up directly on my router. If you don't want to bother with setting up a VPN to access OpenAI, try other options. If you have VPN and OpenAI, then paste the key into the text file: GPT_SECRET_KEY.

Note that I have the second line commented out. When you first run it, you need to change it from the third line. It will be like this:

skyrimvr_folder = E:\vr\mods_new\FUS\overwrite\root
; skyrimvr_folder = E:\SteamLibrary\steamapps\common\SkyrimVR

Run the program (Mantella), make sure there are no errors (warnings for the first two launches are acceptable), and in the game use the “Mantella” spell to start a conversation (button B on the quest). After that, exit the game, return the comments, as in the first example, restart Mantella, enter the game, and everything should work.

Problems and solutions

Bad picture

The picture may be bad for the following reasons:

1. The PC doesn't measure up to the specifications.

2. There are problems with the connection between the PC and the VR headset.

3. Bad settings of the game, mods, programs for connecting to the helmet.

Weak video card

Upscaler mods come to the rescue here. FUS 5+ versions have the following (you only need to use one!):

• DLAA;

• DLSS;

• VR Performance Toolkit;

• OpenXR Toolkit.

The first three are enabled by checking the box in ModOrganizer in the section “1.2 – SkyrimVR Upscaler options”. To install OpenXR, you will also need to check the box for the mod OpenComposite and install OpenXR toolkit. By the way, OpenComposite дает существенный (≈10%) прирост к производительности системы на мета квестах.

You can also take a preset with low settings from FUS\tools\Ini options\01 - Low Quality and insert them into FUS\profiles\<имя вашего профиля>For example FUS\profiles\FUS RO DAH (Basic + Appearance + Gameplay)a copy of which I would recommend using to start.

You can also turn off the tree sway animation (you probably won't even notice it unless you specifically pay attention). FUS\profiles<имя вашего профиля> V Skyrimprefs.ini нужно будет выставить 0:

bEnableTreeAnimations=0

And reduce the distance to “fade” the grass:

[Grass]
fGrassStartFadeDistance=2500.0000
fGrassMaxStartFadeDistance=5000.0000
fGrassMinStartFadeDistance=0.0000

Also disable all mods containing ENB root. And check if dynamic resolution is disabled in the game settings (it should be disabled).

Network bandwidth issues

If you have a powerful machine and play via Wi-Fi, then the network is probably the weakest link. There are several things you can do here. First, set up your router:

1. Use 5GHz band (not 2.4);

2. Use 5GHz only for the helmet, and 2.4 for all other devices in the house;

3. In the router settings, in the Wi-Fi / wireless -> Channel bandwidth section, set a static value, preferably 80Mhz. Most likely, 20/40/80/160 is selected by default.

4. In the quest-link application (if you use it) on the helmet, leave the automatic Mbps detection, but set the default to the maximum (200).

If you have a router with 6e, I'm sure you know what to do. If not, consider buying one, especially if you live in an apartment building where Wi-Fi networks interfere with each other.

Pixel art

Try increasing the resolution. This can be done in different ways:

1. MetaQuest App -> Devices -> -> graphics settings.

2. SteamVR -> ☰ -> Settings -> Video -> Resolution Per Eye.

3. In file FUS\mods\OpenComposite...\opencomposite.ini set the supersamplingRatio value.

4. In OpenXR tool, System -> Display Resolution Per Eye (I use this option).

From what I googled, 1, 2 and 3 are multiplied. In practice, my Skyrim VR plays at a resolution set to 1, and 4 overrides everything, but is multiplied by 3. How the resulting resolution numbers (per eye) work, I still don't understand, because 3264×3520 in SteamVR looks worse and more pixelated than 2414×2600 in OpenXR.

I tried playing with these settings myself, having a 4080Ti card, 5Ghz wifi, MetaQuest3:

1. SteamVR 340% at 80Ghz (80 FPS max) in MetaQuest.

2. SteamVR 100%, MetaQeust 72Ghz, 2414×2600 in OpenXR with CAS method.

The first configuration allowed me to keep the FPS higher (almost always 80) with moderate graphics, but in the second version the picture is much better and the jumping FPS between 40-72 does not bother me much.

Here you can, if the video card allows, use a preset 03 - High Quality (more details in the section above – “Weak video card”).

Low FPS

FPS conflicts with the quality (resolution) of the picture one way or another. You have to find a balance. The Meta Quest Link app has a default limit of 72 Hz. Many say that they don't see the difference between 72 and 90, but they do see the difference in resolution.

Accordingly, here are some tips that will help you to make the picture worse:

• You can use the advice from the “Pixel graphics” section – take 01 - Low Quality preset.

• Set supersampling to 0.9 or even 0.7 in the file FUS\mods\OpenComposite...\opencomposite.iniand also check if the resolution has been increased in other places (see the “Pixel image” section).

The tips in the section “Network bandwidth issues” will also help.

You can tweak the FPS in SteamVR: check the box in the settings next to “show performance graph”. The program allows you to output the graph to the helmet (in the developer option). Another option is through OpenXR, if you use it.

My head and eyes hurt, I feel sick

Skyrim VR is the first game that forced me to debug FPS and dig into mods because of the strain on my eyes. It felt like I was playing in someone else's glasses.

There is no universal solution. We have already discussed the bad picture and FPS above, you can look in several other directions.

“Evil” mods

Mods from the FUS list that may have a negative impact:

• ENB mods (ENBSeries), if the video card is weak, then disable everything;

• VRIK Player Avatar (customizable);

• ReShade VrVision (removing in favor of SharepEye).

VRIK is problematic in that the height in the game and the size of the world can be set incorrectly, causing the brain to receive “incorrect” information when moving. Here you need to synchronize VRIK and change the options in mod settings — VRIK.

ReShade VrVision was lagging and shaking for me.

“Good” mods

“ReShade The Sharper Eye” in FUS v4.1.19 is disabled by default. I found the picture to be much better with it. In v5.0.0-beta18, which I'm currently using, “Sharper Eye” is enabled by default. A good alternative is “ReShade Glamur”.

It was already mentioned above, but I will repeat: I highly recommend setting up OpenXR, which is available in 5+ versions of FUS.

The power of habit

If you keep playing, your brain will adapt to the game. Just give yourself time.

Still having performance issues?

Eat nexus page And performance guide on the FUS wiki. You can also ask questions on reddit at SkyrimVR subreddit.

About gameplay settings

Below I will describe the build that I could recommend and with which I started.

I put 1kg weights on my hands. This adds realism to the game, especially with this build:

• heavy armor;

• one-handed weapon with a shield;

• from magic, only a spell to restore health (if you use a neuro hoop).

Why only recovery magic? At first it will be very difficult to concentrate. But after the fight – or when you run away to a safe distance – you can try to relax, close your eyes or focus on one point, take a deep breath – the power of concentration will soar, and at this peak it is already “profitable” to heal.

With weights on your hands, you won't be able to swing your weapon very quickly, so the highest difficulty level for the described build won't work (at the first 20 levels), but you can't leave the default “adept” either: in it, mana and health parameters are restored too quickly without any concentration.

If you don't have any hand weights, you can try the thief (assassin) build. The main thing here is to remember to turn off the joystick squat – for realism and physical training – in the settings, and also turn on the “Auto Sneak and Jump VR” mod. After the first game session, you will most likely have to wait out the muscle pain for 2-3 days, as after a full leg workout. That's why I play two characters in parallel, to give different muscles time to recover.

As for the controls, you can leave the default preset (Kvitekvist mod), play around, and then see what you missed, wiki.

From the mod settings in the game, I recommend going to Spell Wheel and choose what you would like to see available in the quick menu for each game (“types”).

VRIK allows you to set your height, arm length, world size. This mod also allows you to put your sword in a sheath, daggers on your forearms, a bow, shield, and two-handed weapon on your back – all with the grip button. If you hold down the grip, you can move the sheath to another place.

Finally

After tweaking the mods, getting the game smooth and looking great, I got a level of enjoyment and excitement from the game comparable to when I first launched Skyrim in 2011. This prompted me to write an article with answers to questions that may arise when “colliding” with Skyrim VR.

Is it worth getting a neuro-headband, buying a VPN and an OpenAI subscription right away? I would advise starting with the game itself, making sure it suits you, and then buying and customizing the rest when and if you realize that you want something else. After all, playing is especially interesting while you can learn and explore new things.