Running privateGPT locally on WSL with support for Nvidia GPUs

Private GPT is a production-ready AI project that lets you ask questions about your documents using the power of large language models (LLMs), even in offline scenarios. 100% private, no data leaves your execution environment at any time.

Running on Windows Subsystem for Linux (WSL) with GPU support can significantly improve its performance. In this tutorial, I will walk you through the step-by-step process of installing PrivateGPT on WSL using GPU acceleration.

Before we begin, make sure you have the latest version of Ubuntu WSL installed. You can choose versions like Ubuntu-22-04-3 LTS or Ubuntu-22-04-6 LTS available in the Windows Store.

Ubuntu update

sudo apt-get update sudo apt-get upgrade sudo apt-get install build-essential

It's important to update because something might explode later if you don't.

Cloning the PrivateGPT repository

git clone https://github.com/imartinez/privateGPT

Setting up the Python environment

For Python version control we will use pyenv. Follow the commands below to install it and set up your Python environment:

sudo apt-get install git gcc make openssl libssl-dev libbz2-dev libreadline-dev libsqlite3-dev zlib1g-dev libncursesw5-dev libgdbm-dev libc6-dev zlib1g-dev libsqlite3-dev tk-dev libssl-dev openssl libffi-dev curl https://pyenv.run | bash export PATH="/home/PATH"

Add the following lines to your .bashrc file:

export PYENV_ROOT="PYENV_ROOT/bin:(pyenv init -)"

Restart your terminal
source .bashrc

Installing important missing pyenv components

sudo apt-get install lzma sudo apt-get install liblzma-dev

Installing Python 3.11 and setting it as global version:

pyenv install 3.11 pyenv global 3.11 pip install pip --upgrade pyenv local 3.11

Installing Poetry

Install poetry to manage dependencies:

curl -sSL https://install.python-poetry.org | python3 - Добавьте следующую строку в ваш файл .bashrc: export PATH="/home/<ИМЯ ПОЛЬЗОВАТЕЛЯ WSL>/.local/bin: whoami)

Reload the configuration

source ~/.bashrc
poetry --version # должно отобразиться что-то без ошибок

Installing PrivateGPT dependencies

Go to the PrivateGPT directory and install the dependencies:

cd privateGPT poetry install --with ui poetry install --with local

Installing Nvidia Drivers

Visit the official Nvidia website to download and install Nvidia drivers for WSL. Select Windows > x86_64 > WSL-Ubuntu > 2.0 > deb (network)

Follow the instructions on the page.

Add the following lines to your .bashrc file:

export PATH="/usr/local/cuda-12.3/bin:LD_LIBRARY_PATH"
Check the contents of “/user/local” to make sure you have a “cuda-12.3” folder. Your version may vary.

Reload your configuration and make sure everything works as expected

source ~/.bashrc nvcc --version nvidia-smi.exe

“nvidia-smi” is not available in WSL, so just make sure the .exe version detects your hardware. Both commands should display gobbledygook, but no obvious errors.

Building and running PrivateGPT

Finally, install the CUDA libraries and Python LLAMA links:

CMAKE_ARGS='-DLLAMA_CUBLAS=on' poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python
Let the private GPT load the local LLM for you (mixtral by default):

poetry run python scripts/setup
To start PrivateGPT use the following command:

make run
This will initialize and run GPU-aware PrivateGPT in your WSL environment.

Go to 127.0.0.1:8001 in your browser

Conclusion

By following these steps, you have successfully installed PrivateGPT on GPU-enabled WSL. Enjoy advanced PrivateGPT capabilities for your natural language processing workloads.

If something goes wrong, open the window and throw away the computer. Then start again from step 1.

Original

Notes from the translator:

If it starts and opens in the browser, but when you try to get a response it crashes with an error – update the Nvidia drivers from the official website and reboot.
If poetry install --with ui crashes with an error try
poetry install --extras "ui embeddings-huggingface llms-llama-cpp vector-stores-qdrant"