A procedure that worked for getting gpt4all working

Original forum link: https://forum.qubes-os.org/t/23140
Original poster: ddevz
Created at: 2023-12-24 22:08:27
Posts count: 2
Likes count: 6

Explanation: gpt4all is a "large language model"/chat-gpt like thing that can run on your system without a network connection (no API key needed!), and can use the CPU instead of requiring a GPU.

Objective: neither the gpt4all install instructions or the gpt4all build instructions as stated on their website work for qubes. (I believe the #1 reason is that we don't have a template for the version of ubuntu that they were using.) While this procedure may not be the best way to do it, this is a procedure that actually worked in month 12 of 2023. (future versions could need other procedures)

(note: This comes from a discussion held on the "secret" part of the qubes forum, where @whoami had said:

A side note, maybe this is something for your testing environment: GPT4All

I already tried to get it into a minimal (network off) templateVM … could not fix Python dependency…

Rather then answer there were only certain people can see it, I'm posting it here for everyone to see.

So the procedure I'm showing here is a modified version of a procedure from: https://dwaves.de/2023/12/16/gnu-linux-how-to-install-gpt4all-on-debian-12-gpt4all-j-an-apache-2-licensed-assistant-style-chatbot-a-free-to-use-locally-running-privacy-aware-chatbot-no-gpu-or-internet-required-dialogu/

THE ACTUAL PROCEDURE: Start with debian-12 Add the dependancies to a debian-12 template:

qvm-run --pass-io  debian-12 'sudo apt update'
qvm-run --pass-io  debian-12 'sudo apt install python3-pyqt5 wget'
qvm-shutdown debian-12

(Note: It's fine to add the dependancies manually instead of using the CLI. I put the CLI versions so that people don't have to ask for extra details on what I did)

Create a qube that will be in charge of downloading gpt4all files, not in charge of running the gpt models:

qvm-create --template debian-12 --property netvm=sys-firewall-internet,memory=400,maxmem=4000, --label orange gpt4all-cloneable-apptemplate

(Note: I actually wrote this line after doing it manually, so not sure if the "," is the right syntax for the "--property netvm=sys-firewall-internet,memory=400,maxmem=4000" part.)

at this point, adjust your private volume size. The guy from the procedure that I'm modifying says provision 1 terabyte of space. A single download can take over 8 Gigs of space. Still 1TB seems excessive. Pick what you think. Thank goodness it's thin provisioned :)

qvm-run --pass-io gpt4all-cloneable-apptemplate 'wget https://github.com/nomic-ai/gpt4all/releases/download/v2.5.4/gpt4all-installer-linux-v2.5.4.run'
qvm-run --pass-io gpt4all-cloneable-apptemplate 'chmod +x gpt4all-installer-linux-v2.5.4.run'
qvm-run --pass-io gpt4all-cloneable-apptemplate './gpt4all-installer-linux-v2.5.4.run'

Wait till it finishes installing the executable, then:

qvm-run --pass-io gpt4all-cloneable-apptemplate 'gpt4all/bin/chat'

a download menu should pop up. select at least one "chat capable" model to download (example: mistral is chat capable and does not require a API key)

(if you accidently close the download menu, then click the hamburger button, then click "download" to get the download menu back)

close the app shut down the qube after shutdown is complete, clone gpt4all-cloneable-apptemplate to gpg4all-scratch-1

qvm-clone gpt4all-cloneable-apptemplate gpt4all-scratch-1

turn the networking off entirely on the new gpt4all-scratch-1

qvm-prefs --set gpt4all-scratch-1 netvm none

he says it needs more then 8GB of ram, so change the maxmem for gpt4all-scratch-1 to be 16 Gig or more (and the minimum memory to be at least whatever you chose devided by 10)

qvm-prefs --set gpt4all-scratch-1 memory 2000
qvm-prefs --set gpt4all-scratch-1 maxmem 20000
qvm-prefs --set gpt4all-scratch-1 label purple

finally, start gpg4all-scratch-1 and run the application

qvm-run --pass-io gpt4all-scratch-1 'gpt4all/bin/chat'

Now, finally, in the offline version you can finally begin harassing it with questions about how it's going to take over the world, and asking it questions that have no answer.

To Do Updates: If you want to download a different model: * delete gpt4all-scratch-1 * go back and download the model you want in the gpt4all-cloneable-apptemplate qube * shut down gpt4all-cloneable-apptemplate * then make a new gpt4all-scratch-2 clone the same way you made gpt4all-scratch-1

Comment below: Feel free to add your comments below, or post a better procedure if you have one.