Cataracts are caused by the denaturation of proteins in the lenses of your eyes.

Setting up a Linux folding system with nVidia GPUs
#1
EDIT: This thread is now closed, as I've consolidated several threads into one. You may find the most up to date version of this guide, plus basic information on F@H here.

If you're anything like me, you probably love Windows because if nothing else, it just works. It might have its issues with privacy, security or performance, but if you want to get a system set up, the amount of work typically required would be exponentially less on Windows, than it would be for a system running Linux.

Problem is, when dealing with the realm of citizen science, particularly the vast majority of projects that focus on using computing power to complete tasks, Windows falls short on two key factors: Performance and reliability. The following guide aims to walk you though setting up a system running Linux in a (hopefully) simple and straightforward manner.

Please note that this guide is for users with nVidia cards. As I have no AMD/ATI cards and therefore no experience installing such a system, I cannot provide tested instructions.

For this guide, feel free to choose between either Mint, Ubuntu, or Lubuntu as your distro. They're all Linux systems, but they come with varying software and interfaces. However, these three distros are among the simpler to use, and have good support. For a dedicated system that won't be used outside of F@H, pick Lubuntu. If you wish to use the system regularly while you fold, I'd recommend Mint with the Cinnamon desktop. Whichever distro you go for, pick the 64 bit over the 32 bit. Unless you're running on hardware from the early 2000s, there's no need to use 32 bit.

After downloading, you'll want to download a tool like PowerISO on your Windows system to burn the .ISO file to a disc, or make a bootable flash drive.

Once your removable media is ready to go, insert it into the computer you wish to install your distro on, and boot it up. Keep an eye on the very first thing you see on your screen when the computer turns on, as it will mention somewhere that you should press a certain key to take you to a boot order setting. The boot order option might not exist, in which case you need to look out for the key you need to press to enter the BIOS or UEFI. Press it before the screen disappears. If you're entering through the BIOS/UEFI, look for the boot menu. There will be a boot order list which you will need to swap around, so that your removable media becomes the first in the boot order. Once this is done, save and exit.

Your distro should start loading. Either an installer will appear, or it will load a desktop that would appear as if the system is already installed. If the latter, you will find an icon to install the system, which would bring up the same installer. Follow these instructions as they come. Most should be simple to go through. Once finished, the installer will ask you to remove the disc/flash drive and restart. Normally the computer should then boot into the installed operating system, but if it gets stuck, return to the boot order and set the first device to the one you installed your distro to.

As soon as you get to the desktop, you'll want to open up the command line, which henceforth will be referred to as the terminal. You can usually find a button at the bottom of the screen to open it up, or you can simply use Ctrl + Alt + T. Type or copy each line, one by one, into the terminal:

Code:
wget http://launchpadlibrarian.net/109052632/python-support_1.0.15_all.deb

sudo dpkg -i python-support_1.0.15_all.deb

Note that Ctrl + V will not work in the terminal. You will need to right click inside it and click paste. Ctrl + C in your browser will still work just fine though.

After entering the second line, you will be asked for the password you used when you set up your account during installation. This will then install the .deb file you just downloaded, which will be a prerequisite for the F@H software. Some additional software will also need to be installed, namely the drivers and OpenCL support for nVidia cards. Firstly, you will need to specify an extra place for the updater/installer to look for additional software, which in this case would be for the drivers:

Code:
sudo add-apt-repository ppa:graphics-drivers/ppa

When prompted, press enter. If you are using the same terminal window from before, you will not need to enter your password again, but if you closed it and opened a new one, you will need your password. Next:

Code:
sudo apt-get update

This will update the database that the apt-get function uses to find software as well as inform you of other available updates. We can worry about the latter once everything is set up though. Once this is done, type the following line, but instead of pressing enter, press tab:

Code:
sudo apt-get install nvidia-

Pressing tab will either autocomplete what you're typing, or if there are multiple entries that could complete what you started typing, it will display a list. In this case, it will display several nVidia driver versions. Pick the highest driver version. For example, "nvidia-370". Don't bother with the ones ending with "dev". Once you complete this line, press enter to install. When asked if you want to continue, press "y" and enter.

Note that nvidia-370 is verified stable. If the highest driver version does not work well, you can downgrade. Alternatively, you can check F@H communities to see if anyone else has verified a version to be working. Once the drivers are installed, install the rest:

Code:
sudo apt-get -y install python-gnome2 mesa-common-dev freeglut3-dev nvidia-settings gedit

Next, download and install FAHClient and FAHControl from the Folding@Home site. To keep things easy, download them to your home folder (the one named after your user name). If you can't install them by double clicking and following the graphical interface, type "sudo dpkg -i ", followed by the file name of the FAHClient and FAHControl files you downloaded. Remember that you can use tab to autocomplete the file name if it's too long.

During the installation for FAHClient, you will be asked for your F@H user name, team number, and passcode. Go ahead and put these in. When the two programs are installed, go back to your terminal and type:

Code:
cd /var/lib/fahclient

ls

This will navigate you to the fahclient folder and list its contents. Look for a file called GPUs.txt. If the file does not exist, type:

Code:
sudo wget http://fah-web.stanford.edu/file-releases/public/GPUs.txt

Skip this step if the file is already there. Remember, to type this into the same terminal you performed the cd command in, or else you will download the file to the wrong place.

Go ahead and restart your computer now, either by typing reboot into the terminal, or from the relevant button on your taskbar/start menu. Once you're back into the desktop, open your terminal and type "FAHControl". Keep in mind that this will be case sensitive. You should see that familiar window you've been using on Windows. If you don't see your GPU in the slot list, add it manually by going to Configure > Slots > New Slot. Select the GPU button and save. This should be sufficient for a single GPU setup, and you should see your GPU slot initialising.

For multi GPU setups, you may need to manually assign the ID of each individual GPU when you're adding the slot. This should not be necessary, but if needed, type "nvidia-settings" into the terminal to open up nVidia X Server. On the left, you will see a list showing the device number of each GPU installed (eg, GPU0, GPU1, etc.). The number will be the ID of each card. Regardless of whether or not you need to manually set the ID or leave it at -1 for F@H to decide, save and monitor the folding slots. If they are stuck at Ready, or if X Server shows the GPU usage of a card to be anything other than 85-100%, you may need to manually set the CUDA and OpenCL indexes for each slot.

To do this, pause one GPU slot in FAHControl. Go to the slot settings for that same GPU. Set CUDA and OpenCL to 0, then save and resume the slot. If there are no changes, and it still doesn't fold after 20 or 30 seconds, repeat the process, increasing the index values by 1 each time. Keep repeating this until the slot works, then repeat the entire process for every other problematic GPU you have. This took about 5 minutes on my quad card setups. That said, if your indexes are in the double digits, then there may be other issues.

If everything has gone well, your cards should be up and running. Remember that you generally need one CPU core per GPU slot, so the number of cores used for CPU folding may have to be manually adjusted, if not completely disabled. Whatever the case may be, confirm once again in X server that each GPU is running at full or near full load.

You're now hopefully folding on Linux. But the fun does not stop there. You may wish to set a custom fan speed for your cards. Unfortunately, software like PrecisionX or Afterburner do not exist on Linux, so fire up the terminal once more and type the following:

Code:
sudo nvidia-xconfig -a --cool-bits=28 --allow-empty-initial-configuration

cd ~

gedit fancontrols

Note that you may need to repeat the first line twice in a row, before the "cd ~" command.

A text editor should appear after the final line. Here, you can add commands that will set your fan speed to whatever value you would like. You will need the GPU IDs of each GPU you have first. So let's say you have three GPUs with IDs 0, 1 and 2, and you want all their fans at 100%. Add the following lines to the editor:

Code:
nvidia-settings -a [gpu:0]/GPUFanControlState=1
nvidia-settings -a [fan:0]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:1]/GPUFanControlState=1
nvidia-settings -a [fan:1]/GPUTargetFanSpeed=100
nvidia-settings -a [gpu:2]/GPUFanControlState=1
nvidia-settings -a [fan:2]/GPUTargetFanSpeed=100

The ControlState line enabled fan control the the specified GPU, while the FanSpeed sets the speed as a percentage. If you wish to overclock, you can also add the following lines to the file:

Code:
nvidia-settings -a [gpu:0]/GPUGraphicsClockOffset[3]=
nvidia-settings -a [gpu:0]/GPUMemoryTransferRateOffset[3]=

Adding a number after the "=" allows you to adjust the offset of the core and memory clocks in MHz. It is very important to keep in mind that this is an offset from the stock clocks. Forgetting this little detail can damage your card. If your stock core is 1000 MHz and you want to overclock to 1100 MHz, do NOT type in 1100, or you'll basically tell the software to run your GPU at 2100 MHz. Instead, type in 100.

Whether you choose to overclock or just set fan speeds, save the file and exist. In the same terminal, type:

Code:
sudo chmod +x fancontrols

This allows the file to be executable. Next, go to your startup applications list. This will be listed in your system settings menu, or through a search for startup applications. Add a new startup entry and browse for the file you just saved. If you followed the commands before opening the text editor, it should be in your home directory, or ~. Select the file, give it a name, and save. If you reboot, you should hear your fans adjusting to your set speeds when the system starts up.

Hopefully this should cover the basics. If you like to run dedicated systems, you might want to look into downloading Teamviewer to remotely connect to your following systems from your main computer. Temperature monitoring is possible from X server in the thermal settings of each GPU as well.

I know this all looks like it would be a pain in the ass to do. However, F@H performs better with Linux, as does BOINC. A dedicated folding rig should always run Linux, and unless you're gaming, you could probably also make the switch if you have the patience to learn a little. I myself am not patient in the slightest, and while Linux still gives me aneurysms, I can at least acknowledge the pros of it.
[Image: sigimage.php?FAHUser=hiigaran&FAHTeam=21...hangeling1]
Likes: 10esseepony
#2
Well isn't this just overly convenient...

I just downloaded and burned Linux Cinnamon this morning.  Figured I'd finally figure out linux for my dedicated  on the wall.  Mint Cinnamon looked fairly reminiscent of Windows, so I chose that one.  Haven't tried it yet.   Heart Twilightsmile

[Image: sigimage.php?u=658963&t=212997&b=rainbowdash1]
Likes:
#3
Thanks for the Linux tutorial.  I ran Mint on my Folding rigs some years back so I remembered some of the lesson’s learned then, but the tutorial was very helpful getting my dedicated rigs back to Linux.  I won’t bore everyone with the details, just suffice it to say I found myself down some rabbit holes and your tutorial helped me escape a couple of them. 

It appears to be making a noticeable difference.

Thanks for taking the time to put it together.
Likes:
#4
The thing that has always annoyed me with Linux is that it's just so...finicky. Following this exact same guide on a newer Mint, seems to have issues with nvidia-server. Turns out the solution is to downgrade the kernel. Rather counter-intuitive.

I'm going to have to reinstall my folding rigs in the near future, as I'm upgrading the drives, so I may have to revisit this guide.
[Image: sigimage.php?FAHUser=hiigaran&FAHTeam=21...hangeling1]
Likes:
#5
nvidia on linux is PITA compared to AMD these days. Then again, AMD does not have functional OpenCL drivers quite yet. AMD is moving all their drivers to be open source in the linux kernel, so it will be easier for distros and kernel devs to make sure everything works correctly.
IRC channel Yay for brony@home (irc.canternet.org #foldingismagic)
http://derpy.me/gLbV2

[Image: sigimage.php?u=605723&t=212997&b=twilight3]

Likes:
#6
In case people care, you *can* copy/paste in terminals, you use ctrl+shift+c/v instead of just ctrl+c/v.

Other than that, I was able to setup my Linux box with Folding@Home perfectly fine. Now, it gets rather unusable as my daily driver when I put my GPU to work, so my plan is to just switch out configs so that it only runs on my CPU when I'm actively using my computer, until I find a solution.
[Image: sigimage.php?FAHUser=FoxWithGun&FAHTeam=...=spitfire1]
Likes:
#7
FoxWithGun, thanks for the CTRL+Shift tip.

This guide works beautifully with Mint 18.2/18.3 FOR SINGLE CARDS. Lubuntu's current version, not so much. And I took one look at Ubuntu's desktop and ran away.

I did have some trouble with Mint 18.2, on multiple GPU set-ups, and also that system's MB had secure boot, which may have been a factor as well. But Mint 18.3 is performing as expected. The following is a bit of a build log:

The trouble with Mint 18.2 came after I added more GPUs. I decided to just reinstall. It wouldn't, "CPU stalled" errors. Remove GPUs, install generic card, cpu stalled. Reset BIOS, cpu stalled. Fortunately after messing with Lubuntu and at least LOOKING at Ubuntu, I remembered Mint had updated to 18.3. So far so good, about 1/3rd through this very guide, and all is well.

And, dammit. Nvidia 384 worked well with my single 1080s, but not working with triple 1070s. Reverting to 381 drivers......  To be fair, I added all 3 1070s at once to Folding with the 384 drivers, but only one at a time with 381.

So now I'm GPU folding. Let's up the fan speed via 'fancontrols', restart, and....dammit again..."Cinnamon has crashed and running in fallback mode". Same problem as 18.2 originally.  It appears the manual fan speed setting is causing trouble? So, reinstalling 18.3....

Trying driver 384 again, to be sure it wasn't user error (adding all 3 cards at once to the folding manager)....doing all that Python stuff....Installing FAHclient, FAHcontrol, Aaaaahhhh, I missed a reboot at this point. Back up and running, UNLOCKING fan controls, but not adding fancontrols, reboot, and fudge. "Cinnamon has crashed and running in fallback mode"  Not fan controls, but the unlocking of the overclock part causing trouble, I'm thinking?

Reinstalling 18.3... @2am....3am....384 not folding....reverting to 381.....4am.....am I dozing off or is it really this slow? Almost out of cigs....gonna be worth it though, I know it....was a substantial improvement for my 1080s....

Driver 381 for my 1070s works. NOT EVEN GOING to try to mess with fan speeds or the like at this point. NOPE. Not gonna do it.

And now it is 4.20 time, and you know what that means? Yep, bedtime, finally.
[Image: sigimage.php?u=685285&t=198&b=octavia1]

ThunderStrike: 2P Xeon e5-2695-v3@2.8GHz, 28 cores, 2x GTX1080
BumbleBee, Devastator, WUSS, and Skeletor: Xeon e5-2683-v3@2.5GHz, 14 cores x4, 3x GTX1070/4x GTX1080
Mjölnir: Core i7-5820K@3.3GHz, GTX1080Ti
Likes: Talifan9
#8
I take it your CPU Stalled errors were during the boot stage, right? I think I solved that with adding pci=nomsi to GRUB. I say 'think', because I had to do a lot of tweaks, so I can't be certain if that was the solution to the stalled errors, or other errors I had.
[Image: sigimage.php?FAHUser=hiigaran&FAHTeam=21...hangeling1]
Likes:
#9
Yes, the stalled CPU was during boot. As for adding pci=nomsi to GRUB, uhm, should there be a guide for that? lol :)
[Image: sigimage.php?u=685285&t=198&b=octavia1]

ThunderStrike: 2P Xeon e5-2695-v3@2.8GHz, 28 cores, 2x GTX1080
BumbleBee, Devastator, WUSS, and Skeletor: Xeon e5-2683-v3@2.5GHz, 14 cores x4, 3x GTX1070/4x GTX1080
Mjölnir: Core i7-5820K@3.3GHz, GTX1080Ti
Likes:
#10
I never added it because I wasn't sure if it actually was the solution, but you can add it temporarily by adding it to the boot options. It will remain valid until the next boot. If it works, you can later make the change permanent. Here's a guide on how to do both.
[Image: sigimage.php?FAHUser=hiigaran&FAHTeam=21...hangeling1]
Likes:
#11
What type of performance increase are we talking about? Might be enough for me to finally learn Linux once I finish my semester.
Rig: EVGA X99 Classified, Core i7-5960X 4.5GHz All-Cores, 2x GTX 1080, 32GB DDR4 3200MHz, Windows 10 Edu
[Image: sigimage.php?u=730961&t=212997&b=applejack1]
Likes:
#12
My ppd went from 5.5M on Win10 to 6.2M (or more, hasn't quite stabilized yet [less than a week so far]). The biggest jump for me was with my stock GTX1080's, which went from 650k ppd to 830k ppd, each.
[Image: sigimage.php?u=685285&t=198&b=octavia1]

ThunderStrike: 2P Xeon e5-2695-v3@2.8GHz, 28 cores, 2x GTX1080
BumbleBee, Devastator, WUSS, and Skeletor: Xeon e5-2683-v3@2.5GHz, 14 cores x4, 3x GTX1070/4x GTX1080
Mjölnir: Core i7-5820K@3.3GHz, GTX1080Ti
Likes:
#13
For those with multiple systems (headless/monitor-less), what do you use to monitor your systems? Does HFM.net work with Linux? Any additional steps you recall having to do to make it work?

TeamViewer is 'ok' but one system keeps reverting to 640x480 resolution.
[Image: sigimage.php?u=685285&t=198&b=octavia1]

ThunderStrike: 2P Xeon e5-2695-v3@2.8GHz, 28 cores, 2x GTX1080
BumbleBee, Devastator, WUSS, and Skeletor: Xeon e5-2683-v3@2.5GHz, 14 cores x4, 3x GTX1070/4x GTX1080
Mjölnir: Core i7-5820K@3.3GHz, GTX1080Ti
Likes:
#14
There is a way to monitor/change different systems from the same computer within the folding@home software, but I am unaware of how to properly set this up.
IRC channel Yay for brony@home (irc.canternet.org #foldingismagic)
http://derpy.me/gLbV2

[Image: sigimage.php?u=605723&t=212997&b=twilight3]

Likes:
#15
You can use the FAHControl program to communicate with multiple headless linux clients. By default the clients only allow connections from the local host, but can be configured to allow connections from other machines using a password or from a range of ip addresses. Run FAHClient --help and look for the Remote Command Server section. Also if you have a firewall on the node make sure it allows port 36330 tcp inbound. Personally I don't use this method. We use a software automation package called salt (saltstack.com) on all our nodes already that I use for mass remote execution and pushing files to hosts. It requires a bit more time to figure out, but if you manage a lot of nodes like I do, it's super handy hoofy. That combined with some cron jobs and custom scripts ensures that my nodes are always folding. We use a program called Zabbix for monitoring the nodes with custom checks that will send email, or instant messages when it detects issues with the hosts. I haven't looked at HFM.net before, so I don't know if it will work with the linux client or not.
[Image: nekolepsy.gif]
Likes:




SOON