Jump to content

FAH throwing away almost completed job

I returned to my FAH client this morning to find it had thrown my current job away. It was 95% complete, representing 66.5 out of 70 hours of work. Looking at the log I see it timed out. This is madness.

 

Recently I've noticed that although a typical job takes my machine 3-5 hours to complete, a minority take over 2.5 days of uptime. This was such a job: at 8 hours, 5 days a week it would take nearly two weeks to complete. It happened there was a long weekend involved.

 

The title of this project is Folding At Home, not on your managed server farm. I don't want to leave my computer running at max heat output when I'm not there. But neither do I want to waste MIPS, cash and CO2 on a task that bails at 95%. Especially not with energy prices heading for the stratosphere.

 

Here are some helpful suggestions. I've read the threads saying it's hard to predict how long a unit will take before you download it. But within 15 minutes the client has calculated this pretty accurately. So why not provide a control saying how long you're prepared to wait for and get the client to abort if it's going to take longer. Or, better, extend the timeout if you're clearly on track to complete soon. Or even just provide a button allowing the user to reject the job early on?

Link to comment
Share on other sites

Link to post
Share on other sites

I can see how that would be frustrating. If you could provide some details about your hardware and how your limiting it to folding for just 8 hours a day for weekdays we might be able to offer some suggestions.

 

Personally I don’t fold on my CPUs anymore. I just use GPUs for Folding and I run BOINC on my CPUs which tends to be a little more forgiving of being interrupted and more easy to schedule to run when electricity is least expensive.

FaH BOINC HfM

Bifrost - 6 GPU Folding Rig  Linux Folding HOWTO Folding Remote Access Folding GPU Profiling ToU Scheduling UPS

Systems:

desktop: Lian-Li O11 Air Mini; Asus ProArt x670 WiFi; Ryzen 9 7950x; EVGA 240 CLC; 4 x 32GB DDR5-5600; 2 x Samsung 980 Pro 500GB PCIe3 NVMe; 2 x 8TB NAS; AMD FirePro W4100; MSI 4070 Ti Super Ventus 2; Corsair SF750

nas1: Fractal Node 804; SuperMicro X10sl7-f; Xeon e3-1231v3; 4 x 8GB DDR3-1666 ECC; 2 x 250GB Samsung EVO Pro SSD; 7 x 4TB Seagate NAS; Corsair HX650i

nas2: Synology DS-123j; 2 x 6TB WD Red Plus NAS

nas3: Synology DS-224+; 2 x 12TB Seagate NAS

dcn01: Fractal Meshify S2; Gigabyte Aorus ax570 Master; Ryzen 9 5900x; Noctua NH-D15; 4 x 16GB DDR4-3200; 512GB NVMe; 2 x Zotac AMP 4070ti; Corsair RM750Mx

dcn02: Fractal Meshify S2; Gigabyte ax570 Pro WiFi; Ryzen 9 3950x; Noctua NH-D15; 2 x 16GB DDR4-3200; 128GB NVMe; 2 x Zotac AMP 4070ti; Corsair RM750x

dcn03: Fractal Meshify C; Gigabyte Aorus z370 Gaming 5; i9-9900k; BeQuiet! PureRock 2 Black; 2 x 8GB DDR4-2400; 128GB SATA m.2; MSI 4070 Ti Super Gaming X; MSI 4070 Ti Super Ventus 2; Corsair TX650m

dcn05: Fractal Define S; Gigabyte Aorus b450m; Ryzen 7 2700; AMD Wraith; 2 x 8GB DDR 4-3200; 128GB SATA NVMe; Gigabyte Gaming RTX 4080 Super; Corsair TX750m

dcn06: Fractal Focus G Mini; Gigabyte Aorus b450m; Ryzen 7 2700; AMD Wraith; 2 x 8GB DDR 4-3200; 128GB SSD; Gigabyte Gaming RTX 4080 Super; Corsair CX650m

Link to comment
Share on other sites

Link to post
Share on other sites

Hi, I can empathize with how irritating it is when that happens because I had similar problems when folding on my Surface 6.

 

This is more of a workaround than a solution, but what I found works for me is reducing the # of threads to 4 (or maybe 6, haven't tested this). The big WUs are only released for CPUs with more threads, and whilst F@h productivity might be reduced this way, at least it's far less likely to go to waste. Also, this would reduce electricity costs and heat generated.

Desktop 1 : Ryzen 5 3600 (O/C to 4Ghz all-core) | Gigabyte B450M-DS3H | 24GB DDR4-2400 Crucial(O/C to 2667) | GALAX RTX 2060 6GB | CoolerMaster MWE 650 Gold

 

Desktop 2 : i5 10400 | 32GB DDR4-3200(@ 2667Mhz) |  EVGA GTX 1070 SC 8 GB | Corsair CV450M

                        

Laptop : ASUS ROG Strix G17 : i7-10750H, 16GB RAM, GTX 1660Ti 6GB(90W), 1TB NVMe SSD

 

Yoga 3 14 - i7-5500U, 8GB RAM, GeForce GT 940M, 256GB SSD

Link to comment
Share on other sites

Link to post
Share on other sites

  • 2 weeks later...

Sorry for the late response @Gorgon

 

My hardware is 2 x (2013-era) Xeon 2630s with 8 cores each. No GPU as it's a server.

 

I've tried max-packet-size = small and it's so far limited downloads to those taking 1 day to complete. But too small a sample size of jobs to be sure yet.

 

Link to comment
Share on other sites

Link to post
Share on other sites

5 hours ago, nigelramsden said:

Sorry for the late response @Gorgon

 

My hardware is 2 x (2013-era) Xeon 2630s with 8 cores each. No GPU as it's a server.

 

I've tried max-packet-size = small and it's so far limited downloads to those taking 1 day to complete. But too small a sample size of jobs to be sure yet.

 

Nice! the e5-2630 v3 Haswells? (They're the only 8-core ones in the E5 lineup I believe) I'm still running an e3-1231-v3 on my Windows Daily Driver so around the same vintage.

 

How may threads are assigned to the Folding slot? If your running say 12 you could split it and assign 2 x 6t instead by changing the number of threads for the existing CPU slot from -1 to the new amount then creating a second CPU "slot" as desired. That may force smaller tasks to them. Also there is this weird issue with F@H where slots with threads equaling multiples of primes above 5 cause issues so what you set may be adjusted.

FaH BOINC HfM

Bifrost - 6 GPU Folding Rig  Linux Folding HOWTO Folding Remote Access Folding GPU Profiling ToU Scheduling UPS

Systems:

desktop: Lian-Li O11 Air Mini; Asus ProArt x670 WiFi; Ryzen 9 7950x; EVGA 240 CLC; 4 x 32GB DDR5-5600; 2 x Samsung 980 Pro 500GB PCIe3 NVMe; 2 x 8TB NAS; AMD FirePro W4100; MSI 4070 Ti Super Ventus 2; Corsair SF750

nas1: Fractal Node 804; SuperMicro X10sl7-f; Xeon e3-1231v3; 4 x 8GB DDR3-1666 ECC; 2 x 250GB Samsung EVO Pro SSD; 7 x 4TB Seagate NAS; Corsair HX650i

nas2: Synology DS-123j; 2 x 6TB WD Red Plus NAS

nas3: Synology DS-224+; 2 x 12TB Seagate NAS

dcn01: Fractal Meshify S2; Gigabyte Aorus ax570 Master; Ryzen 9 5900x; Noctua NH-D15; 4 x 16GB DDR4-3200; 512GB NVMe; 2 x Zotac AMP 4070ti; Corsair RM750Mx

dcn02: Fractal Meshify S2; Gigabyte ax570 Pro WiFi; Ryzen 9 3950x; Noctua NH-D15; 2 x 16GB DDR4-3200; 128GB NVMe; 2 x Zotac AMP 4070ti; Corsair RM750x

dcn03: Fractal Meshify C; Gigabyte Aorus z370 Gaming 5; i9-9900k; BeQuiet! PureRock 2 Black; 2 x 8GB DDR4-2400; 128GB SATA m.2; MSI 4070 Ti Super Gaming X; MSI 4070 Ti Super Ventus 2; Corsair TX650m

dcn05: Fractal Define S; Gigabyte Aorus b450m; Ryzen 7 2700; AMD Wraith; 2 x 8GB DDR 4-3200; 128GB SATA NVMe; Gigabyte Gaming RTX 4080 Super; Corsair TX750m

dcn06: Fractal Focus G Mini; Gigabyte Aorus b450m; Ryzen 7 2700; AMD Wraith; 2 x 8GB DDR 4-3200; 128GB SSD; Gigabyte Gaming RTX 4080 Super; Corsair CX650m

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×