Remote testing setup

[martward]

Hi all,

 

I am an AI student, and often I have to run tests or train models which can take multiple hours to even days. Running this on my main PC is very annoying since it means I can't use it for any extensive things while it is busy.

I was thinking of making a second system which would be used solely to run my tests on so that I can still use my PC when I am running tests.

I mostly use Python and Matlab, so Linux would be the most fitting OS for this project.

Basically I want to have a render farm that does AI testing in stead of rendering, and I am not sure how to set it up OS wise.

 

Thanks

[Hamosch]

I kind of do this but not with AI programming.

 

I would suggest installing Arch and getting a CLI environment set up there (I preffer this to a GUI), http://se.mathworks.com/matlabcentral/newsreader/view_thread/90363 for matlab and just the python cli on the server.

 

Then I would suggest setting up a nfs/samba file share (whatever OS your running on your desktop) on that machine with or without raid (could also be a nas but thats outside the scope right now). 

Now you just work on your projects in that shared folder from your main desktop and when you're ready to run a simulation/compile just ssh into the server and start the program/compile/whatever from there.

 

This is the easiest most reliable way to me.

[martward]

8 minutes ago, Hamosch said:

I kind of do this but not with AI programming.

 

I would suggest installing Arch and getting a CLI environment set up there (I preffer this to a GUI), http://se.mathworks.com/matlabcentral/newsreader/view_thread/90363 for matlab and just the python cli on the server.

 

Then I would suggest setting up a nfs/samba file share (whatever OS your running on your desktop) on that machine with or without raid (could also be a nas but thats outside the scope right now). 

Now you just work on your projects in that shared folder from your main desktop and when you're ready to run a simulation/compile just ssh into the server and start the program/compile/whatever from there.

 

This is the easiest most reliable way to me.

That sounds like what I'm looking for, couple of questions:

- Any reason you'd use Arch in stead of Ubuntu for instance?

- Would I be able to set the file share up for remote access outside my home? I sometimes work on the university so I can't reach the files that would be a problem.

- Is there away to use something like wake-on-lan for this setup, or would I need to have the PC on if I want to work from the university?

 

Thanks

[Hamosch]

14 minutes ago, martward said:

- Any reason you'd use Arch in stead of Ubuntu for instance?

- Would I be able to set the file share up for remote access outside my home? I sometimes work on the university so I can't reach the files that would be a problem.

- Is there away to use something like wake-on-lan for this setup, or would I need to have the PC on if I want to work from the university?

 

 

- I just preffer arch because it has rolling releases, pacman and more customizable and in my expirience is better for development, you could use any distro won't really matter, just personal preference.

- For this I would using a VPN (openVPN) to connect to your private network from the outside (might be the most secure option, but others might have other ideas)

- I'm guessing with the VPN option from the suggestion above you should be able to use WoL remotly (but never tried it personally, just have a 247 home lab/server myself)

[.:MARK:.]

Simplest way I can think of is set up a debian machine, port forward SSH, and set up public keys for security. Then install screen so you can run your python files and detach the session, so you can close the SSH client.

[martward]

@Hamosch @.:MARK:. 

 

Thanks for the information, I will do some more research but I definitely got more insight into the problem :D.