LTT Labs Machine Learning Testing

[benjamincburns]

I vaguely recall from one of the recent LTT Labs videos (or maybe it was the $1M computer series) that Linus was talking about wanting to do more use-case specific hardware testing, especially for machine learning applications. Is that true?

 

I believe that right now there are far more hobbyist ML practitioners/researchers out there than there are professional ones, and some of those are forming communities that are doing some really impressive work. For example, the RLGym community uses reinforcement learning and the OpenAI Gym specification to build Rocket League bots that compete in the RLBot tournaments. For most of these communities, including RLGym, access to compute by far the most limiting factor for progress. Admittedly Rocket League might not sound like all that worthy of a cause, but the community is actually advancing the state of the art in multi-agent learning as well as hybrid imitation learning/reinforcement learning techniques, and these advancements can have major implications for other fields such as advanced robotics control systems.

 

With that in mind, if LTT/LTT Labs is considering doing more use-case specific testing for ML applications, have you considered whether it's possible to perform that testing in such a way that supports community-lead AI/ML projects like RLGym? If it's something that you'd like to do but aren't quite sure how, I'd be very happy to volunteer some time to help figure it out.

 

Full disclosure: I'm an active participate in the RLGym community, but I'm not an RLGym project maintainer, and I don't own or receive revenue from the RLGym twitch channel or anything else affiliated with RLGym. I'm just a hobbyist who loves hacking on distributed systems. I focus most of my community time/effort on improving the efficiency of the learning systems we use so that we can squeeze every last bit of impact out of any/all donated compute.

[Roswell]

12 minutes ago, benjamincburns said:

I believe that right now there are far more hobbyist ML practitioners/researchers out there than there are professional ones

I would image the ML industry is much larger than you think it is. Corporate interest in machine learning is aimed more for behind the scenes deployment rather than public facing products.

 

I also can’t fathom how much money gets funneled into government agencies for development of surveillance/logistics/military tools.

[benjamincburns]

6 minutes ago, Roswell said:

I would image the ML industry is much larger than you think it is. Corporate interest in machine learning is aimed more for behind the scenes deployment rather than public facing products.

I'm not sure it's fruitful to debate this, really. Maybe I overstated things and there are in fact more professional ML practitioners than there are hobbyist practitioners, but there are huge numbers of hobbyist practitioners all the same. I think it's still fair to say that the impact that these community groups can have on the field could be much more substantial if they had more compute to support their projects.

[Biohazard777]

36 minutes ago, benjamincburns said:

With that in mind, if LTT/LTT Labs is considering doing more use-case specific testing for ML applications, have you considered whether it's possible to perform that testing in such a way that supports community-lead AI/ML projects like RLGym?

39 minutes ago, benjamincburns said:

I focus most of my community time/effort on improving the efficiency of the learning systems we use so that we can squeeze every last bit of impact out of any/all donated compute.

22 minutes ago, benjamincburns said:

I think it's still fair to say that the impact that these community groups can have on the field could be much more substantial if they had more compute to support their projects.

Maybe I am just tired and reading all of this wrong...
Are you asking LTT labs to donate compute? 

[benjamincburns]

35 minutes ago, Biohazard777 said:

Maybe I am just tired and reading all of this wrong...
Are you asking LTT labs to donate compute? 

Yes and no. I'd be thrilled if they wanted to donate compute to my project/community, but that's not exactly what I'm asking. I'm mostly wondering if this is something that they've considered, and if so, how deeply they've thought about it, what barriers they see, what their goals/plans are, etc.

[benjamincburns]

2 minutes ago, benjamincburns said:

I'm mostly wondering if this is something that they've considered...

And by "this" I mean donating compute in such a way that it could be consumed by community-run, FOSS ML projects.

[Biohazard777]

7 hours ago, benjamincburns said:

Yes and no. I'd be thrilled if they wanted to donate compute to my project/community, but that's not exactly what I'm asking. I'm mostly wondering if this is something that they've considered, and if so, how deeply they've thought about it, what barriers they see, what their goals/plans are, etc.

7 hours ago, benjamincburns said:

And by "this" I mean donating compute in such a way that it could be consumed by community-run, FOSS ML projects.

IIRC Linus mentioned he is interested in practical applications of AI / ML in the lab, as in something they could use in their daily operations.
As for LTT / LMG supporting FOSS (be it AI related or not)... I am unaware of them doing anything like that in the past, be it via donations or code contributions on open source projects. They could be doing it privately, and if they do - hats off to them (but I kinda doubt it). The only exception to that being Folding@Home, but then again I don't know much of their own compute they are contributing, IMO their role is more about rallying people to join the F@H. That brings me to what they do and have done is talk about FOSS sometimes and provide exposure, so if they find RLGym to be interesting material for many of their viewers maybe they will upload a video about it, but I wouldn't count on them donating compute.

[ouchie]

They are looking for a Full-Time ML/Computer Vision Engineer that will assist LTT Labs and design/develop software for testing purposes.

 

[benjamincburns]

16 hours ago, Biohazard777 said:

That brings me to what they do and have done is talk about FOSS sometimes and provide exposure, so if they find RLGym to be interesting material for many of their viewers maybe they will upload a video about it, but I wouldn't count on them donating compute.

Yeah, I suspect you're right. That said, I kinda wish that I hadn't mentioned anything about RLGym in this post. While it's true that I want to see that community gain access to more compute, I think I actually care more about solving a broader problem.

 

That problem is some amalgamation of the following:

1. Given that I have idle compute, how can I add it to some giant distributed/decentralized compute pool for others to consume (obviously in a secure way with minimal setup and maintenance effort)?
2. Given that I'm a volunteer/community researcher/practitioner who is limited by my available compute, how can I set up my project to run on some giant distributed/decentralized pool of idle compute?

I understand that projects like BOINC aim to solve this, so maybe I should start there, but the main problems I see with that platform are:

• Compute owners need to find your specific project and join it to be able to contribute to it, meaning that projects generally need to market themselves
• Getting your project listed on the BOINC website isn't a transparent process
• Writing BOINC applications is fairly complex, even if you already have a distributed application that you'd like to put on the platform

So I dunno, maybe I'll have to try building a BOINC for ML, or maybe some sort of BOINC client that lets compute donors sign up for a general pool that all projects can pull from.

 

Either way, this is somewhat far removed from my original post, so I suppose I'll leave it at that.