Hello and sorry in advance if I posted this in a wrong part of the forum - I'm new here.
I wanted to suggest to the LTT team to start doing machine learning benchmarks and wanted to provide them with a relatively simple way to do it.
I am a physicist and doing particle physics research is almost entirely based on machine learning nowadays. Seeing some benchmarks of new GPUs and CPUs would be rather beneficial - knowing stuff like whether a much cheaper 1080Ti is just as fast as a K80 is definitely useful to many people in that niche - all you nerds screaming now about the bonus of double precision I know it is important but not for all research.
I wanted to propose to start benchmarking using TensorFlow from Google, mostly because of the ease of use and automated threading - trust me, you don't want to use CERN's proprietary ROOT, it's ugly and outdated by around 20 years. Below I link a project on my GitHub which uses MNIST (handwritten digits) examples passed through a Convolutional Neural Network for classifying them - extremely simple computer vision if you'd like. I also added almost full support for TensorBoard results visualisation software which is rather marvellous though its small precision on time measurements may make you want to add a snipett of code to measure it yourself (or make many more epochs than this example - easily tweakable - and divide by total time shown and live with the slight error).
Link: https://github.com/krisfur/TensorBoard-CNN-Visualization-Example
For benchmarking CPUs I'd suggest using the not GPU-accelerated version of TensorFlow.
For benchmarking GPUs naturally install the GPU accelerated version, but remember that if installation of that has problems you can overcome some of them by disabling the integrated GPU of your CPU - definitely need to do that on laptops and we are reaching a point where laptops could actually do some basic machine learning stuff.
Also you should run it with at least 16GB of ram as the final evaluation basically loads the whole thing onto the memory for evaluation. More ram isn't a problem for you guys anyway.
I will attach some screenshots of how nice the TensorBoard output looks. I'd love to provide something different than the usual MNIST example but I'm not in possession of some good practice data that I would be allowed to share and can't be bothered to make some mock data as you may still never use it.
Sidenotes:
Benchmarking based on final accuracy isn't always the best choice, there's a lot of randomness involved - you should only care about the time. You could save a trained model and just run test batches through it if you want to do that, but that's a bit more work than I have time to describe here - not necessarily hard, but you wouldn't be able to get a lot of data about training speed if you don't do the training (duh!).
MNIST images are tiny so training times are short, meaning you can benchmark a lot quickly, with bigger images you'd need to do some tweaking to the CNN and it would take a long time. I remember doing computer vision on ~1500x1500px images and it literally took days on a system with two K80s and we did have to introduce a downsampling mechanism. Also using MNIST data means you just download them every time and delete afterwards, so do have an internet connection. Using your own images requires making a way of reading and batching them which is a day's work.
If you do end up using this could you please mention me in a video, my mates would die haha. If not it's fine anyway. Cheers from London.
