MarkBench Development and Feature Requests

[joser93]

Would be cool to team up with Phoronix for more benchmarks and since markbenchmark is coded in go and python, can be cross platform and even include some of the benchmarks that Anthony has scripted on Linux.

 

For visualization on LTT should consider make a custom dashboard that help them to select certain cards or CPU's and dynamically have your charts and tables to check out quickly, if you want suggetions about this and keep it open source based, consider D3Js or ChartJS if you want to build a custom website for the labs or using Pentaho if you don't want to code too much and process all that database without so much mess. For everyone else, maybe in the future a PR or an independant open source project can help to load this CSV files and create charts in the application, even some charts using R can be an option. I leave the ideas if someone wants to build something around it:

 

I'm glad that this tool will be open source and I hope the best for the future of this project.

[zetacu]

Would be cool if this tool can be linux compatible. Not an expert on writing this kind of software but I can help.

[TheTheos]

Feature request for cpu testing:

-island search and slicing in lychee

-detect and repair in uvtools

 

This is software that most sla 3d printing enthusiasts use and those processes are cpu intensive. TBH I have no idea if they can be proxied with some synthetic bench and how close that is to real life. So far I haven't seen those brought up in any cpu benchmarks. I would love to see in what times different cpus handle supporting, slicing and then repairing sliced models.

[T8or]

I appreciate graphing the data in grafana, however I would love to be able to get all the csvs as I use other, better graphing tools  

 

this is such a cool project! Love it. 

[ScrappeyDP]

From what I have been told this is just a way to automate benchmarks that were developed by others. 

Could you please clarify what this "MarkBench" is and how it functions so that we can better be able to address your request for feedback. 

Is this actually coming with its own benchmarking software? Or is it an automation tool for running other benchmark/stress test software? 

 

If it is the latter I would like to see stress test type software along with the benchmarks that give scores/results. When dealing with used hardware I find it is just as important to stress test components as much as it is to test in benchmarks and verify/validate the scores/results. 

 

I also find it useful to do comprehensive testing rather than relying on only a few tests. 

 

The bundle of software I would like to see included to run with automation (but not limited to):

 

3DMark

PCMark

Unigine - Heaven, Valley, Superposition

PassMark Performance Test

Novabench

Kombustor

Basemark

Geekbench

GFXbench

UserBenchmark

Cinebench - R15 and beyond

OCCT

Furmark

BurnInTest

AIDA64

Sandra SiSoftware

Prime95

LinX

Crystal DiskMark

Crystal DiskInfo

HWiNFO

CPU-Z

GPU-Z

HWMonitor

Speccy

Fraps

Afterburner

EVGA Precision

 

And easy to automate software that can log the data, screenshot results and produce graphs and comparisons would be ideal. "Set it, forget it and walk away until it's finished" is the goal. Run, log, move onto the next and repeat. 

 

 

[PrestonC]

You should test out DearPyGui for the GUI and graphs, it’s a python framework directly from C++ then you don’t have to use GO just for the gui 

[PrestonC]

1 minute ago, PrestonC said:

You should test out DearPyGui for the GUI and graphs, it’s a python framework directly from C++ then you don’t have to use GO just for the gui 

It has super simple live frame based graphs and you can control the frame rate of the graphs and gui to reduce load on the gpu by freezing the gui from updating 

[RareGriffin]

i dont know about anyone else but maybe a feature that would recommend different types of upgrades based on the stats system. For example after testing it tells you that you are not getting the full potential of your cpu because you are running only 16G ram in a single slot, or that based on cpu and ram that the next option to upgrade would be your gpu and give some that would better fit your system. because i will be honest knowing what GPU i should upgrade to out of Nvidia, AMD, Intel, ASUS, Apple, GIGABYTE, ZOTAC, EVGA, MSI, Sapphire gets really confusing. Just an idea

[PrestonC]

2 minutes ago, PrestonC said:

It has super simple live frame based graphs and you can control the frame rate of the graphs and gui to reduce load on the gpu by freezing the gui from updating 

DPG is also cross platform for Unix and apple 

[LofiLute]

Sounds like there’s some implied MacOS support. Which, Good.

 

But I’m hoping there are plans to support Linux. Proton has made it a viable platform but we still don’t have a lot of people doing real in depth research into performance. 

[Darth Tenebrea]

Something really cool would be if you could provide a website where everyone can upload their test data. So that you can search your build and compare your results or see what performance your future rig will have. With such a website it would be possible to see even the performance of very rare scenarios, e.g. a new high-end CPU with an old GPU or whatever someone is using.
 

[CaptainDarkstar42]

I would be interested to see if this could be applied to testing how multiple operating systems affect different programs and games with the same hardware. Like when the RTX 5090 comes out, it would be interesting to test how different OS' had different performances with apps and games on that hardware. Obviously, MarkBench couldn't run on those computers natively without reinstalling the OS and then running the scripts again, but it would be cool to see it run on another machine connected to the machine you are testing. I know that's an outlandish idea, but it would be cool to see!!

[GodofGrunts]

1 hour ago, AlphaS said:

Great tool! I really want it for Linux.

With the steam deck bringing Linux gaming to the regular public, I think more Linux coverage is going to be amazing. No one else is really doing it right now, so getting out ahead will really let you take the early market for it.

 

You can automate this too using [bcdedit](https://learn.microsoft.com/en-us/windows-hardware/manufacture/desktop/bcd-system-store-settings-for-uefi?view=windows-11) on Windows and [efibootmgr](https://www.linuxbabe.com/command-line/how-to-use-linux-efibootmgr-examples) in Linux to switch back and forth.

 

Linux also has the advantage of not needing to reinstall drivers since you can just disable the ones you aren't using in modprobe.

[redditor9800]

Make sure to add Linux support, and package management backends to smooth out issues install software.

[Upking]

It would be great to be able to toggle on/off video recording to measure the impact of NVENC and similars on the results of the benchmarks.

 

Also, Nvidia Broadcast and it's features would be a plus too

[Tarabustor]

Did you guys think about the difference an update will make in a game ? Testing a GPU on early game release like Cyberpunk with a 3080 will be messy because how badly optimized the game was at launch. When releasing a new gpu, say 3090ti, cyberpunk would have been patched up and testing the 3090ti on that game will result in the old data with the 3080 reflect poorly on the gpu, or rather, making the 3090ti looks better than it is 

 

So did you guys considered updating old data as well in your reviews when benchmarking games ?

[Frukas]

I got really happy this is made in Golang! I love this language and I have been studying this technology for years! Are you guys planning to share the GitHub of the project? Are you also planning to make a more deep view of the software for developers or students?

[Mousetwentytwo]

Collect data from people too, to increase sample sizes!
Is there something we can help with?

[1n4]

Hi! If you're interested in benchmarking machine learning performance, I'd go for 

1) Fine tune ResNet150 to test convolution training performance

2) Fine tune T5 to test transformer training performance

3) Exectute StableDiffusion to test inference performance and for sex appeal

[Euphoriann]

I would much appreciate apex legends being added to the supported games, still a lot of players who play that game and it can be a little difficult to run on some hardware so would be nice to see how some cards stack up. Thank you

[HunterAP]

I know lots of people would love to be able to use this tool to measure the performance of their systems and upload those results to a public database. My question is, what is the plan for validating the results, or that the project executable wasn't tampered with? I figure there would need to be some validation steps either locally or on the remote machine where the database is (assuming that there will be a public one at all) would be needed - any ideas so far?

[Mousetwentytwo]

Add Warzone to supported games please.
It is pretty though to satisfy even with very pricy hardware as its hungry for resources and easily bottlenecked by CPU.

How I usually try to benchmark is by watching GPU and CPU frame times in milisecs, good indicator of bottleneck, since both are parallel but the longer one will be effectively the real frame time. With a good GPU and a bad CPU some games feel unplayable and reducing graphics does not help it. Setting Turbo power limits on recent intel processors can help these times a lot according to my experience.

[GuilhermeBeco]

Hello everyone! First time here.

First, big kudos to the dev team for getting this to be a reality.

 

I have some suggestions, that i want to present in a timeline order.

 

1 - A way to add auto clicker scripts for mouse and keyboard, to test games without a benchmark. (Example: Valorant, where you would add a script to get into the firing range and another to move arround it)

2- Linux integration would be great, specially if we could provide a specific proton version

3- On the same topic, a possibility to run the benchmarks on different kernels (i know it envolves reboots and it could break an installation, but it would help new Linux users decide if they should invest the time into exploring if the best performing kernel is the right for them)

4- As people already mentioned above, a way for us to upload data into the database, labeled as "User submitted". This way, we could compare our results with LTT's results, but also with the community results.

5- Log patch versions of the games. This could not only detect improvements, but also save lots o trouble when trying to understando why the game does not perform the same as it used to

6-Also read this on other comments, but adding non game benchmarks (if possible, ones that can be executed with scripts)

7- A GUI to tweak the settings for the game

8- Before running a benchmark, based on the selected settings, it consults the database and calculates an expected results, based on hw specs, game and game settings (ML could be used here, i believe.

9- A framework where we can create our benchmarks for unscriptable tests. Let's say that blender bmw does not support that. Beeing able to record mouse keys with a python autoclicker lib and a way for us to "draw" a shape arround the parameter(s) that we want to use. (A big up would be to have some sort of community benchmarks tests that people could download, up and down vote leading to it even being used on LTT or being removed from the community page)

10- Based on the 8th point and with a good ammount of data, a ML implementation could, even before you start any benchmark, report a log with performance ranges for the various titles, for the most popular settings pressets.

 

Please forgive me for any error on my writting.

A rest of a good day to everyone

[Cdubs2209]

CITY BUILDERS!!! 

I'm not knowledgeable on this so idk what to say other than please showcase at least a chart of what's better for city builders.

[MatthewKhouzam]

Hi, I am a long time fan who just created an account to ask you to consider using Eclipse Trace Compass for visualization. This software is designed to debug stutters/janks, can go down to nanosecond resolution and works well on many data formats.

 

Here is the youtube channel for it. https://www.youtube.com/channel/UCN_j_soDvo1dLHEmhljmMzw

It is something we take great pride in working on. It is also 100% open source. I am not certain it's a perfect fit, but I do know we reproduced your NVME being too fast for threadripper Wendell bug from way back when.

 

You can get the software here http://www.eclipse.org/tracecompass, it will be web integrated soon(tm) also patches are welcome if you want to collaborate. This could lower your costs... but really, I like your work, I think you do computer engineering a major service and want to share in the joy.

 

edit: removed a comma, thanks MouseTwentyTwo!

 

Hope this helps.