MarkBench Development and Feature Requests

[Mousetwentytwo]

3 minutes ago, MatthewKhouzam said:

Hi, I am a long time fan who just created an account to ask you to consider using Eclipse Trace Compass for visualization. This software is designed to debug stutters/janks, can go down to nanosecond resolution and works well on many data formats.

 

Here is the youtube channel for it. https://www.youtube.com/channel/UCN_j_soDvo1dLHEmhljmMzw

It is something we take great pride in working on. It is also 100% open source. I am not certain it's a perfect fit, but I do know we reproduced your NVME being too fast for threadripper Wendell bug from way back when.

 

You can get the software here http://www.eclipse.org/tracecompass, it will be web integrated soon(tm) also patches are welcome if you want to collaborate. This could lower your costs... but really, I like your work, I think you do computer engineering a major service and want to share in the joy.

 

Hope this helps.

Had a comma on the end of the link making it 404, here without: ihttps://www.eclipse.org/tracecompass/

 

[mattyice0224]

Once available to the public, make an option to have user scores uploaded to a public website. That website should contain the ability to search by game, ram space, os version, video card, or CPU. This would allow people to essentially know how powerful of a system they need for whatever game they're running.

 

Idea 2: create a tool users can install that allows them to add more games to the benchmark with LTT reviewing the update & pushing it to all users. This community is probably so good / smart that a small python script that users could run and add more games would greatly reduce development time for LTT while provding the community more games they can benchmark.

[nonae]

PresentMon only works on Windows. Use Mangohud or something like it. Mangohud works really well. It's a Vulkan Layer, that means it works in between the app and driver and can log every frame drawn. Your own layer could also log much more like latency, triangle count and in general complexity as well as the actual frame of the scene, to discover weather the game is at fault and unoptimized or the driver is broken. Sadly anti-cheat software doesn't like Layers.

 

Phoronix has a great tool that automates everything. But Phoronix sometimes misses lags.

 

LibreHardwareMonitor seems to be specialized in Windows too. It doesn't support Linux arm (Raspberry Pi) and riscv. If it doesn't work look how btop does it's measurements on Linux.

 

Some important notes when benchmarking  on Linux, note the kernel, userspace driver and glibc version.

[BobserLuck]

Suggestions:

* Make a framework/plugin system that would allow the community to develop their own tests and distribute them.

* Add tests that go beyond gaming such as AI model training (push those Cuda cores).

 

Questions:

* This is planned to be "Freeware" right? 

"Free for personal use"

* What would be the cost for commercial use cases?

* Would this be collecting data to then upload to LTTs service like other "free" benchmarking solutions?

[CotyTernes]

PLEASEPLZplzplzplzplzplz release a Linux version as well, even if you have to gimp it without the game functionality for a while.

[PurpleCodes]

I don't think Grafana is the best option for graphics fyi, its primarily made for live data to monitor systems. A much better solution would be to use D3 JS which allows you to have literally hundreds of types of graphs, from there you could use puppeteer to export it to an image, pdf or you could have live interactive graphs on the labs website.

[nk-c]

I didn't see people mention this or missed it.

Add an option to see if environment data is included. Ambient temperature, Open test system or case system. For obvious reasons, when you make this available for public, they would be running them in case I assume. Similarly, let users add some #custom options, like any popular mods if they are using any. Some of data needed by some fanboys, niche use-cases may be too exhausting for you to do yourself. So add APIs to let developers or modders make "mods" for your application. This way you keep it in-house, but still let us poke around.

 

[JordanLikesThermals]

For your tests, integrate an array of environmental sensors outside of the PC that you can monitor. There's many that are even accessible by USB. And yes, it's extra things to buy for us, but it's really quite useful data. 

Focus on the temperature differentials between the environment and the chips instead of just the actual high temperature on a chip. In the context of thermal throttling and varying climates, it would be more beneficial for a lot of us to have the differential temperature in these benchmarks and information on where the thermal throttling starts instead of just evidence that the tested setup is not actually thermal throttling. We all don't live and play games in a 68F (20C) air conditioned room. 

Maybe down the road, for your testing, move your benchmarks into an environmental chamber to test in rooms that reach 80F (27C) or higher. When PCs are experiencing power creep and running >750w, that can be an additional environmental load that will have significant effects on rooms temperature. Especially when you consider that most electric space heaters are 1200-1500w and meant to heat rooms much larger than a 150 ft² (14 m²).

[unijab]

3 hours ago, ToboRobot said:

A database of system scores so users can compare and validate their system performance.

what are you trying to put a end to that unnamed user benchmark website?

[zombykilr777]

I would like to be able to have a comparison tool that will allow me to put in older GPUs like 9 series or 10 series and compare to the latest releases. Same with CPUs. It is hard to find reviews and comparisons that really dig into how much performance difference there is between older hardware to newer hardware. I could see this being useful for people that do not upgrade often or use hardware as a depreciating asset for work.

[carloskay]

As a very small tech latin american site, this is for us very promising.

[ikjadoon]

4 hours ago, AdamFromLTT said:

 

We want to hear from you! What do you think of MarkBench? What features would you like to see?

 

 

Some features focused on "snappiness" that most users want, but are sometimes hard to quantify

• Home productivity: browser benchmarks (requires forcing a single browser version, though), large PowerPoint or Excel file manipulation. Not sexy but what a lot of us end up doing and it's quite CPU-bound
• Latency tests: frame time / mouse-to-display (for CPUs and GPUs), JavaScript/browser, multi-tasking, app startup
• Multi-tasking: three or four simultaneous applications + perhaps a foreground test of latency? Or a game's FPS? Will be useful for analyzing normal vs hybrid CPUs (big.LITTLE or P/E-cores)
• POST or boot to Windows Time: while pretty niche, with DDR5 training + larger CPU sockets + more CPU cores + feature-packed motherboards, boot times are increasing and it'd be nice to know who is optimizing and who's not giving a shit.

And, because battery life is so important:

• Perf-watt tests: these would be especially useful for laptops to help understand battery life

[Arika]

This thread has a lot of good suggestions...but also a lot of suggestions that are along the lines of 'this software should be actual magic!"

[Badger 2-3]

1 minute ago, Arika S said:

This thread has a lot of good suggestions...but also a lot of suggestions that are along the lines of 'this software should be actual magic!"

Fair, but... on the otherrr handdd - as the saying goes - "Any sufficiently advanced technology is indistinguishable from magic"

[ReaperCrewGreen]

A feature that I think would be great for personal testing and for LTT's internal testing is an heartbeat service that runs on a separate computer/server that would allow for checking to see if a computer crashes during a test (I.e. allowing to test overclocking profiles and unstable hardware while still being able to let it go automatically). This could be extended in the future to allow the test data to also be sent to to the server as the test is happening so if the test crashes it could still have the data. Additionally, you could setup a wake-on-LAN system to allow the tests to restart themselves should they fail.

 

This might be a bit beyond the scope of markbench, but accepting a display input and using computer vision and ML, you could also add some semi-automation of updating bios settings.

[Sparky64]

Are you going to make separate programs for things like CPU's, SSD/NVME, DRAMM? 
Or 
Are you going to try and eventually make this an all inclusive benchmarker.

[straight_stewie]

5 hours ago, AdamFromLTT said:

you can contribute by providing feedback and feature suggestions in this thread!

One big feature that I've never seen in another (free) kit would be test suite scripting.

That is to say, if I would like to compare runs of multiples of the same test to see score trends, I currently have to manually record information.

It would be nice to have a tool that would allow me to say something like:

for num_runs in range(0, 10):
  result = run_test()
  result.append_to_file(myPath)
  

Or whatever it might look like. 


This helps in finding relatively high, but still very stable, daily driver overclocks.

Results could include everything from benchmark scores to various machine temps as well.

[Bonquische]

I would hope that it would be able to recognize if I don't have a certain game and then not include that game in the test suite. I also hope that it works with a combo of games installed through Epic Games, Ubisoft Connect, and Steam.

[micah686]

Glad to see that you will be releasing this tool. Here are my suggestions so far:

• Instead of exporting as a .csv file, export as .json. Json is a lot more developer friendly, can support multiple data types (integers, decimals, strings, booleans,...). Using Json would also make it significantly more web friendly, as you would be able to easily format it to be pushed to a web REST API, or any other service. Not a lot of services use csv over json.
• Make a console/command line/CLI tool to perform the work instead of a GUI. You should be able to do something like

  ./MarkBenchCli.exe /run CSGOProfile.xml /run DOOMProfile.xml

  which would enable more automation from testing tools. The JSON data should be piped to Standard Output/STDOUT, so that other programs can listen for the data and automatically feed into another program/web tool/save to disk/...

• Related to the post above, make the GUI just interact with the console window behind the scenes. That way, people who want a nice GUI can use it to click through multiple elements, but programmers and those doing automation can use the command line app.

• Expose internal functions/allow overriding internal functions so that programmers would have better control over the internal data and structures that are being generated. This should have a flag so that people know methods have been overwritten

• On the outputted data (preferably json), it should include the build number and hash of the actual tool itself. That way, people can compare that version 1.2.3 with hash 7f7ed3a9... got values a,b,c, but version 1.2.4 got values d,e,f.

• The data should also include the build version (if applicable), and hash of the executables tested. That way, an update the improves/reduces performance will be easily filterable

• Have a database where users can submit their results, and potentially other tech reviewers(maybe a small badge indicating what tech reviewer published results).

• Add a few custom properties when generating the data for data that software can't detect, such as the tests being run on aircooled, watercooled, or LN2.

• Have an extensible framework for allowing for custom external tools to add more datapoints to the final output, so the data can be really fine-tuned

• When you release the software, release it on github.com, so the community can identify issues, add more features, and keep the software maintained

• When you release it as OSS, it would be nice if the license was a *GPL license, so that changes that someone makes would become public.

I'm sure there is a lot more, but these are my main suggestions. I think the most important parts are allowing arguments to be passed to the tool, and exporting the data as JSON.

[caosio]

use a link with android/iphone to make a noise test using the mic from this devices; is just and idea with many variables but maybe your team can do it.

[DariuszL]

First of all Hi!. Great project love the that you want to make it plublic. Base on suggestions posetd here, they are more suitable for multiple smaller programs (api's) then one giant monster. I had program a few end of line testers in python so don't make my mistake. Make it in in static, type safe language. Python is slow, very slow. But as a framework for fetching from one point to another it should be ok and user friendly. That rises a qustion, do you realy need a gui, Gui is hard it need's to be easy to folow and all of that nonsens, belive I KNOW. For what I saw you are on fueature implementation stage. Simple configuration file is easier and you can have precooked configuration files for most of the tests. Thanks for poiting to Libre Hardware Monitor I will give it a shot maybe write some api to use later.

[TraceBivens]

6 hours ago, AdamFromLTT said:

Presenting... MarkBench!

This is the home of MarkBench discussion for the time being! While this post is relatively empty right now, you can look forward to occasional development updates, answers to common questions and (eventually) a link to the software!

 

What the heck is MarkBench?
Wouldn't you like to know? And wouldn't we like to tell you! Check out our announcement video!

 

When will it be released?

When it's ready!
 

Is it free?

When it launches, MarkBench will be completely free. 

 

Can I contribute?

In the future? Maybe! In the now? Development will be done in-house. BUT, you can contribute by providing feedback and feature suggestions in this thread!

 

 

We want to hear from you! What do you think of MarkBench? What features would you like to see?

 

 

6 hours ago, AdamFromLTT said:

Presenting... MarkBench!

This is the home of MarkBench discussion for the time being! While this post is relatively empty right now, you can look forward to occasional development updates, answers to common questions and (eventually) a link to the software!

 

What the heck is MarkBench?
Wouldn't you like to know? And wouldn't we like to tell you! Check out our announcement video!

 

When will it be released?

When it's ready!
 

Is it free?

When it launches, MarkBench will be completely free. 

 

Can I contribute?

In the future? Maybe! In the now? Development will be done in-house. BUT, you can contribute by providing feedback and feature suggestions in this thread!

 

 

We want to hear from you! What do you think of MarkBench? What features would you like to see?

 

Seems like a cool idea. Personally I would have gone with one of the mature solutions from the scientific community like Xarray and stored the data locally in hdf5, but for this kind of data that’s probably overkill. If y’all aren’t familiar with it tho, you may find it helpful for visualization. It’s all just matplotlib under the hood of course but it’s got some nice convenience methods that I have found very helpful the last few years. 

[LunarJimmy]

A versus section comparing individual part to individual part with global ranking. Right now Userbenchmark.com is the main one the shows up with a simple Google search (ie. 3060 vs 3060 Ti) but their software is super sketch.

 

[koeglike]

You should add the feature to make harness ( I mean that editing existing harnesses to change a benchmarke settings) making a lot more easier by giving it a GUI or a text file that you can edit, I cow this is hard b cous different games have different types of APIs ( I don't know what they are called in games) , and to have save slots for these custom harnesses 

[Biohazard777]

On 10/13/2022 at 11:46 PM, sergiogd112 said:

It might be a good idea to include non-gaming related test. Such as Blender rendering, Ai training, encoding and decoding, etc. That way you may be able to test other components, such as CPUs

On 10/14/2022 at 12:48 AM, Joshua W. said:

I saw someone else suggest including non-gaming tests (video encoding, AI, etc.) in the software as well.

On 10/14/2022 at 2:24 AM, GuilhermeBeco said:

6-Also read this on other comments, but adding non game benchmarks (if possible, ones that can be executed with scripts)

On 10/14/2022 at 2:38 AM, BobserLuck said:

Add tests that go beyond gaming

5:02 in the video...

----------
 

I have only one suggestion for now: open source it... sooner rather than later (or never).