EtaCarinae

Concerns regarding Epomaker’s sponsorship. 
 
hello! I’m a long time watcher, back in the NCIX days. I’ve been in the Mechanical Keyboard hobby since 2008 and currently moderate /r/mechanicalkeyboards. In the past I’ve been in direct contact with Plouffe via X formerly twitter, however I felt it pertinent to address this here. Epomaker was a recent sponsor on the scorpion chair. We felt that it was necessary to inform your team that Epomaker has a long standing ban on Reddit for quite a few anti consumer practices. I have a lot of respect for LTT and feel that it’s necessary to raise these concerns as Epomaker does not fit the mold for a quality sponsor. I’ve included links to previous Reddit posts regarding Epomaker that I feel is important. If you have any other questions, please feel to reach out here or through Reddit/Discord. My account is Cobertt on all platforms. 
 
https://www.reddit.com/r/MechanicalKeyboards/s/jzsX83iCsq
https://www.reddit.com/r/MechanicalKeyboards/s/AGHQvHk3Ea
https://www.reddit.com/r/MechanicalKeyboards/s/5Qs6euKi2h
https://www.reddit.com/r/MechanicalKeyboards/s/Wl6tXw9mLk
https://www.reddit.com/r/MechanicalKeyboards/s/zH7yFYMsax

Replacement of videos is a line for me. Despite the ability for someone to do them - they simply shouldn't. At bare minimum a correction should be put out visibly and the video removed. When someone can replace a video, and if I'm watching on Android TV, Fire, Roku, Smart TV etc. I have no idea if I am watching an original or replacement video. So unless part of the process would be to have in the first few seconds a VO stating that this video was replaced due to x, y, or z this does very little to repair any of the trust.

In all seriousness there are smaller Youtubers who do error correction better. They carry it out in a way that trounces this process. Look outside of the tech sphere and look at how other info-tainment manages to error correct. Don't do the thing of pretending that tech is it's only precious thing - it isn't. It's a theme, and you're a media production company if you like it or not. Look to other media production companies to see how they handle these processes.
 
See the following example:
Corrections and Clarifications « Tom Scott
 

Perhaps one thing that may also help catching errors proactively is to release the video a few hours (maybe a day) early on Floatplane and see if the public finds any inconsistencies, thus providing a safety margin before releasing the video to a much larger audience on Youtube. (Note: I am not a Floatplane subscriber)

The AI voice thing is ridiculous. Honestly, it's literally a 10 second section or something just get the host to record it again or delay the video. 

Internally this is a living document, and may change as we move forward, but as part of our push for further transparency, we are publishing it in its current form to help our community better-understand how we classify errors and what action we will be taking to rectify them.
 
Types of errors
Flubs: the host simply misspoke
Incongruencies: information within the video is not in agreement (e.g. host says “$45” while the product page shows “$47”
Bad spec: a table or MOGRT contains incorrect information (e.g. “48MB” instead of “96MB”
Bad data: a graph or visualization contains data (generated by us) that is erroneous, misplaced, or spurious. 
Bad info: a statement or representation in the video that is misleading or factually inaccurate
Types of error correction
Proactive (before the video is posted)
Video pickup: the segment is re-shot with both audio and video
Human audio pickup: new audio is recorded for the segment (visuals have coverage)
AI audio pickup: new voiceover is generated. In the best case this uses a voice model that sounds like the host. However, using a robot voice  can still work.
Text on screen: the editor overlays clarifying or correcting text
This will not be received by viewers who are not (or cannot) watching the screen
Reactive (after the video is posted)
Video Replace: replacing the video with a new version without re-uploading
This relies on YouTube and takes some time. There are fairly strict guidelines around the use of this tool and strong justification must be provided for all changes. This is preferred to a re-upload, but there is likely a soft-limit on how often we can use this resource.
Re-upload the video: Set the original, erroneous video to Private and upload a new version. This will have algorithmic effects, but must be done if replacement is not an option
Pinned comment: add a comment describing the correction. 
This will not be received by viewers who do not check the comments (common, especially for those watching on a Smart TV) and should only be used for low-severity errors.
 
Assessing & Responding to Errors
If any of the following types of errors are discovered prior to filming (e.g. during script review) , fix them before the shoot.
 
1. Very Low Severity
- The statement could possibly be misunderstood, but it’s generally true and most people would be fine with how it’s currently presented.
- eg. The host says, “One of DisplayPort’s main advantages over HDMI is its higher bandwidth,” but this is only true when comparing certain generations of the standards. HDMI 2.1, for example, has much higher bandwidth than DP 1.1.
 
Action: no action
 
2. Low Severity
- The statement is incorrect, and we should try to clarify it better in the future, but we can leave it for now because it doesn’t meaningfully impact a viewer’s purchase decision or general knowledge.
- eg. The host says, “This monitor features moderate pixel response times and contrast because it uses an IPS panel”, but the panel manufacturer is actually AUO, rather than LG, so it’s not an IPS panel - a term which is trademarked by LG - but rather an IPS-type or IPS-like panel.
 
Action: Pinned comment
Note: Pinned comment text must be approved by the Community Coordinator before being posted.
 
3. Medium Severity
- The statement is incorrect, and a minority of the audience could be upset or misled, even if it doesn’t really affect the outcome of the product evaluation for the majority.
- eg. The host says, “This GPU has AMD’s AMF video encoder, which is basically the same as NVIDIA’s NVENC,” but while the stated functionality is the same, NVENC offers better image quality, which could be important to a small minority of buyers.
 
Action: Video replaced if possible according to YouTube policy, otherwise pinned comment and above fold mention of the issue in the video description
If doing a video replace, a pinned comment should communicate this is coming and note the expected changes. This comment should be un-pinned when the video replace is complete. 
Note: Pinned comment text must be approved by the Community Coordinator before being posted.
 
4. High Severity
- The statement is incorrect, and a significant portion of the audience could be misled or make a poorly-informed purchase decision.
- eg. The host says, “This is a great gaming monitor,” when in fact, the monitor is a 60Hz VA display that, while ‘fine’ for playing games, lacks important gaming-centric features that could be easily had for the same price, making it a poor choice. 
 
Action: Video replaced if possible according to YouTube policy. Otherwise, set unlisted and disable monetization until corrective action can be taken. 
If doing a video replace, a pinned comment should communicate this is coming and note the expected changes. This comment should be un-pinned when the video replace is complete. 
Note: Pinned comment text must be approved by the Community Coordinator before being posted.
 
5. Very High Severity
- The statement is VERY incorrect—basically the opposite of correct. There is no charitable interpretation that could make this anything but completely irresponsible.
- ex. The host says, “Most games require only one high-performance core to perform their best,” when that hasn’t been true for over a decade.
 
Action: Video removed or remade

That's a text post not a video. 

I’m quite concerned that evidence is building that what Madison said is true.
 
Firstly, making accusations like this is not a case of type out some random crap, hit post and then put your feet up and watch the fireworks. You are, as the accuser, going to be hounded by EVERYONE. From fans who don’t believe you to media who want every detail they can get, no matter what you say it will never be *enough* evidence. Your social media is basically unusable for a good while, which is problematic if that’s what you do. You risk legal action, obviously, and it takes one lie to unravel for your story to fall apart.
 
Basically it’s a lot of shit to go through for what? The pain of half the world thinking you’re dirt for a vague hit back at a company you used to work for? It’s something that would sound easy to do at first thought, but if you actually sat down and considered doing it for real then there’s no way you’d do it voluntarily. Especially as there was a massively negative reaction when she tried to do it anonymously on glassdoor. Why do it again, but this time attach your name to it, if it’s just going to cause a huge negative reaction?
 
Secondly, frankly it could be worse. If you’re going to make up a sexual assault you may as well go the home run and say you were raped. In fact why the whole long thread? Just say ‘Linus raped me in a bathroom, all covered up, couldn’t report it to anyone as Yvonne was the HR manager’ - job done. She didn’t. She listed very specific examples of relatively low level behaviour and the highest ‘level’ she alleges is being groped. That could be explained away by saying that the person groping her did it accidentally or wasn’t aware she was uneasy etc… claim rape and there’s no explaining that away.
 
Third, when Madison left, there was a company wide meeting as previously posted. Now, I don’t think anything particularly wild was said in the meeting, and it has been posted by someone with an axe to grind, but the fact is that there was one. If Madison had made an obviously false allegation or said nothing on her way out, then there would be no reason to have the meeting after she left. This also contradicts Linus’s statement to the Verge today that he does not recognise any of the events or know of any such problems from her time at LMG. He is literally telling the company in this interview that something was reported.
 
Fourthly, an active LMG employee (David Gauthier) has been going through the twitter feed liking various parts of it. Why? Why would you do that, unless you agreed with what has been alleged? Liking those accusations, something which everybody (including your employer) can see, is pretty risky if it could cost you your job. How many more people agree with it, but can’t publicly put their name to it for fear of retribution?
 
As time goes on, Madison’s story is looking more and more confirmed.

have you never worked a job before?  How would you bring receipts to sexual harassment allegations?  How often you walk around your work place with a recorder?
 
Madison: quick colton repeat what you just told me but say it louder and into this audio recorder
Colton: Sure no problem. 
 
You must never have worked a job or you fried the last 2 brain cells you have up there. 

I understand the joke just fine, it was a SaaS joke turned into a innuendo joke by the way it was said and pointing at the camera.
The angry mob are the ones taking sides with LMG on this, LMG isn't your friend.
Making jokes in a serious video, especially innuendo is in bad taste for what should be a serious apology video.
Also here's the definition of innuendo, in case people don't know what that means.
 

When someone else said earlier in the thread about other women who have worked for LMG not having said anything negative, I was thinking about how Maxine didn't like to talk about LMG much at all after she left. Same way Madison didn't like being asked about what it was like working there (while on Twitch).

She cannot be silenced. She can be punished for her actions if they are deemed criminal or civilly damaging without truth.
 
LMG can respond to this. They will do so in way of the third-party company they suggested they'll bring in to check for any wrong doing.
 
Yes. It is workplace harassment and bullying. Who in the actual hell goes to the police for this? Maybe it's a cultural difference, but the company has a duty to provide a non toxic workplace which would be typically seen as civil not criminal. And yeah, I can totally see Madison bank rolling lawyers for this. If she had the forums would be going ape shit about getting lawyers involved and not trying to sort it out in private.

People asking why she waited until now and didn't go to the authorities. Look at the response she's getting. Look at how rabid Linus fans are tearing into her here and on Reddit. You really expected her to come forward before now and risk getting ripped apart by the cult of personality surrounding Linus? She's taking this opportunity to tell her story because there's already controversy around Linus and it might soften the blow she's going to get in response.
 
I mean, you don't have to believe her outright, but you don't have to rip her apart just because she's intruding on your parasocial relationship with Linus either. We've seen in the last few days how unethical and dishonest Linus has been, so I'm willing to at least give her the benefit of the doubt. Especially considering we even have it on record that many of the LTT employees have complained about being overworked, etc. 

Linus at the meeting:
What a weasel.

Making a sex joke during a meeting?  This particular meeting? 

Well I'm just gonna leave this here LTT needs to lawyer up 
 
 

I would read this, but this would cost me 100, 200, 3 00, maybe even $500 of my time. Sorry but I don't have time to read this half-assed clap back. Like why bother responding if you're not going to actually produce something useful to stand on?

so when Linus calls out sponsors (without directly contacting them first) and "scams" them to get product replacements, its ok, but when other channels call Linus out without contacting him first, its a problem. it's clear linus does not handle criticism handle well. GN doesn't owe LTT anything. 
 

Linus' response:
 

Buddy, you seem to be confusing things. The whole point is that this performance COULD HAVE been this much better even if it's expensive.
 
It's my money and I decide what to do with it. What I expect from a reviewer is to give me the most accurate picture of the product so then I CAN DECIDE whether it's worth my money or not - even if they think it's expensive.
 
The delightful irony is that Linus keeps hawking overpriced screwdrivers, water bottles and a $250 backpack, while whining about other companies.
 
It's like his unbearable whining about Nvidia. Look, I get that GPUs are expensive these days but, ultimately, I will be the judge if it makes sense for me to drop $1600 on a 4090. This is ENTIRELY subjective.

contacting linus personally, and resolving it quietly. is the exact opposite of "proper journalistic practices", what an absolute narcasistic arrogant buffoon

There won't be a big WAN Show segment about this or anything. Most of what I have to say, I've already said, and I've done so privately.

To Steve, I expressed my disappointment that he didn't go through proper journalistic practices in creating this piece. He has my email and number (along with numerous other members of our team) and could have asked me for context that may have proven to be valuable (like the fact that we didn't 'sell' the monoblock, but rather auctioned it for charity due to a miscommunication... AND the fact that while we haven't sent payment yet, we have already agreed to compensate Billet Labs for the cost of their prototype). There are other issues, but I've told him that I won't be drawn into a public sniping match over this and that I'll be continuing to move forward in good faith as part of 'Team Media'. When/if he's ready to do so again I'll be ready.

To my team (and my CEO's team, but realistically I was at the helm for all of these errors, so I need to own it), I stressed the importance of diligence in our work because there are so many eyes on us. We are going through some growing pains - we've been very public about them in the interest of transparency - and it's clear we have some work to do on internal processes and communication. We have already been doing a lot of work internally to clean up our processes, but these things take time. Rome wasn't built in a day, but that's no excuse for sloppiness.

Now, for my community, all I can say is the same things I always say. We know that we're not perfect. We wear our imperfection on our sleeves in the interest of ensuring that we stay accountable to you. But it's sad and unfortunate when this transparency gets warped into a bad thing. The Labs team is hard at work hard creating processes and tools to generate data that will benefit all consumers - a work in progress that is very much not done and that we've communicated needs to be treated as such. Do we have notes under some videos? Yes. Is it because we are striving for transparency/improvement? Yeah... What we're doing hasn't been in many years, if ever.. and we would make a much larger correction if the circumstances merited it. Listing the wrong amount of cache on a table for a CPU review is sloppy, but given that our conclusions are drawn based on our testing, not the spec sheet, it doesn't materially change the recommendation. That doesn't mean these things don't matter. We've set KPIs for our writing/labs team around accuracy, and we are continually installing new checks and balances to ensure that things continue to get better. If you haven't seen the improvement, frankly I wonder if you're really looking for it... The thoroughness that we managed on our last handful of GPU videos is getting really incredible given the limited time we have for these embargoes. I'm REALLY excited about what the future will hold.
 
With all of that said, I still disagree that the Billet Labs video (not the situation with the return, which I've already addressed above) is an 'accuracy' issue. It's more like I just read the room wrong. We COULD have re-tested it with perfect accuracy, but to do so PROPERLY - accounting for which cases it could be installed in (none) and which radiators it would be plumbed with (again... mystery) would have been impossible... and also didn't affect the conclusion of the video... OR SO I THOUGHT...
 
I wanted to evaluate it as a product, and as a product, IF it could manage to compete with the temperatures of the highest end blocks on the planet, it still wouldn't make sense to buy... so from my point of view, re-testing it and finding out that yes, it did in fact run cooler made no difference to the conclusion, so it didn't really make a difference.
 
Adam and I were talking about this today. He advocated for re-testing it regardless of how non-viable it was as a product at the time and I think he expressed really well today why it mattered. It was like making a video about a supercar. It doesn't mater if no one watching will buy it. They just wanna see it rip.  I missed that, but it wasn't because I didn't care about the consumer.. it was because I was so focused on how this product impacted a potential buyer. Either way, clearly my bad, but my intention was never to harm Billet Labs. I specifically called out their incredible machining skills because I wanted to see them create something with a viable market for it and was hoping others would appreciate the fineness of the craftsmanship even if the product was impractical. I still hope they move forward building something else because they obviously have talent and I've watched countless niche water cooling vendors come and go. It's an astonishingly unforgiving market.
 
Either way, I'm sorry I got the community's priorities mixed-up on this one, and that we didn't show the Billet in the best light. Our intention wasn't to hurt anyone. We wanted no one to buy it (because it's an egregious waste of money no matter what temps it runs at) and we wanted Billet to make something marketable (so they can, y'know, eat).
 
With all of this in mind, it saddens me how quickly the pitchforks were raised over this. It also comes across a touch hypocritical when some basic due diligence could have helped clarify much of it. I have a LONG history of meeting issues head on and I've never been afraid to answer questions, which lands me in hot water regularly, but helps keep me in tune with my peers and with the community. The only reason I can think of not to ask me is because my honest response might be inconvenient. 
 
We can test that... with this post. Will the "It was a mistake (a bad one, but a mistake) and they're taking care of it" reality manage to have the same reach? Let's see if anyone actually wants to know what happened. I hope so, but it's been disheartening seeing how many people were willing to jump on us here. Believe it or not, I'm a real person and so is the rest of my team. We are trying our best, and if what we were doing was easy, everyone would do it. Today sucks.
 
Thanks for reading this.

"stop it" couldn't have said it better myself. Please go and learn some proper enterprise architecture before you do this kind of project.   With 24 drives in a RAID, you've culled off most of your performance off the top. RAID is inherently NOT scalable with flash and with over 8 disks. With that many SSDs, you should have MILLIONS of IOPS. What you need is not a single server. For this scale of project what you need is AT LEAST three. Split up the drives (8 each), and make a cluster. With that many drives distributed on an RDMA network and using an appropriate VSAN you should have a very solid high availability storage cluster. Also, add some spinners. Yeah, I said it. With tiered storage you'll have some cold spots that can get put in the back corner on rotational media. With your scheme it's still a blown out UPS or a kernel exception away from disaster. With multiple machines, almost nothing short of a kiloton can knock your SAN offline.   What I'm saying here is that you need to read up on enterprise arch. Throwing youtube money on a bunch of expensive shit doesn't make you a sysadmin. Please call in some consultants and storage guys for this sort of project, because when you do this young techs see it and think it's the way to go. I'm not saying you need certs, but fuck man read a book.   Reading material: BTRFS documentation, Windows Storage Spaces. And if you want to go the cluster route, read up on Storage Spaces Direct, Gluster, CephFS, and the redhat vsan solution.   Seriously man, your entire business is going to be cached/stored on this box, make sure you get it right the first time.

I would recommend trying a low latency (1,000 Hz) kernel. The default generic kernel, in Ubuntu, is 250 Hz. From a performance standpoint 250 and even 100 Hz are better because the system has less "interruptions", but 1,000 Hz can probably offer better stability because it gives the system more opportunities to check in on the needs of the system as a whole. Also recommend enabling x2APIC mode and MSI-X, these will provide more interrupts for your system.
 
I strongly recommend ZFS (For my work I've done extensive comparison testing with Ext4 and XFS), but I must urge you to use the latest 0.8.x branch because it's way faster; in some cases it offers 2x the performance of the 0.7.0 branch. This is available in Ubuntu 19.10 natively, and via PPA in Ubuntu 16.04 and 18.04... https://launchpad.net/~jonathonf/+archive/ubuntu/zfs
 
In the video you said nothing about NUMA domains, you are most assuredly hitting inter-NUMA transfer bandwidth limitations. You need to ensure everything is pinned to the same NUMA domains as what the NVMe devices are attached to. You may be better off segregating all of the storage subsystem to a single NUMA domain.
 
For a ZFS tuning perspective 128 KB record size offers the best overall net performance gain (assuming a mixed workload), however, 64 KB is a really good choice too if you work with small files with random access patterns. The qcow2 format uses 64 KB as the default cluster size so that is also a good choice if the primary use case is VM storage. For 24 NVMe I would recommend 4x 6-disk raidz (or raidz2) vdevs, this will be nearly as fast as striped mirrors but offer a lot more useable capacity. Finding the right ashift value is not straightforward, it's best to simply performance test each value (i.g. 9, 12, 13, etc.). Use atime=off
 
zpool create -O recordsize=64k -O compression=on -O atime=off -o ashift=9 data
 
For benchmarking I recommend the following:
 
fio --randrepeat=1 --ioengine=libaio --direct=0 --gtod_reduce=1 --name=test --filename=test --bsrange=64k-192k --numjobs=16 --group_reporting=1 --random_distribution=zipf:0.5 --norandommap=1 --iodepth=24 --size=32G --rwmixread=50 --time_based=90 --runtime=90 --readwrite=randrw

https://blackember.com/collections/the-citadel-urban-backpack-collection/products/the-minimal-kit