Why is this PCIe Card RADIOACTIVE?

[power666]

Why worry about accurate packet time stamps generated within a machine when you can leverage determinism to know when within a precise range information will arrive at the far end before you send it?  This video gets so, so close as it shows the data sheet of the Intel 210 Ethernet controller at 10:13 regarding the hardware timestamp feature but skips over the AVB protocols for determinism listed right above that line item. 

 

For more accurate network time, there are devices that can sync to GPS  with their own atomic clock and then act as a IEEE 1588/802.1AS master clock.  The result is pretty much the same effect of the atomic clock card in the video but implemented at the network level that is shared.  Because it is shared now the result is lower cost per node.

 

The security aspects mentioned of knowing that a packet took to long to arrive is already part of the AVB specification due to the bounds on jitter.  The packet may not be discarded by default though as what happens in that scenario is configurable.  It can be discards as mentioned, flagged which can then be programmed for quarantine or reinserted properly into the data stream.  All valid choices depending on the task at hand and these policies can be applied on a per data stream basis. 

 

The benefits for streaming mentioned at 12:20 are real as well, I've using AVB.  The main benefit is that if you know what your maximum jitter is, you know what the maximum size of buffering is needed for any potential reorder.  The end point applications can display various stream issues but since the switch is involved, you can actually watch data stream performance at that level depending upon the switch itself.  I'm also fairly certain that LTT is already using some form of IEEE 1588 on their network given some of the production gear I've seen used.

[Jaesop]

50 minutes ago, ked913 said:

I am only eplying because I wish the content and the videos were better.

 

 Most of the use case is inspired from Google Spanner for use in async distributed databases. It is basically the Facebook equivalent of Google truetime.

 

They didn't even bother testing the open source claim with even statements that we could build and make it ourselves. They have the resources to prove that cclaim. Would be nice for that to happen in general rather than just checking for a github link.

 

Considering they don't have the vhd image for the FPGA I imagine that will go far.

 

Yea all those gamers are really gonna switch to Linux to use this.

 

Maybe he should have gone through the hackernews post as I think it would have been better than this vague speculative use cases regarding security and gaming.

This isn't a response to what I said, which is odd because my response was about how your prior response wasn't about what Alex said.

[Jaesop]

3 minutes ago, ked913 said:

I am trying to be constructive and move focussed on the topic.

 

Can you perhaps not be a pedantic fan boy trying to defend people who don't need defending?

 

You aren't contributing anything to the discussion at all apart from being a dick.

Posting responses that aren't related to what you're responding to isn't constructive nor focused.

 

And I am in no way being pedantic - I'm pointing out that your responses quite literally are not actually responding to what you're quoting. That also isn't being a fanboy - it's just straight up pointing out that you're essentially speaking in non sequiturs.

 

Also, I haven't been rude to you or resorted to name calling and I'd appreciate you doing the same.

[Jaesop]

28 minutes ago, ked913 said:

Fine, their 5 hour interview showed a shocking lack of basic knowledge on the subject in the areas I mentioned.

 

Knowledge areas that could have been fleshed out if they bothered to look further in the subject. Which is something I would have expected. Something that they could have done if they looked in the hacker news thread which is conveniently where I incorrect assumed they learned about the subject.

 

That contributed nothing to the topic of ptp and is extremely aggressive direct response which has done nothing to advance or constructively improve. Are you satisfied?

 

Happy now you useless pedant. You would be awful as a moderator please give up. I also don't see anyone else asking you to moderate how others reply. Probably for good reason.

 

Those other points aren't irrelevant tidbits, it is basic information that I would have expected to have learned from the video.

Yeah, you're just being childish and rude. And still not understanding what 'pedantic' means.

 

Have a good one.

[stevecam]

10 hours ago, ked913 said:

You guys need to do a better job before just skimming and copying ideas from hackernews.

 

The main use for these ptp cards is for distributed databases in where having precision databases are needed for better temporal guarantees and better performance. In a distributed system, transactions need to be delayed based against the precision of the timestamps in order to provide guarantees on the ordering of the transactions.

 

Other main use is for High Frequency Trading firms where it's important to record the precise timestamps of buying and selling for obvious audit purposes.

This is why the main player for these precision time cards have traditionally been smartnic vendors like Mellanox/Xilinx as it makes sense to bundle the two.

 

This is not something which everyone can just build, the most important aspect of the card is the firmware recorded on the Xilinx FPGA which is so far proprietary and not-publicized on github.

 

Also this is not something that is mature, I am not even sure how you would even use this on Windows which I would imagine would have way too much latency from the kernel (no smartnics for Windows im afraid).

 

I also don't quite think the software is mature just yet. The concept of NTS is just being rolled out, with patches not even merged with systemd just yet.

 

Oh wow, I went to the forum to find out more information about this and I would just like to thank you personally for informing me about this..... 

 

Well, to be honest it just looks like you pulled down your trousers and took a huge dump on your keyboard! Congratulations!

I mean, I am sure LTT looks at hackernews, staying relevant if important for any media organisation

 

I know coming across posts like yours is to be expected in any group but I have to call it out

You have become a tech-bully right here and shit like this does nothing but destroy a community

 

Did you feel any better after writing this post? I don't even feel like you are half as informative as the video published here

If you are depressed go talk to a counsellor or a psychologist, that's what they are there for, this is a tech forum

 

[RejZoR]

This is pointless. Big ass card with massive isotope thing and yet it does atomic time syncing still?

 

Wrist watches, the higher end quartz ones have high frequency thermo compensated quartz oscillators that keep accurate time within 1 second per year deviations (for comparison, basic quartz watches have deviation up to 15 seconds per month and non thermally compensated high frequency oscillators do like 5-10 seconds a year deviations). And they are tiny. If industry was really interested in doing it, they could just use this exact same tech used in these higher end wrist watches and still do atomic time synchronization we already do with most computers on top of it. This means if computer syncs atomic time every day just once, all systems should be running in ridiculously accurate timings compared to current quartz oscillators used on motherboards which I don't even think are even thermally compensated. And we all know temperature swings within cases can be pretty dramatic.

 

It would be a start if they only included this stuff on mainframe/server equipment in the beginning and later on home motherboards too.

[Egg-Roll]

At 45 seconds in the video when Alex hit the Linus plush that smiles at the end states it wasn't towards the hackers stuck on but likely the person the plush is made after 

I mean it's one way of taking frustration out towards your employer...

[Bicycle.Boi]

23 hours ago, AlexTheGreatish said:

Buddy, I spend 5 hours interviewing the founder of the Open Compute Project to make this, not skimming hackernews

I'm really not sure how a video confusing "open source" with "open hardware" gets past any final reviews if you conducted a 5 hour interview with the "founder" of the Open Compute Project.  Making a false claim in a video that "The entire project is open source" really isn't a great look for LTT.  The project is open hardware, but not entirely open source in its current state.  That's a major misstep in communication of the proper factual information to your audience that I would strongly urge LTT to correct.

[stevecam]

On 8/25/2021 at 3:01 AM, AlexTheGreatish said:

By using an Atomic Clock the clocks between different computers can be synced to within a dozen nanoseconds, and with that performance can sky rocket.

 

Check out the Open Compute Project: https://www.opencompute.org/

Build your own Time Card: https://github.com/opencomputeproject/Time-Appliance-Project

 

 

 

Imagine the potential this has for bringing back parallel interfaces in a big way, should also experiment with overclocking the GPU/CPU using this type of hardware also

[Forbidden Wafer]

On 8/25/2021 at 3:45 AM, RejZoR said:

Wrist watches, the higher end quartz ones have high frequency thermo compensated quartz oscillators that keep accurate time within 1 second per year deviations. And they are tiny. 

They are 100x less stable than atomic clocks. Look at the spec sheets.

[LAwLz]

On 8/24/2021 at 8:16 PM, MaxiTB said:

Hi, software dev here first time.

I am confused. Routing of IP packages is not fixed over WANs, the distance and nodes are not clear hence you can also not predict how long it takes for one package to arrive on the other endpoint. While in a LAN this might have some advantages (in closed networks) even there collisions can happen, causing delays or dropped packages. In other words no matter how precise your clocks are running, the road have different speed limits and you often don't know which one is used or if the road is blocked entirely. So most examples brought in the video are just not applicable, because in practice you won't ever use timestamps (oh god, that's what people did in the 80ties/90ties) but concurrency tokens in combination with CQRS.

Or what am I missing here?

Please note that I have not watched the video so I might be complete off base here, but it sounds like whatever the video is about involves syncing clocks over a network. If that's the case, it probably uses NTP or PTP.

 

NTP is used for synchronizing clocks over a network. It involves quite complicated maths and I don't fully understand the details of it myself.

1) The client creates an NTP packet and time stamps it just before it is sent out. (let's call this t0)

2) The NTP server adds a time stamp when the packet arrives to it. (let's call this t1)

3) The NTP server processes the packet and adds another time stamp just before the reply packet is sent. (let's call this t2)

4) The client receives the reply and adds a time stamp (let's call this t3)

Round trip delay = (t3 - t0) - (t2 -t1)

Time offset = ( (t1-t0) + (t2-t3) / 2 )

 

Do this a couple of times, maybe even against multiple servers, then apply some filters to the various results you get (for example remove outliers, estimate the average jitter, etc) and you can get the time accuracy down to within 1 ms.

 

Most of the magic happens in the client itself, not the data transmission part. If you want more info on how the various filters work then I recommend you look up how ntpd is implemented. Since it's a very good ntp client, and it is open source, you should be able to find plenty of good resources regarding it. Here is a good place to start:

https://www.eecis.udel.edu/~mills/ntp/html/filter.html

[MaxiTB]

Ah no, the video wasn't about synchronizing clocks, it was about the benefits of having synchronized clocks in various machines. And some very outlandish examples up to knowing when a video stream is up or detecting package spoofing

[Rex Hite]

I haven't seen the video and I can already tell it's great.

When bro-'splainers get their bro-'splaining 'splained back at them by agitated bro-'splainers you know it's a video destined for even more 'splaining.

 

 

 

 

 

[Forbidden Wafer]

3 hours ago, MaxiTB said:

Ah no, the video wasn't about synchronizing clocks, it was about the benefits of having synchronized clocks in various machines. And some very outlandish examples up to knowing when a video stream is up or detecting package spoofing

It is about synchronized clocks. There are many benefits. The examples weren't that bad, except for the streaming one which was complete nonsense. But you can detect package spoofing depending based on timings, but you need tight timings, stable/reliable/uncongested route, lucky enough not to get your package discarded randomly, etc. What you can certainly do is "this package is from the future (or from a distant past), discard automatically".

[Freeman]

This is one of the coolest videos you have done and I almost didn't click it because of the terrible title. Fortunately, it popped up in search results. I mean, you even went back to the video, according to metadata, you came back and added subtitles (which is great, it makes the video easily searchable), why not add at least "PTP" or "precise synchrnonization" to the title?

 

Ahmad and Alex, as well as the rest of the team, had made a great job explaining this, I've learned a lot and am glad to have found the video, but many more wont. I assume there is some reason why you don't just prepare more descriptive title and set like a 3 day reminder to change it, once the views have already spiked, I just can't fathom what it could be.

[geerlingguy]

While not wanting to wade into some of the more heated things discussed in this thread, I've been testing this card on a Raspberry Pi as well, and do agree that it's annoying the FPGA code is not open/available in the Time Appliance project codebase. I'll be asking about that and hope to have more news on it at some point.

 

I think the biggest thing that makes me interested in the card is the potential for use in applications beyond just data center time synchronization (NTP Stratum 1 and PTP grandmaster clock usage). SMPTE 2110 (for broadcast / live event timings) and new AES standards mean a cheaper box could help smaller studios integrate this timing equipment without breaking their budget. Same thing with DAC setups that need/want better timing but don't want to spend the $5K+ many devices currently cost.

 

Even though (currently) the software for the FPGA doesn't seem to be released, just having the hardware design out there means other people can remix it to their needs, too. Check out this interesting board, that integrates the same hardware into a small board for the Raspberry Pi Compute Module 4: 

 

It's sometimes more about the possibilities it unlocks than the immediate application. I know as a programmer the less I have to worry about time accuracy, the better. Time is hard. Ridiculously hard (and that's before even considering human factors like calendars, dates, and timezones!).

 

[geerlingguy]

And specifically regarding the open source FPGA software, the problem is there is some IP from NetTimeLogic that cannot be open sourced, and apparently the OCP is trying to build a version that does not include that IP. Check out this issue for details: https://github.com/opencomputeproject/Time-Appliance-Project/issues/17

[Forbidden Wafer]

1 hour ago, geerlingguy said:

I've been testing this card on a Raspberry Pi as well

What have you not been testing on the Pi?  xD
**Completely sidetracking**

Spoiler

The Pi Blade was super awesome, great video.
Watched all the NAS videos, but ended up with a Helios64 made by Kobol, which is flipping awesome.
Sadly they just announced that they're closing (not enough parts + price hikes = doom).

 

[geerlingguy]

On 9/1/2021 at 3:32 PM, Forbidden Wafer said:

What have you not been testing on the Pi?  xD
**Completely sidetracking**

  Hide contents

The Pi Blade was super awesome, great video.
Watched all the NAS videos, but ended up with a Helios64 made by Kobol, which is flipping awesome.
Sadly they just announced that they're closing (not enough parts + price hikes = doom).

 

Unfortunately, still can't get SCSI working on a Pi. Would love to plug in my old 5.25" disks and see if I can read the old files off them

[D C]

On 8/24/2021 at 11:01 AM, AlexTheGreatish said:

By using an Atomic Clock the clocks between different computers can be synced to within a dozen nanoseconds, and with that performance can sky rocket.

Could I please get the source for the 20ms delay added to every request claim? Google isn't returning any hits for ntp 20ms delay

[Forbidden Wafer]

2 hours ago, D C said:

Could I please get the source for the 20ms delay added to every request claim? Google isn't returning any hits for ntp 20ms delay

Nobody said every request has a 20 ms delay. I believe he said Facebook did that, but it probably was just an example they got from the FB people.
Also, you would never have to wait nowhere near 20 ms for most sites that have a proxy cache, which doesn't mean the back-end service doesn't have that kind of delay.

 

This part of their post probably represent better what they said:
 

Quote

However, can we trust these numbers? If ntpd reports that the time is off by 0.185 ms, is that accurate? The short answer is no. Server estimates the offset based on multiple timestamps in a packet. And the actual values are supposed to be within a 10x bigger window. In other words, a result saying time is off by 0.185 ms means it’s probably within +/-2 ms (so, 4 ms in total). However, our testing demonstrated that accuracy is generally within 10 ms with ntpd.

 

We had technical requirements for better precision. For example, multi-master databases translate microseconds and even nanoseconds of precision directly into their theoretical throughput. Another example for which moderate precision is required is logging — in order to match logs between nodes of a distributed system, milliseconds of precision is often required.

https://engineering.fb.com/2020/03/18/production-engineering/ntp-service/

 

[D C]

15 minutes ago, Forbidden Wafer said:

Nobody said every request has a 20 ms delay.

"08:27 It turns out that the most foolproof solution to this problem is to just add a massive 20 millisecond delay to every single request"

 

 

[Forbidden Wafer]

1 hour ago, D C said:

"08:27 It turns out that the most foolproof solution to this problem is to just add a massive 20 millisecond delay to every single request"

 

Pretty sure it's just hyperbole, but would fit their ±10ms NTP margin.

 

Even if they did that internally, which I doubt is the case and probably was a misunderstanding (everyone makes mistakes, so not really a problem), you wouldn't notice as you access cached stuff most of the time.

[D C]

I've since dove deep into it and the 100x throughput occurred back in 2020 when they switched from NTPd to Chrony. NTPd was about 10ms and as of 2020 became 1.6ms by switching to Chrony, 6.25x faster. Chrony is the same protocol that Amazon AWS has been using since 2017, Facebook was behind the curve by 3 years.

 

However, the Time Card is game changing and pushes Facebook, and anyone else that uses it, ahead of Amazon AWS. Time Cards now allow for PTP and for it to be reduced from Chrony's 1.6ms to 0.08ms for all servers that have the pcie card installed in it, 20x faster.

 

If all of their servers had the Time Card in them, would they see an additional 320x throughput increase?

[RustySpoon]

On 8/24/2021 at 2:16 PM, MaxiTB said:

Hi, software dev here first time.

I am confused. Routing of IP packages is not fixed over WANs, the distance and nodes are not clear hence you can also not predict how long it takes for one package to arrive on the other endpoint. While in a LAN this might have some advantages (in closed networks) even there collisions can happen, causing delays or dropped packages. In other words no matter how precise your clocks are running, the road have different speed limits and you often don't know which one is used or if the road is blocked entirely. So most examples brought in the video are just not applicable, because in practice you won't ever use timestamps (oh god, that's what people did in the 80ties/90ties) but concurrency tokens in combination with CQRS.

Or what am I missing here?

This is the precise problem Precision time protocol solves by providing a universal known highly accurate time stamp to the operating system that could then be used by any time aware application to time stamp anything form packets to log data.  Either having an internal GPS clock or a PTP Grand Master would be able to provide sub micro second accuracy.  At a point when you have system geographically diverse all trusting a known good time source the network traffic delay becomes a moot point because the data can be put in order by time stamp regardless of the location of the senders and receivers as long as they have an accurate known good source of time.

Put differently if routing was fixed and the delays were known a universal source of time would not be as important because the known and constant delay could be taken into account.