Lies, damned lies, and latency — Behind Databento’s NTP time service

Shortly after Databento launched its real-time market data, we had a big problem: Some customers were reporting negative latencies when they diff’ed against our outbound timestamps. Something they’ve never seen before.

Latency in the cloud

On the feed and execution software side of the trading world, most will market median latencies; some will share their 90/95/99 tails, and even fewer will distinguish between wire-to-wire, socket-to-socket, and port-to-port. But you can trust that ISVs will always describe some methodology behind their latency measurements and how to replicate it.

That’s not the case with normalized feed/execution or any cloud-based solution. Most of our competitors don’t provide latency benchmarks. The few who do will measure some unknown part of their internal stack and make up some number that’s optimistic and useless.

This is usually benign, which is why they get away with it. Do milliseconds of error matter if you’re already taking a WAN hop?

One scenario where it matters is if you’re trying to match book updates (e.g. BBO) to trade; your application logic can get thrown off if trades are inexplicably printing outside of your BBO.

This is a more common problem where there’s actually seconds of latency through the vendor’s stack. This seems hard with the speed of even the most bloated of tech stacks, but is quite possible around bar aggregates because they’re artificially forced to print at the same time, which results in a huge burst when there are many symbols. It’s also possible due to TCP congestion control. We’ve seen vendors who will claim “1 ms of latency” while their customers are seeing tens of seconds of delayed OHLC aggregates.

Negative latencies

Back to the negative latencies, this was a problem of our own making.

1. Our feed was so fast compared to our competitors that this was the first that our customers experienced NTP sync error larger than the actual feed delay.
2. We provide our own outbound timestamps, so our customers can actually measure our feed latency and see if we’re stale. This matters because the exchange itself can be stale — a topic for another day. Other vendors just supply the exchange’s native timestamps and call it a day.

Typical event on Databento includes three timestamps, ts_event, ts_recv, and ts_in_delta.

Databento live client: Enabling ts_out informs the Databento live gateway to provide the fourth timestamp.

Databento public time service

So that was how our time service was born. It syncs to our PTP source and is the only NTP service based out of primary trading colos. You can ping the DNS names in the pool to get RTT to those data centers, which have few pingable public IPs. It’s free.

Learn how to add it to your NTP sources from our documentation.