Ambitious hacker reduces worst-case memory latency by up to 93%, but with severe downsides — 1960s bottleneck overcome by hedging memory accesses to avoid running into DRAM refresh stalls

The project logo for TailSlayer, the software described in the article.
(Image credit: LaurieWired)
10

A method devised by YouTuber, Googler, and security researcher LaurieWired could have huge implications for a very few specific use cases and workloads that are highly sensitive to "tail latency," or near-worst-case memory access latency. The project is called TailSlayer, and fundamentally, leverages hedging memory accesses to avoid running into DRAM refresh stalls.

Without getting fully into the weeds, the type of memory we all use, DRAM, has one serious downside: it has to be refreshed constantly. The cells where DRAM stores its data are fundamentally tiny capacitors, and they are highly leaky by design, so we have to continually top up the charge in them to make sure that they retain their data. This is known as DRAM Refresh. The refresh cycle happens at a rate that varies widely depending on the system and type of DRAM in question, but generally it's going to happen at an interval that is measured in microseconds, meaning that your memory is refreshing hundreds of times in the time it takes you to blink.

Latest Videos FromTom's Hardware
Zak Killian
Contributor

Zak is a freelance contributor to Tom's Hardware with decades of PC benchmarking experience who has also written for HotHardware and The Tech Report. A modern-day Renaissance man, he may not be an expert on anything, but he knows just a little about nearly everything.

  • bit_user
    This is clever.

    The article said:
    You're trading memory capacity and CPU cores for latency determinism, as this effectively multiplies memory requirements for any given application by a factor of the number of hedges you're willing to make.
    Not just memory capacity, but also bandwidth! I'd argue that's the limited quantity being burned by this technique much more than cores. Modern server CPUs have oodles of cores, but they have just ~12 memory channels. So, if you tie up 12 cores, in a race to see which achieves the lowest latency, it might only be ~10% of the total core count (e.g. on a 128-core CPU), but you could be burning close to 100% of the memory bandwidth! Moreover, it would be self-defeating to try and have those other cores doing very much that would compete for that memory bandwidth, so most of them will need to be kept fairly idle.

    Basically, you're sacrificing scalability for the sake of reducing latency. In mostly-serial, realtime applications, that might be a worthwhile tradeoff. But, where you can effectively parallelize your workload - even in latency-sensitive applications - it should always be preferable to do so!

    I think this point is worth clearly spelling out: this technique effectively turns a modern, multi-core, multi-channel CPU into a single-threaded machine! That said, if you don't need the absolute minimum latency, you could regain a little parallelism, but the most distinct threads you can run is no more than half of the memory channels. Quite a severe tradeoff!
    Reply
  • bit_user
    I hope DDR6 has more granular self-refresh. That would go a long ways towards both minimizing collisions and reducing the amount of time they take, when they do happen.
    Reply
  • IntelUser2000
    bit_user said:
    I hope DDR6 has more granular self-refresh. That would go a long ways towards both minimizing collisions and reducing the amount of time they take, when they do happen.
    That would require a significant architectural change on the DRAM chips itself, since self-refresh is fundamental to the operation of memory.

    If just using another core reduces latency that much though, essentially solving the problem, I doubt the hardware "fix" would happen.
    Reply
  • davidjkay
    This seems like a flaw that doesn't need to exist... dram refresh is a read/write cycle. So a cache that reads the dram and writes it back and counts as a cache hit... the cache also answers cpu call when cpu accesses same ram would turn these worst case scenarios into best case
    Reply
  • davidjkay
    davidjkay said:
    This seems like a flaw that doesn't need to exist... dram refresh is a read/write cycle. So a cache that reads the dram and writes it back and counts as a cache hit... the cache also answers cpu call when cpu accesses same ram would turn these worst case scenarios into best case
    In other words turn a refresh collision into a cache hit with a special line of cache
    Reply
  • bit_user
    IntelUser2000 said:
    If just using another core reduces latency that much though, essentially solving the problem, I doubt the hardware "fix" would happen.
    Like I said, this isn't merely tying up another core. It's burning a multiple of the memory bandwidth and you have to basically idle the rest of the cores, so that they don't fill up the transaction queues in the SoC and the memory controller. In other words, it basically reverts your CPU to a single core, like we had more than 20 years ago.
    Reply
  • bit_user
    davidjkay said:
    dram refresh is a read/write cycle. So a cache that reads the dram and writes it back and counts as a cache hit... the cache also answers cpu call when cpu accesses same ram would turn these worst case scenarios into best case
    I had a similar thought: why can't a simple read be used to drive a row refresh, if it happens to come in at the right time? Then again, my understanding of DRAM is fairly simplistic. I think probably the issue with that is that refreshes happen at a granularity that's far too high for that to work. I also suspect there could be some parallelism happening to make refreshes faster, that would interfere with single cell access.

    davidjkay said:
    In other words turn a refresh collision into a cache hit with a special line of cache
    First of all, this can only work if refreshes are granular and happen concurrently with accesses to other parts of the chip. Otherwise, the cache would have to be as large as the entire DRAM chip, which is clearly infeasible. If those preconditions were true, then refreshes would be far less intrusive than they currently are, and that much more ignorable. However, if you did also bolt on a cache to hold values that have just been refreshed, then it only solves half of the problem, because you're still stuck waiting to read cells that haven't yet been refreshed, or writes to cells that have already been refreshed.
    Reply
  • bit_user
    I wonder whether GDDR6 or GDDR7 memory does refreshes any differently than DDR5? I think LPDDR and DDR memory is basically designed around optimizing GB/$, whereas GDDR memory might make more concessions in area-efficiency to enable higher performance.
    Reply
  • N6543
    This is a cool idea, but the article is flawed. The title is wrong, for a start: the technique doesn't reduce worst-case latency, despite greatly reducing p99. 99 latency.
    It's also wrong in my view to say it is "trading memory capacity and CPU cores for latency determinism", since the behaviour remains probabilistic.
    Reply
  • IntelUser2000
    bit_user said:
    I had a similar thought: why can't a simple read be used to drive a row refresh, if it happens to come in at the right time? Then again, my understanding of DRAM is fairly simplistic. I think probably the issue with that is that refreshes happen at a granularity that's far too high for that to work. I also suspect there could be some parallelism happening to make refreshes faster, that would interfere with single cell access.
    It's not for parallelism, but to simplify manufacturing. Because if you had the ability to read/write control every single bit, then you increase component count(thus die size) substantially.
    Reply