As part of its series of revelations and announcements at GDC 2022, the green team today revealed Nvidia Streamline. In brief, this is a framework that Nvidia has devised to ease the integration of super-resolution technologies into games. Thankfully, Streamline is both open-source and can accommodate super-resolution technologies from diverse hardware and game engine vendors. Intel is already on board, but we have no word from AMD on its plans.
The rate at which super-resolution technologies are advancing and proliferating may be mind boggling for users. If you feel that way, share a little empathy with game developers who may have to dance around coding and recoding their games to work smoothly and be optimized for the likes of DLSS (Deep Learning Super Sampling), NIS (Nvidia Image Scaling), FSR (AMD FidelityFX Super Resolution), XeSS (Intel's Xe Super Sampling), potential updates to these techniques, and other options besides.
It is generally felt that the best place to hook in an advanced super-resolution technology is within the graphics pipeline, before the final screen render, so that in-game imagery is scaled using the various techniques but graphically-light elements like the common-or-garden HUD isn't, so it remains pixel-sharp. Some solutions work at the end of the pipeline, so they can work without any significant game developer support (or don't need any dev support at all) but they are often viewed as sub-optimal compromises.
Nvidia Streamline has been designed so that developers have an easy plug and play framework between the game and render API. As the creator of Streamline, Nvidia already has DLSS and DLAA plugins ready, and it is preparing an NIS plugin, as well as a real-time denoiser filter.
"Instead of manually integrating each SDK, developers simply identify which resources (motion vectors, depth, etc.) are required for the target super-resolution plug-ins and then set where they want the plug-ins to run in their graphics pipeline," explains Nvidia. "Making multiple technologies easier for developers to integrate, Streamline benefits gamers with more technologies in more games."
Adding some weight to the value of its work, at this early stage, Nvidia already has Intel onboard. In the diagram above you can see Intel plugins are being prepared for Streamline, and this will enable Intel XeSS with just a modicum of effort from devs who have the framework ready, especially compared to pre-Streamline days. Last but not least, Nvidia has left the door open to "Hardware Vendor #3," (aka, He Who Must Not Be Named, aka AMD) to play ball and produce a plugin, to speed integration of their super-resolution technology alongside the green and blue team choices.
The above open-sourced framework sounds like an appealing solution for the games development industry. If it gains mass adoption it might prove a significant gain for Nvidia, which will almost certainly get DLSS hooked into all the games that use this framework. Nvidia has quite an established lead with image upscaling tech adoption at this time, and Streamline isn't going to hurt it. In some ways Streamline could make sure DLSS will never be usurped by rival tech, especially more open ones which may have bloomed separately later in 2022.
On the other hand, Streamline might just help everyone and ensure users get their pick of the litter when it comes to super resolution techniques. AMD's FSR is already open-source, so even if AMD doesn't directly support Streamline, someone in the OSS community could create the required plugin. Intel's XeSS is a bit of a latecomer, but any game using Streamline for DLSS could easily add XeSS support as well. Why not, since it's simply a matter of putting in a few hooks?
"Intel believes strongly in the power of open interfaces," said Andre Bremer, VP of AXG and director of game engineering at Intel. "We are excited to support Streamline, an open, cross-IHV framework for new graphics effects. This will simplify game developers’ integration efforts and accelerate the adoption of new technology."
Nvidia says that Streamline is available today on GitHub supporting both DLSS and DLAA (Deep Learning Anti-Aliasing). NIS support is "coming soon." The framework is extensible beyond super-samplers, as evidenced by Nvidia including its Real-time Denoiser. Please note that Streamline supports DirectX 11 and DirectX 12 titles, but Vulkan compatibility is still in the beta testing stages.
