Level of detail is simply astounding—realtime global illumination and more.

(Source: Epic)

I have been trying to write about Epic’s new Unreal Engine 5 since they announced it in May. Every time I do, I am awestruck by the visuals, and mind-boggled trying to figure out how they did it. It is making a bazillion zillion quadrillion polygons fit in a 4k screen. The company said at the time that one of their goals for the new engine was to achieve photorealism on par with movie CG and real life. They also wanted to be accessible to development teams of all sizes with productive tools and content libraries. I think they’ve accomplished that mission.

To demonstrate the power of this new engine, power, in this case, is measured in terms of fine detail—enabled through an order of magnitude more polygons than pixels—the Unreal team created a short 9-minute movie, Lumen in the Land of Nanite, which you can (and should) watch here. The demo was originally made for the PlayStation 5 (more on that later).

The visuals are stunning and made with what Epic is calling Nanite virtualized micro-polygon geometry. This new LOD geometry, says the company, will free artists to create as much polygon detail as the eye can see—that may be an understatement. Nanite virtualized geometry means that film-quality source art comprising hundreds of millions or billions of polygons can be imported directly into Unreal Engine—from ZBrush sculpts to photogrammetry scans to CAD data. Nanite geometry, says Epic, is streamed and scaled in realtime so there are no more polygon count budgets, polygon memory budgets, or draw count budgets; there is no need to bake details to normal maps or manually author LODs, and there is no loss in quality.

One of the keys is dynamic global illumination in the system they call, Lumen. GI is the challenge in realtime graphics, there’s just so many things to deal with—at once.

Virtualized geometry allows the composition or partition geometry as well as creates new virtual geometric entities. Virtual Geometry operations are most often used as a tool for adjusting the geometry to allow mapping, sub-mapping, or sweeping mesh generation schemes to be applied. That capability improves workflow efficiencies and lets an artist do art and not IT chores cleaning up a highly detailed (high-quality) image to fit within your geometry budget.

Epic’s VP of Engineering Nick Penwarden says the demo hit 1440p 30 FPS on the PlayStation 5 with dynamic resolution scaling enabled—the sweet spot for gaming resolution is between 1080p and 4K.

A million triangles each in 8k textures—a billion triangles of source geometry in each frame and the engine reduces it to 20 million triangles losslessly.

If you have 20 million triangles and color code as Epic did in the previous screen, and you have an 8-bit display (not HDR) you have 16 million color possibilities, which means there more triangles in the above image than can be displayed with a unique color. It takes time for that to sink in. Now, to just make it more mind-warping assume you have a 4k screen—that’s only 8.3 million pixels so there are 2.4 triangles per pixel. The LOD exceeds display capability which means some type of weighting or decision has to be made to cull down the choices and use it—and that’s part of Epic’s secret sauce.

And then there are shadows. Mind you there’s been no discussion yet of ray tracing. So how Epic gets shadow data out of the billion or so triangles it started with is more magic. Epic says all the lighting is dynamic and includes multi-bounce illumination without any lightmaps or baking. It is a genuine global illumination calculation—in realtime.

Epic didn’t only take pixels into consideration, they also developed a new audio engine and used sampled sounds (of rocks falling in the example they show in their video) to get realistic immersive sound, which they are calling sound field rendering.

The UE5 also incorporates fluid dynamics and objects for NPCs communications. UE5 also has the Chaos physics system used for rock dynamics in the demo and the character’s hair or scarf.

Epic also reworked its animation system so human and animal bodies move and look natural. The added predictive foot placement and motion-warping are used to dynamically modify inverse kinematics (IK) and body position. They combine that with a procedural AI animation to allow the character to react to its environment. In the demo explanation, you can see the character reach out to hold the edge of a door that’s partially open—that is pretty fantastic, and her hand touches it at the right surface, fluidly and smoothly.

Lumen immediately reacts to scene and light changes. The system renders diffuse interreflection with infinite bounces and indirect specular reflections in huge, detailed environments, at scales ranging from kilometers to millimeters. This will allow artists and designers to create more dynamic scenes using Lumen says Epic, for example, changing the sun angle for time of day, turning on a flashlight, or blowing a hole in the ceiling, and indirect lighting will adapt accordingly. Lumen erases the need to wait for lightmap bakes to finish and to author lightmap UVs—huge time savings for artists. Now, an artist can move a light inside the Unreal Editor and lighting looks the same as when the game is run on the console.

Epic has employed GI very effectively and gets a kind of HDR effect in dark places. It also gives a specular effect on surfaces adding to the overall realism and believability. They even set up a reactive capability, illustrated in the demo where bugs react (run away) from the character’s flashlight. Really clever stuff and all in realtime, truly amazing.

Within a scene, the hero comes across a large statue of a soldier standing guard. Epic says they imported it directly from Z-Brush and it alone is over 30 million triangles (in a 2.3 million pixel frame).

The soldiers have no LOD parameters or normal maps (such as one would use with bump-mapping). This gives artists an almost infinite detail to work with, it makes me think of fractals where you can just keep digging in and in and there’s more detail. But Epic didn’t stop with just the one big statue, they replicated it all over the room which amounted to over 16 billion triangles. One of the demo’s reveals lights up rows of statues like terracotta warriors. Staggering numbers and really difficult to grasp. Epic says that in the overall model there are hundreds to billions of triangles. That’s a lot of sorting.

The demo looks like a big scale movie that was rendered one frame an hour, not 30 times a second.

Doing a demo for (and on) a specific (dedicated) type of hardware is an easier job than doing one for a general-purpose class of hardware with various associative drivers, OS versions, and memory speeds and capacities. The demo is tuned to the PS5, and one of the features of the PS5 is it’s very large, and very fast SSD memory structure, Epic has made full use of it.

According to comments made on Tweaktown, the demo required intense horsepower for its Nanite system to render the billions of triangles needed for the scenes and wasn’t able to hit 60 FPS on the PlayStation 5’s 10TFLOP Navi GPU. It also required dynamic resolution scaling, which adjusts resolution on-the-fly to lock in a consistent frame rate. For reference, most of the PS4 Pro’s 4K games have dynamic resolution scaling and few are native 4K.

“Nanite works very well with our dynamic resolution technique. When the GPU load gets high we can lower the screen resolution a bit and then we can adapt to that. In the demo, we actually did use dynamic resolution, although it ends up rendering at about 1440p most of the time,” Penwarden told Eurogamer.

Epic will be offering more technical details over time and I can’t wait to see it. This is the most amazing thing in CG I’ve seen in a very long time.

Looking at the results one might ask does this make ray tracing obsolete? I asked a few of my friends about it.

Steven Parker, Mr. Ray Tracing at Nvidia said, “Ray tracing is great for high polygon count. And secondary illumination still matters, where RT is invaluable.”

Brian Savery, Mr. Ray Tracing at AMD, told me, “Much more so. The difference is that pure ray tracing is more dependent on the number of pixels times the number of ray samples. Rasterization techniques are more dependent on the number of polygons. Having these “magic” LOD’s is very nice with ray tracing as they fit well with BVH acceleration structures and diffuse rays, etc.  That is, if you have a sharp reflection you might trace against a “fine” LOD vs a rough reflection or shadow might trace against the “coarse” LOD.”

David Laur, Mr. Ray Tracing at Pixar said, “In fact, this situation can be when ray tracing is most relevant. The cost of shading is the critical metric. A good ray tracer will only shade one or a couple of hit points on tiny triangles that are visible and won’t run expensive shading at all on parts of triangles that are hidden. Most scanline/gl style renderers will need to fully shade every triangle no matter what, then z-buffer it in, whereupon it might be totally hidden, or all smashed into a subpixel. In the case of a triangle much smaller than a pixel, the ray tracer can also adjust things like texture mipmap level lookups based on incident ray-cone footprint size, e.g. when far away or after a couple of bounces off of curved car body surfaces, tree leaf textures can be accessed at their coarsest level, but those same nearby leaves seen directly from the camera would shade in finer detail.”

And Tim Sweeney, the boss (and founder) of Epic, told me, “Certainly fixed-function rasterization hardware will be obsolete in less than a decade. What replaces it is a mix of raytracing in the traditional sense, and new hybrid compute shader algorithms that map pixels to primitives and shader inputs using a variety of algorithms that are optimized to particular data representations, geometry densities, and static versus dynamic tradeoffs.”

And last but not least, if you’d like an authoritative and somewhat detailed explanation of Epic’s demo, check out this video.