Ge

ne

rati

ve A

rt —

Ma

de

wit

h U

nit

y

Leveraging Ray Tracing Hardware Acceleration In UnityDigital Dragons ‘19

Anis Benyoub

Graphics Engineer@Auzaiffe

Replace with image

High Definition Render Pipeline (HDRP)

3

Scriptable Render Pipeline

Modern design

Designed for AAA productions

Physically Based Lighting

Book of the dead

https://github.com/Unity-Technologies/ScriptableRenderPipeline

Replace with image

HDRP

4

Harold HalibutThe Heretic

Replace with image

Motivation

5

Ge

ne

rati

ve A

rt —

Ma

de

wit

h U

nit

y

6

Replace with image

Ray Tracing Effects

7

Replace with image

Ray Tracing Effects

8

Left: real footage. Right rendered with Unity

Replace with image

Ray Tracing Effects

9

Ambient Occlusion

Replace with image

Ray Tracing Effects

10

Ambient Occlusion Indirect Diffuse (GI)

Replace with image

Ray Tracing Effects

11

Ambient Occlusion Indirect Diffuse

Recursive Tracing

Replace with image

Ray Tracing Effects

12

Ambient Occlusion Indirect Diffuse

Indirect Specular

(Reflections)

Recursive Tracing

Replace with image

Ray Tracing Effects

13

Ambient Occlusion Indirect Diffuse

Indirect Specular Stochastic Area Shadows

Recursive Tracing

Replace with image

Ray Tracing Effects

14

Ambient Occlusion Indirect Diffuse

Indirect Specular Stochastic Area Shadows

Recursive Tracing

Replace with image

Ray Tracing Effects

15

Ambient Occlusion Indirect Diffuse

Indirect Specular Stochastic Area Shadows

Recursive Tracing

Hybrid (Rasterization/Ray Tracing) Renderer

Replace with image

HDRP Render Graph

16

A wild digital dragon appears!

Replace with image

Render Graph - Deferred/Forward

17

HDRP’s hybrid deferred/forward render graph

Replace with image

Render Graph - Deferred/Forward

18

HDRP’s hybrid deferred/forward render graph

(blue are optional steps)

Replace with image

Render Graph - Deferred/Forward

19

Replace with image

Render Graph - Deferred/Forward

20

Replace with image

Render Graph - Deferred/Forward

21

Replace with image

Render Graph - Deferred/Forward

22

Replace with image

Render Graph - Deferred/Forward

23

Replace with image

Render Graph - Forward Only

24

HDRP’s forward only render graph

Replace with image

Cornell Box

25

Cornell Box in HDRP with stochastic area shadows and ray traced indirect diffuse

Replace with image

Cornell Box

26

Replace with image

Cornell Box

27

Replace with image

Classic Command List

28

Replace with image

Classic Command List

29

SSR, SSAO, Conact Shadows…

Replace with image

Classic Command List

30

Replace with image

Classic Command List

31

Prepass, GBuffer, Forward

Opaque, Forward Transparents, …

Replace with image

Classic Command List

32

Replace with image

Classic Command List

33

SSR, SSAO, Conact Shadows…

Replace with image

DXR API

34

Ray Tracing Acceleration

StructureRay Tracing Shader

Pipeline

Replace with image

Ray Tracing Acceleration Structure

35

Replace with image

Ray Tracing Acceleration Structure

36

BLAS

Replace with image

Hybrid Renderer Command List

37

Replace with image

Hybrid Renderer Command List

38

Replace with image

Hybrid Renderer Command List

39

Replace with image

Hybrid Renderer Command List

40

Replace with image

Missing Features

41

Replace with image

Missing Features

42

Replace with image

Missing Features

43

Replace with image

Missing Features

44

Replace with image

Missing Features

45

Replace with image

Missing Features

46

Replace with image

Missing Features

47

Replace with image

Surface vs Effect Type Shaders

48

Replace with image

Surface vs Effect Type Shaders

49

Replace with image

Frustum Culling vs Volume Culling

50

Replace with image

Frustum Culling vs Volume Culling

51

Replace with image

Camera Filtering vs Per Effect Filtering

52

Replace with image

Camera Filtering vs Per Effect Filtering

53

Replace with image

Tiled Lighting

54

Replace with image

Cluster Lighting

55

Replace with image

Render Graph - Deferred/Forward

56

HDRP’s hybrid deferred/forward render graph

Replace with image

Hybrid Render Graph

57

HDRP’s hybrid rasterization/ray tracing render graph

Replace with image

Hybrid Pipeline

58

● Blue solved using an integrator (path tracer, photon mapper, etc.)

● Needs to be rephrased for our hybrid approach

Replace with image

Hybrid Pipeline

59

● Red evaluated using rasterization (almost)

● Blue evaluated using rasterization or raytracing

Replace with image

Hybrid Pipeline

60

● Indirect specular is approximated as an Isotropic GGX lobe

● Indirect diffuse is approximates as a Lambert lobe

● Ambient occlusion affect indirect diffuse or both

Replace with image

Ray Generation

61

Depth Buffer

Replace with image

Ray Generation

62

Depth Buffer

Replace with image

Ray Generation

63

Depth Buffer

Replace with image

Ray Generation

64

Depth Buffer

Replace with image

Sampling Based Integration

65

● Our budget is 1spp per effect per frame (or less)

● We use spatio-temporal reprojection/accumulation to increase

the coverage

○ Spatial coverage improved by “Blue-noise Dithered

Sampling” [Georgiev and Fajardo 2016]

○ Temporal coverage improved by iterating

over a Sobol sequence

Replace with image

Sampling Based Integration

66

Scrambling Tile 256x256x2

Owen Scrambled Sobol Sequence256x1x4

Replace with image

Indirect Specular

67

Replace with image

Indirect Specular

68

Replace with image

Indirect Specular

69

*

Replace with image

Indirect Specular

70

*

Replace with image

Indirect Specular

71

● Split Sum approximation [Karis 2013] is useful for variance reduction

● Blue is precomputed

● Red evaluated with rasterization or ray tracing

*

Real Shading in Unreal Engine 4 (Brian Karis)

Replace with image

Indirect Specular

72

Small office scene rendered with ray traced indirect specular

Replace with image

Indirect Specular

73

Variable Smoothness

*

Ray traced indirect specular term

Replace with image

Indirect Specular

74

Smoothness = 0.0

Ray traced indirect specular term

Replace with image

Indirect Specular

75

Replace with image

Indirect Specular

76

Replace with image

Indirect Specular

77

Replace with image

Indirect Specular

78

Replace with image

Indirect Specular

79

Replace with image

Indirect Specular

80

38 ms for 1 spp

2080Ti @ 1920x1080

Replace with image

Indirect Specular

81

Replace with image

Indirect Specular

82

Replace with image

Indirect Specular

83

17 ms for 1 spp

38 ms for 1 spp

2080Ti @ 1920x1080

Replace with image

Indirect Specular

84

CS

Replace with image

Indirect Specular

85

CS

Replace with image

Indirect Specular

86

CS

Replace with image

Indirect Specular

87

CS

Replace with image

Indirect Specular

88

CSCS

Replace with image

Indirect Specular

89

8.74ms for 1 spp

17 ms for 1 spp

38 ms for 1 spp

2080Ti @ 1920x1080

Replace with image

Ray Binning

90

”It Just Works”: Ray-Traced Reflections in ’Battlefield V’Johannes Deligiannis Jan Schmid EA DICE

Replace with image

Ray Binning

91

Screen tiles of 32x32 pixels for binning

Replace with image

Ray Binning

92

Screen tiles of 32x32 pixels for binning

Generated rays in the unit sphere for a tile

Replace with image

Ray Binning

93

Octahedral space for ray direction binning

Generated rays in the unit sphere for a tile

Replace with image

Indirect Specular

94

Replace with image

Indirect Specular

95

CS

Replace with image

Indirect Specular

96

CS CS

Replace with image

Indirect Specular

97

CS CS

Replace with image

Indirect Specular

98

CS CS

Replace with image

Indirect Specular

99

CS CS

Replace with image

Indirect Specular

100

CSCS CS

Replace with image

Indirect Specular

101

6.8ms for 1 spp

8.74ms for 1 spp

17 ms for 1 spp

38 ms for 1 spp

2080Ti @ 1920x1080

Replace with image

Indirect Specular

102

Variable Smoothness (Min Smoothness = 0.6)

Replace with image

Indirect Specular

103

2.7 ms for 1 sppVariable Smoothness (Min Smoothness = 0.6)

2080Ti @ 1920x1080

Replace with image

Indirect Specular – Additional compromises

104

”It Just Works”: Ray-Traced Reflections in ’Battlefield V’Johannes Deligiannis Jan Schmid EA DICE

Variable rate tracing

Mixing with ray marching

Replace with image

Area Shadow

105

Replace with image

Area Shadow

106

Replace with image

Area Shadow

107

Sponza Scene - Reference

Replace with image

Area Shadow

108

Replace with image

Area Shadow

109

Replace with image

Area Shadow

110

Variance Reduction?

Replace with image

Area Shadow

111

Sponza Scene - LTC only

“Real-Time Polygonal-Light Shading with Linearly Transformed Cosines”Eric Heitz, Jonathan Dupuy, Stephen Hill and David Neubelt

Replace with image

Area Shadow

112

“Combining Analytic Direct Illumination and Stochastic Shadows”Eric Heitz, Stephen Hill and Morgan McGuire

Sponza Scene - LTC + Stochastic Shadow

Replace with image

Area Shadow - Visibility

113

Replace with image

114

Area Shadow - Stochastic

Replace with image

115

Area Shadow - Stochastic

Replace with image

116

Area Shadow - Stochastic

Replace with image

117

Area Shadow - Stochastic

Replace with image

118

Area Shadow

LTC

Un

Sn

Replace with image

119

Area Shadow - Stochastic

Replace with image

120

Area Shadow - Stochastic

CS CS

Replace with image

121

Area Shadow - Stochastic

12.8 ms for 4 spp

2080Ti @ 1920x1080

Replace with image

122

Area Shadow - Stochastic

CS CS

Replace with image

123

Area Shadow - Stochastic

Use simplestshader variant

CS CS

Replace with image

124

Area Shadow - Stochastic

9.9 ms for 4 spp

12.8 ms for 4 spp

2080Ti @ 1920x1080

Replace with image

125

Area Shadow - Stochastic

CS

Replace with image

126

Area Shadow - Stochastic

CS

Replace with image

127

Area Shadow - Stochastic

CS

Replace with image

128

Area Shadow - Stochastic

CS CS CS

Replace with image

129

Area Shadow - Stochastic

5 ms for 4 spp

9.9 ms for 4 spp

12.8 ms for 4 spp

2080Ti @ 1920x1080

Replace with image

Recursive tracing

130

Replace with image

Alpha Blend Transparency

131

Replace with image

Alpha Blend Transparency

132

Replace with image

Alpha Blend Transparency

133

Replace with image

Alpha Blend Transparency

134

Replace with image

Alpha Blend Transparency

135

Replace with image

Screen Space Refraction

136

Replace with image

Screen Space Refraction

137

??

Replace with image

Screen Space Refraction

138

Replace with image

Ray Traced Transparency

139

Replace with image

Ray Traced Transparency

140

Replace with image

Ray Traced Transparency

141

Replace with image

Ray Traced Transparency

142

Replace with image

Ray Traced Transparency

143

Replace with image

Ray Traced Transparency

144

Replace with image

Hybrid Transparent Render Graph

145

Replace with image

146

Takeaways

● Naive is possible but not viable performance wise

● Reduce variance with analytic approximation/precomputation

● Forward materials are a real problem

● Sometimes rasterization is good enough

Replace with image

147

Takeaways

● Bin rays when distribution requires it

● Avoid anyhit as much as possible

● Take advantage of compute shaders when possible

● Some rasterization paradigms are incompatible with ray tracing

Replace with image

148

Takeaways

● Manage resources lifetime to avoid render target allocation

● Temporal reprojection/accumulation is key for denoising

● For real time, ray budget is extremely tight

● It might not make your image prettier, simply more accurate

Replace with image

149

Future Improvements

● Mix ray traced effects with their rasterized variant

● Lower resolution effects and up-sample all effects

● Use ray tracing for non rendering applications

● Path tracer for reference

● Skinned meshes and particle support

Replace with image

150

Acknowledgement

Unity Graphics TeamSebastien Lagarde (Unity)Kate McFadden (Unity)Dany Ayoub (Unity)Alexandre Cepisul (L&S)Awen Couellan (L&S)Aymeric du Chéné (L&S)Mike Geig (Unity)Emmanuel Turquin (Unity)Laurent Harduin (Unity)Ionut Nedelcu (Unity)Arnaud Carré (Unity)Joel de Vahl (Unity)Tim Cooper (Unity)

Eric Heitz (Unity)Francesco Cifariello Ciardi (Unity)Antoine Lelievre (Unity)Cyril Jover (Unity)Lewis Jordan (Unity)Natasha Tatarchuk (Unity)Jesper Mortensen (Unity)Tian Ning (Unity)Melissa Chou (Unity)

Replace with image

151

Questions

https://github.com/Unity-Technologies/ScriptableRenderPipeline

Replace with imageAdditional materials

152

Replace with image

Indirect Specular

153

Replace with image

Indirect Specular

154

Replace with image

155

Area Shadow

LTC

Un

Sn

Replace with image

156

Area Shadow

LTC

Un

Sn

denoise(Un)

denoise(Sn)

Replace with image

157

Area Shadow

denoise(Un)

denoise(Sn)

denoise(Sn) / denoise(Un)

Replace with image

158

Area Shadow

denoise(Sn) / denoise(Un) LTC LTC + Stochastic Shadow