Path-tracing with Nonlinear projections (for wider FOVs)

[shaunlebron]

Congrats on the project 🎉 I wanted to add that path-tracing isn't just the holy grail for lighting—it also frees the renderer to use different projections for showing wider FOVs without terrible distortion:

• I explored this in Quake, but only using the software renderer.
• We tried it in Minecraft by rendering a cubemap in OpenGL, but couldn't get better than 30 fps.
• It inspired partial inclusion in the Unreal engine, but with only standard FOVs to save frame rate.

Before I jump in—could you weigh in with your thoughts? I imagine this wouldn't be hard to do since each pixel just has to map to a different ray determined by the chosen projection, but I'm not sure if anything in the pipeline assumes the standard projection. Thanks for your thoughts.

[cschied]

This is entirely possible. Care needs to be taken that the forward-projection in is changed accordingly.

The main path tracer lives in , search for global_ubo.VP

Some of the reprojection code and the reconstruction filter might need some fiddling with the epsilons.

[shaunlebron]

Thanks, looks like the call to get_primary_ray is where we can do the projection. 👍

q2vkpt/src/refresh/vkpt/shader/path_tracer.h

Line 484 in a4de8e9

Ray ray = get_primary_ray(screen_coord_cs);

[cschied]

Yes, you just need to make sure to also change all the code that uses global_ubo.VP

[BurningNorth]

My practical knowledge of path tracing is too limited to be any help here, but as someone who's spent far too much time trying to come up with needlessly clever raster projections to do this in other engines without the post-process step (spoiler alert: haha nope), I'm super excited to see if you can make this work. 😃

[shaunlebron]

vp = vector projection? found this reference:

q2vkpt/src/refresh/vkpt/shader/asvgf_fwd_project.comp

Line 90 in a4de8e9

vec4 pos_cs_curr = global_ubo.VP * vec4(pos_ws, 1.0);

[cschied]

This is the view-projection matrix, going from world-space to clip-space.

[shaunlebron]

Ah I see now, thanks. Here's a plan, if you wanna check it:

Standard Projection Matrices

These matrices are used for standard projection:

• global_ubo.VP - forward projection matrix
• global_ubo.invVP - inverse projection matrix

Non-linear projection functions

NOTE: Non-linear projections can't be performed by matrices—it's a linear algebra afterall.

To swap out the standard projection with a non-linear one—any time the above matrices are multipled by a vector, we have to replace them with calls to custom projection functions:

• Replace global_ubo.VP * ws -> global_ubo.customVP(ws)
• Replace global_ubo.invVP * cs -> global_ubo.customInvVP(cs)

Where to change?

Add projection functions to global_ubo.h:

• customVP(ws) - from world space to clip space (forward projection)
• customInvVP(cs) - from clip space to world space (inverse projection)

Modify to use inverse projection (customInvVP) here:

q2vkpt/src/refresh/vkpt/shader/path_tracer.h

Line 282 in a4de8e9

get_primary_ray(vec2 pos_cs)

Modify to use forward projection (customVP) here:

q2vkpt/src/refresh/vkpt/shader/asvgf_fwd_project.comp

Line 90 in a4de8e9

vec4 pos_cs_curr = global_ubo.VP * vec4(pos_ws, 1.0);

[cschied]

The plan looks good! Let me know if you need any additional information. There might be the need to fix some of the reconstruction filter but we'll have to see the results first.