Shader hooks #7149

davepagurek · 2024-07-26T19:21:22Z

Resolves #7031

Changes

Screenshots of the change

Wobbly material:

Custom normal material colors:

Fuzzy line rendering:

PR Checklist

npm run lint passes
[Inline documentation] is included / updated
[Unit tests] are included / updated

This reverts commit 286fc2f.

davepagurek · 2024-07-26T19:33:46Z

One pattern I've noticed so far: If you just wrap individual inputs in a hook, then users are forced to handle their inputs in the order the code executes. This is a problem if, for example, they want to adjust the color based on the texture coordinate, but your shader runs its getColor hook before its getTextureCoordinate hook -- the user has no way to access the texture coordinate when it is time to update the color.

The solution I've come to is to make a struct with all the inputs, and then make a single hook for all of them at once. Then users can use whichever they want to modify whichever others they want -- or store them to a global so then use in a later hook.

davepagurek · 2024-07-29T19:53:55Z

I think rather than handling different GLSL versions, which would require some level of cross-compilation or asking the user to use our special macros, it's ok if shaders just declare what version they use, and it is the user's responsibility to adhere to that. For default p5 shaders, that means GLSL ES 300 if you're using WebGL 2 (the default), and GLSL ES 100 if you're using WebGL 1.

davepagurek · 2024-08-20T23:44:25Z

Ok I think this is code-complete for this repo. So far, I think the best we can do for the inline docs is a bullet list explaining the possible hooks. That currently looks like this:

While this is still rather imposing, I'm hoping that this will eventually serve as a reference, and tutorials will be the place that lays out information incrementally instead of everything all at once. The next step for me will be to start drafting some of that in the p5.js-website repo.

Still left to do here: test this on older computers to make sure there hasn't been a major degradation in performance

davepagurek · 2024-09-03T15:46:27Z

In this test https://editor.p5js.org/davepagurek/sketches/kSsPHuHY6 on my old 2015 Intel mac, Im getting:

	Firefox		Chrome
	1.10.0	With hooks	1.10.0	With hooks
Large curve	8fps	7fps	11fps	11fps
Many lines	25fps	25fps	30fps	35fps
Shading	25fps	25fps	32fps	31fps

These are all in the same ballpark so I feel safe assuming this won't affect performance.

davepagurek · 2024-09-07T18:51:37Z

Try it out!

On the p5 web editor

Here's a web editor sketch you can fork to try it out!

Manually

Here's a zip of the built library you can add to a project.

Reference

I put up a build of the site with the shader hooks docs here. Check out shader.inspectHooks(), shader.modify(), and the base shaders, colorShader(), materialShader(), normalShader(), and lineShader(): https://davepagurek.github.io/p5.js-website/reference/#3D

Quick Tutorial

Shaders provide a way to efficiently modify the color and position of shapes. You might think you haven't used shaders before, but p5.js is actually using them behind the scenes any time you use WebGL mode! While you can always create your own shader from scratch, it can be easier to start from a p5.js shader and make modifications to the parts of interest to you.

Let's say you're drawing a circle in WebGL mode, and you'd like to make a custom material for it.

function setup() { createCanvas(200, 200, WEBGL); } function draw() { background(255); noStroke(); fill('red'); circle(0, 0, 50); }

When you draw with solid colors, under the hood, p5.js is using its materialShader(). You can call materialShader().inspectHooks() to see what things you can modify about it:

==== Vertex shader hooks: ==== 
void beforeVertex() {} 
vec3 getLocalPosition(vec3 position) { return position; } 
vec3 getWorldPosition(vec3 position) { return position; } 
vec3 getLocalNormal(vec3 normal) { return normal; } 
vec3 getWorldNormal(vec3 normal) { return normal; } 
vec2 getUV(vec2 uv) { return uv; } 
vec4 getVertexColor(vec4 color) { return color; } 
void afterVertex() {} 

==== Fragment shader hooks: ==== 
void beforeFragment() {} 
Inputs getPixelInputs(Inputs inputs) { return inputs; } 
vec4 combineColors(ColorComponents components) {
                vec4 color = vec4(0.);
                color.rgb += components.diffuse * components.baseColor;
                color.rgb += components.ambient * components.ambientColor;
                color.rgb += components.specular * components.specularColor;
                color.rgb += components.emissive;
                color.a = components.opacity;
                return color;
              } 
vec4 getFinalColor(vec4 color) { return color; } 
void afterFragment() {}

Each one of these is called a hook because it lets you attach your own bits of code into the shader. The documentation for a p5.js shader will include a description of what each one does, and inspectHooks() will show you what each one is set to by default.

Let's say we want to wiggle the shape by adding a sine wave to every vertex. We can do that by filling one of the position hooks. We do that by calling .modify() on the original shader, and then passing in an object with the hooks we want to change. The key of the object is the hook we're providing, and the body is the function, written in GLSL.

let myShader;

function setup() {
  createCanvas(200, 200, WEBGL);
  myShader = materialShader().modify({
    'vec3 getWorldPosition': `(vec3 pos) {
      pos.y += 15.0 * sin(pos.x * 0.15);
      return pos;
    }`
  });
}

function draw() {
  background(255);
  noStroke();
  fill('red');
  shader(myShader)
  circle(0, 0, 50);
}

To animate that sine wave, we'll want to tell the shader what the current time is so that it can factor that in. You can add a declarations property to the object to stick some GLSL code at the top of your vertex or fragment shader. To add a new input to your shader, you can declare a new uniform variable, and then update it each frame with setUniform:

let myShader;

function setup() {
  createCanvas(200, 200, WEBGL);
  myShader = materialShader().modify({
    declarations: 'uniform float time;',
    'vec3 getWorldPosition': `(vec3 pos) {
      pos.y += 15.0 * sin(time * 0.002 + pos.x * 0.15);
      return pos;
    }`
  });
}

function draw() {
  background(255);
  noStroke();
  fill('red');
  shader(myShader)
  myShader.setUniform('time', millis());
  circle(0, 0, 50);
}

If you want a declaration to only be in a vertex shader (the one that edits positions) or a fragment shader (the one that colours the pixels), you can use vertexDeclarations or fragmentDeclarations instead. These can also be useful if you want to add a new function that you want to use in your hooks. Here's a hook that uses a noise function (courtesy of Lygia) to change the color of the shape:

let myShader;

function setup() {
  createCanvas(200, 200, WEBGL);
  myShader = materialShader().modify({
    declarations: `
      float rand(float n) {
        return fract(sin(n) * 43758.5453123);
      }
      float rand(vec2 n) { 
        return fract(sin(dot(n, vec2(12.9898, 4.1414)))
          * 43758.5453);
      }

      float noise(float p){
        float fl = floor(p);
        float fc = fract(p);
        return mix(rand(fl), rand(fl + 1.0), fc);
      }

      float noise(vec2 n) {
        const vec2 d = vec2(0.0, 1.0);
        vec2 b = floor(n),
        f = smoothstep(vec2(0.0), vec2(1.0), fract(n));
        return mix(
          mix(rand(b), rand(b + d.yx), f.x),
          mix(rand(b + d.xy), rand(b + d.yy), f.x),
        f.y);
      }
    `,
    'vec4 getFinalColor': `(vec4 c) {
      c *= noise(gl_FragCoord.xy * 0.15);
      return c;
    }`
  });
}

function draw() {
  background(255);
  noStroke();
  fill('red');
  shader(myShader)
  circle(0, 0, 50);
}

src/webgl/p5.Shader.js

perminder-17 · 2024-09-12T00:53:07Z

Try it out!

On the p5 web editor

Here's a web editor sketch you can fork to try it out!

Manually

.....

Hi Dave, your work looks amazing as always. I was experimenting with the shader hooks you created, and I think they’re going to be one of the standout features in version 2.0. There’s so much to appreciate in this implementation, but I’ll focus on giving my feedback and suggestions for further improvements.

Everything looks great to me. I just reviewed the phase 2 ideas of shader hooks, and it already covers most of the ideas I had in mind.

Is there a way we can remove the need for declarations: when passing uniforms or functions? The goal is to simplify things for users so they don't have to dive deep into GLSL. For example, if a user calls myShader.setUniform('time', mill()), we could automatically set the time uniform in the shader, allowing them to use the time variable across the shader without needing to write uniform float time;, which can be a bit complex.

Additionally, we have vertexDeclarations and fragmentDeclarations, which feel a bit odd. If a user passes a function to the fragment shader but doesn't want it in the vertex shader, they currently have to use fragmentDeclarations: yourFunction. Would it be possible to avoid declarations altogether, and instead streamline this process?

myShader = materialShader().modify({

     float noise(vec2 n) {
        const vec2 d = vec2(0.0, 1.0);
        vec2 b = floor(n),
        f = smoothstep(vec2(0.0), vec2(1.0), fract(n));
        return mix(
          mix(rand(b), rand(b + d.yx), f.x),
          mix(rand(b + d.xy), rand(b + d.yy), f.x),
        f.y);
      }
  
    'vec4 getFinalColor': `(vec4 c) {
      c *= noise(gl_FragCoord.xy * 0.15); // the function is only defined at fragment shader since it's dealing with colors.
      return c;
    }`

Or perhaps, if we're modifying color values that relate to fragment shader hooks and the user has a relevant function, we could automatically move that function to the fragment shader. Similarly, if something involves position, we could directly move that function to the vertex shader. Or maybe we could define it int both shaders.

If not possible due to some reasons then we can add documentation for declaration (fragmentDeclerations, decleration and vertexDecleartion).

One key feature I believe users will find helpful while learning is shader.inspectHook(), which shows all the modified hooks. Could we include a sample sketch along with a screenshot of the logged values in the reference? This would help users understand how to explore and learn shader hooks by experimenting with shader.inspectHook().

Rest, looks awesome. These are just some ideas, and they might be a bit off, so feel free to take them as you see fit.

Co-authored-by: Perminder Singh <[email protected]>

davepagurek · 2024-09-13T22:48:56Z

Thanks @perminder-17! These are all interesting ideas. I'll comment a bit on the technical challenges we'd have to overcome, hopefully we can figure something out that addresses those points!

Making uniforms easier

It's a little hard generating uniforms from setUniform because we currently we bind a shader when a user calls shader(yourShader), which means having its code compiled, and if that happens before a call to setUniform, we wouldn't yet know which uniforms to include when we compile the code. We could record just that we will bind a shader on shader(), and then actually compile right before we start drawing, but that might be too large of a refactor for now. There might be some other ways to make uniforms easier though!

Another thought: instead of removing the uniform declaration and shifting it to setUniform, maybe for some cases we could remove the setUniform and shift it to the uniform declaration? e.g. if you're going to always pass in millis() every frame, maybe you could include a default value right in the shader declaration? Something like:

let myShader;

function setup() {
  createCanvas(200, 200, WEBGL);
  myShader = materialShader().modify({
    uniforms: {
      'float time': () => millis()
    },
    'vec3 getWorldPosition': `(vec3 pos) {
      pos.y += 15.0 * sin(pos.x * 0.15);
      return pos;
    }`
  });
}

function draw() {
  background(255);
  noStroke();
  fill('red');
  shader(myShader)
  // no need to setUniform!
  circle(0, 0, 50);
}

...and then maybe every time you call shader(myShader), we can automatically setUniform with the default values? It would be nice to also have a way to add uniforms without default values to that, but I haven't thought of a great way to also include that in an object. Maybe just float myUniform: null or undefined?

Making functions easier

I really like the idea of being able to include other functions in with the hooks like that. I could see having addons that package useful functions, random and noise functions being the ones I'm copy-and-pasting into 99% of my shaders, and then you could mix them into your hooks like this:

addon.js

sketch.js

const noise = {
  'float rand': `(float n) {
    return fract(sin(n) * 43758.5453123);
  }`,

  'float rand2': `(vec2 n) { 
    return fract(sin(dot(n, vec2(12.9898, 4.1414)))
      * 43758.5453);
  }`,

  'float noise': `(float p){
    float fl = floor(p);
    float fc = fract(p);
    return mix(rand(fl), rand(fl + 1.0), fc);
  }`,

  'float noise2' `(vec2 n) {
    const vec2 d = vec2(0.0, 1.0);
    vec2 b = floor(n),
    f = smoothstep(vec2(0.0), vec2(1.0), fract(n));
    return mix(
      mix(rand2(b), rand2(b + d.yx), f.x),
      mix(rand2(b + d.xy), rand2(b + d.yy), f.x),
    f.y);
  }`
}

// mix in all noise functions
myShader = materialShader().modify({
  ...noise,
  'vec4 getFinalColor': `(vec4 c) {
    c *= noise(gl_FragCoord.xy * 0.15);
    return c;
  }`
})

The two consequences of this are:

Because so far the keys of the object are just the name + return type of a function, using this syntax means you can't overload functions (e.g. have float rand(float n) and float rand(vec2 n) in the same shader), since they'd both be overwriting the same key of the object.

Honestly, this is maybe fine, overloading isn't something everyone will use anyway, and we can still give them the more advanced declarations to use if they really want it.
Because there is now a valid use for functions that aren't already hooks, we can't show a friendly error if e.g. you make a typo in the name of your hook, since that would be a valid helper function declaration.

Some thoughts on ways to address this:
- If we could detect when a function is used, we could show a friendly error if you declare a function that nothing uses. However, this would need us to parse the GLSL strings, and bring in a shader parser library, which I think might not be worth the additional size right now. I would love to have this as part of [p5.js 2.0 RFC Proposal]: (Phase 2) R&D to explore GLSL alternatives #7188 though maybe!
- We could separate hooks: { ...} and helpers: { ... } into separate objects. It's a little more verbose to type though.
- We could just not have this friendly error. Maybe not a huge loss? inspectHooks() could maybe still tell you when your hook hasn't been added correctly, if it ended up in a helper instead.

davepagurek · 2024-09-14T18:01:24Z

I've updated the code with an implementation of that uniforms section and also support for helper functions! I've also added @beta to the docs and made a separate PR in the website repo to handle that.

perminder-17 · 2024-09-15T23:32:39Z

Thanks @davepagurek for considering my feedback on simplifying uniforms and functions. I'm looking forward to help on the phase 2 R&D, where we can work together to find effective solutions for the helper functions.

Just a question,

Do you know why the example you pushed for inspectHook is displaying in just one line? Can we format it to show across multiple lines? Right now, I have to scroll horizontally to view the whole example 😅😅

davepagurek · 2024-09-16T11:47:17Z

oops good catch, I fixed that bug on the main branch of the website repo but haven't updated my hooks fork to include that!

davepagurek · 2024-09-17T20:53:16Z

Thanks for your input everyone! I'm going to merge this in as a beta API so that we can get more testers in the next 1.x release and so 2.0 projects can build off of it. Feel free to still give more feedback and open more issues so we can improve this feature!

davepagurek added 8 commits April 13, 2024 12:05

Add shader hooks

e18794f

Remove per-pixel lighting

286fc2f

Start adding more hooks

7ab36a1

Add line shader hooks

9904125

Revert "Remove per-pixel lighting"

fb65c8f

This reverts commit 286fc2f.

Start documenting hooks

7815679

Add examples, fix bugs

f8cfce5

Add examples, update hooks format

cad9d54

davepagurek marked this pull request as draft July 26, 2024 19:21

davepagurek added 4 commits July 26, 2024 15:23

Merge branch 'main' into shader-hooks

69fdeb9

Merge branch 'main' into shader-hooks

4a77779

Remove accidentally committed file

8ee8867

Fix missing semicolon

b9e89d6

davepagurek added 2 commits July 29, 2024 13:07

Write out struct members as bullets

4744428

Add more examples, document shader version

22addb5

davepagurek added 2 commits July 30, 2024 11:26

Make metalness be editable too

420bac4

Add more line examples

d50d0cb

davepagurek marked this pull request as ready for review August 20, 2024 23:38

davepagurek changed the title ~~[WIP] Shader hooks~~ Shader hooks Aug 20, 2024

Add some unit tests

82a92f8

davepagurek added 3 commits September 7, 2024 09:13

Test different reference layout

9fe21e3

Try it as a table

db29371

Use tables for all hooks

5af4635

perminder-17 reviewed Sep 11, 2024

View reviewed changes

src/webgl/p5.Shader.js Outdated Show resolved Hide resolved

Update src/webgl/p5.Shader.js

003f3cc

Co-authored-by: Perminder Singh <[email protected]>

davepagurek added 2 commits September 14, 2024 12:28

Add ability to add other functions and uniforms

51f1b36

Add beta tag

6f24ac5

davepagurek mentioned this pull request Sep 14, 2024

Handle @beta in reference items processing/p5.js-website#544

Merged

davepagurek mentioned this pull request Sep 14, 2024

Shader hooks for dev-2.0 #7267

Merged

davepagurek merged commit 5042087 into processing:main Sep 17, 2024
2 checks passed

davepagurek deleted the shader-hooks branch September 17, 2024 20:54

perminder-17 mentioned this pull request Sep 19, 2024

Make shaders define what they get used for. #7256

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Shader hooks #7149

Shader hooks #7149

davepagurek commented Jul 26, 2024 •

edited

Loading

davepagurek commented Jul 26, 2024

davepagurek commented Jul 29, 2024

davepagurek commented Aug 20, 2024 •

edited

Loading

davepagurek commented Sep 3, 2024

davepagurek commented Sep 7, 2024 •

edited

Loading

perminder-17 commented Sep 12, 2024 •

edited

Loading

Try it out!

On the p5 web editor

Manually

davepagurek commented Sep 13, 2024

davepagurek commented Sep 14, 2024

perminder-17 commented Sep 15, 2024 •

edited

Loading

davepagurek commented Sep 16, 2024

davepagurek commented Sep 17, 2024

Shader hooks #7149

Shader hooks #7149

Conversation

davepagurek commented Jul 26, 2024 • edited Loading

Changes

Screenshots of the change

PR Checklist

davepagurek commented Jul 26, 2024

davepagurek commented Jul 29, 2024

davepagurek commented Aug 20, 2024 • edited Loading

davepagurek commented Sep 3, 2024

davepagurek commented Sep 7, 2024 • edited Loading

Try it out!

On the p5 web editor

Manually

Reference

Quick Tutorial

perminder-17 commented Sep 12, 2024 • edited Loading

Try it out!

On the p5 web editor

Manually

davepagurek commented Sep 13, 2024

Making uniforms easier

Making functions easier

davepagurek commented Sep 14, 2024

perminder-17 commented Sep 15, 2024 • edited Loading

davepagurek commented Sep 16, 2024

davepagurek commented Sep 17, 2024

davepagurek commented Jul 26, 2024 •

edited

Loading

davepagurek commented Aug 20, 2024 •

edited

Loading

davepagurek commented Sep 7, 2024 •

edited

Loading

perminder-17 commented Sep 12, 2024 •

edited

Loading

perminder-17 commented Sep 15, 2024 •

edited

Loading