Add more supported Tekton variables

[Tomcli]

We want to convert some of the Argo pipeline into Tekton. Below is the variables that Argo support, what variables would make sense for us to implement it to Tekton? We would like to at least have variable to refer to global pipeline parameters and metadata from the pipelinerun.

https:/argoproj/argo/blob/master/docs/variables.md

[bobcatfish]

Hey @Tomcli !

Potentially we would be interested in adding more contextual information - we already have a need to add more metadata for features like #1684 but it would really help if you could specifically list what extra info you need and what your use cases are for it. Or it doesn't have to be you - basically it looks like the next step here is to go through each of the variables in https:/argoproj/argo/blob/master/docs/variables.md and see how they compare to what we have in Tekton. But understanding your specific use cases for them and which ones are high priority for you would really help us @Tomcli

[Tomcli]

Thanks @bobcatfish. We would love to find out a list of variable substitution supported in Tekton #2267. If you could point us to all the docs on the different kinds of Tekton variable substitution then that will be great.

Right now we only found input/output parameters and path for variable substitution in task, pipeline, condition, and pipeline resource. Is there other type of variable substitution in Tekton?

[bobcatfish]

I think you covered pretty much everything, a couple others that come to mind are workspaces and results. I'm wondering if it would make sense to break those out into one doc so you can see all the substitutions available at a glance like you can with argo! ( + @sergetron )

Also + @sbwsg who might know of some substitutions and/or existing docs that I'm missing.

[ghost]

Egads, I just went through to confirm the ones I could remember. Here's what I found:

Available in Tasks

• params.<param name>
• resources.inputs.<resource name>.path
• resources.outputs.<resource name>.path
• results.<result name>.path
• workspaces.<workspace name>.path
• workspaces.<workspace name>.volume
• credentials.path

Each PipelineResource type also defines its own variables that I believe are available in Tasks:

• Git PipelineResource Variables
• PullRequest Resource Variables
• Image Resource Variables
• GCS Resource Variables
• GCS Build Resource Variables
• Cluster Resource Variables
• CloudEvent Resource Variables

(These are all accessed through resources.inputs.<resource name>.<variable> or resources.outputs.<resource name>.<variable>)

Available in Pipelines

• params.<param name>
• tasks.<task name>.results.<result name>

I wonder if I missed any ¯\_(ツ)_/¯ We should definitely aggregate these into a single doc.

[ghost]

OK, I think I've collected them all together here: https:/tektoncd/pipeline/pull/2358/files

[animeshsingh]

@Tomcli can you be specific around which of the variables are currently needed for supporting Kubeflow, and what are alternatives?

[Tomcli]

@animeshsingh We would love to have variable substitutions for pipelinerun name, taskrun name, and pod name because these variables are often used in Kubeflow. Right now we have workaround to get those values from the metadata labels and pass them as environment variables. But if Tekton can also support it as variable substitutions, then the integration will be a lot smoother.

[bobcatfish]

Oh perfect @Tomcli we've actually got #1522 to add pipelinerunname and taskrun name (I was even talking about that with @jerop today in case she gets to it 😉 )

pod name is a bit trickier because that would mean having a kubernetes specific variable replacement - in theory we like to limit the amount of k8s specific stuff we have in the API as much as we can. Could you talk a bit more about your use case around needing the pod name?

[Tomcli]

Many of the pod name use cases in Kubeflow is for referencing a task run because Argo doesn't have names for task run. Therefore, some API logic assumes the task run name is the pod name. Also, some users take advantage of the pod name variable substitutions and use it in their tasks to do additional work using the k8s api, so that also created some edge cases for the translation.

[ghost]

Tekton labels pods with the name of the TaskRun/PipelineRun that own it. These labels take the form of tekton.dev/taskRun=<name> and tekton.dev/pipelineRun=<name>. So once #1522 is implemented you'd be able to look up the pods from inside a Step by using a label selector. Bear in mind also that a TaskRun can be retried and that when a retry happens a new Pod is created. Retries can happen because of resource limits forcing delayed startup or because the user explicitly added retry: <num> to their Tasks. So there isn't really a 1:1 mapping between TaskRun and Pod.

I guess all of this is to say: if we are going to expose the Pod name as a variable inside a Step we probably need to look at the specific usecases and ensure they make sense for Tekton.

[animeshsingh]

@skaegi @afrittoli please take a look

[skaegi]

Ha! I just had someone over the week-end bring up the need for seeing the current retry number. Basically they send a conditional slack notification if a retry is required. I think the annotations are ok, but it probably makes sense to look at adding an additional variable scope that authors can use at definition time instead of just using the downward api in pod env. I'll bring it up today in the API call.

[skaegi]

Also see #1522

[bobcatfish]

It seems like we're adding (or planning to add) more variables which is great! And now we actually have docs on what is available which is super useful.

As people discover more variables they'd like to see, it's probably easier to detail with them one at a time - any objections if I close this issue and folks can open new issues as they discover specific values they'd like to be able to use?