-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal: Dynamic Output Resources #1180
Labels
kind/feature
Categorizes issue or PR as related to a new feature.
Comments
I've added this as a proposal to solve #1279 (https://docs.google.com/document/d/13TZdWU6PEaOwpe--Zw96lcpjRqksJ7zAQ-mxk1qnZ30/edit#). |
Updated the example to use Pipeline and PipelineRun. Also updated the above design doc. |
vdemeester
added
the
kind/feature
Categorizes issue or PR as related to a new feature.
label
Nov 7, 2019
ghost
mentioned this issue
Dec 3, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Expected Behavior
Currently a
PipelineRun
binds a task's output to predefined static resources, which is not intuitive. A more natural way to model a pipeline is to dynamically generate new resources based on predefined templates as the output of a task. For example:In the above example, the PipelineRun and Pipeline does not bind any predefined image resource to the
img
output resource of Taskbuild-and-push
. Instead, Taskbuild-and-push
provides how the output image resource should look like, and defines output values for each step that can be used for templating the output image resource. Ideally, the following resource would be generated automatically:Since output resources are not statically bound, they are immutable once generated. The deletion of the
PipelineRun
should clean up all output resources it generates.We could further support generating arbitrary resources like what a
TriggerTemplate
does:This would immediately deploy a new Knative service YAML, provided that Knative serving is installed in the cluster. Otherwise the
TaskRun
should fail, with the side effect that the built image is uploaded by thekaniko
step. The generated output objects are notPipelineResources
, thus cannot be used as inputs to otherTasks
in aPipeline
.Actual Behavior
The current designs of
Task
,TaskRun
andPipelineRun
impose static bindings betweenpre-existing
PipelineResources
and the output resources declarations ofTasks
, and there is no way to "update" the output resources with data produced inTasks
. As a result, there areworkarounds such as #721, and issues like #216 and #552.
Also, right now deploying an image built by a
Task
in aPipeline
usually involves defining anotherTask
that 1. fetches a resource YAML from some source repository, 2. modifies the resource YAML through e.g.yq
orkustomize
to use the built image, and 3. useskubectl
to upload the modified resource YAML. The is fairly routine and should have a built-in mechanism in Tekton pipeline.The proposed mechanism of dynamic output resources could resolve the above issues.
Additional Info
Output Resource Naming
There are a couple use cases to consider:
Static names. This could be achieved as follows:
If a
PipelineResource
with the same name already exists, theTask
should fail.Dynamic names that are not referenced by other resources. This seems to be the typical use case and the proposed mechanism relies on Kubernetes generating names for the output resources.
Dynamically names that need to be referenced by other
Tasks
. I'm not proposing any solution here, but it might be good to align with Add Support For Naming Modifiers triggers#50.Implementation
A possible implementation is to run a sidecar container that receives key-value pairs through HTTP POST requests for output resource tempating. At the end of each step, the redirector birany would send the outputs of the step to the sidecar. Once all template variables are resolved, the sidecar could generate the output resources and submit them to the API server; it could serve the resource YAML through HTTP GET requests, and use another step to control when to submit the output resources.
The text was updated successfully, but these errors were encountered: