reuse Tasks inside the same Pod?

[jstrachan]

Feature request

is there any way to create a Pipeline with multiple Tasks so that it uses the same Pod? Effectively combining multiple Tasks into a single effective Task?

Currently each Task creates a separate Pod which then means you need to create a PVC and workspace to share content between the Tasks.

Use case

I've been looking at migrating the Jenkins X Pipeline Catalog to reuse a git clone task like the one in the catalog.

So my first attempt was to reuse a Task for git-clone then a second Task reads the output by creating a workspace between the two + having the PipelineRun create a PersistentVolumeClaim and so forth. It works - which is great - but from an infrastructure perspective, creating 2 pods and a PVC per pipeline run - to run 2 steps (one git clone and one step to process the output of the git clone) does seem a little over the top.

I do really like the composability of Pipelines into Tasks which can either be inline or separate via references and bundles; I wondered if there was a way to co-locate multiple Tasks together to avoid the workspace/PVC issue?

For lots of common CI checks where there's no real need for persistence; its one step to git clone and another step to do some kind of checks/validation; using multiple pods with PVCs doesn't seem ideal.

Maybe a different kind of reuse/composability is required to allow steps to be reused from one Task to another Task but without using the current Pipeline / Task composition model? e.g. maybe a source code transformation step that copies the step from a shared git-clone Task into the start of the other Task - so we get reuse but can avoid the complexity of the Workspaces/PVCs if not totally necessary?

Or should I just not worry about the overhead of the extra pod + PVC for each pipeline run instance? :)

[ghost]

Or should I just not worry about the overhead of the extra pod + PVC for each pipeline run instance? :)

Definitely worth worrying about! We've had reports from users on Slack that, for example, their org doesn't allow them to allocate storage, which prevents them from using Pipelines + Workspaces almost entirely. We've also seen similar feedback that writing a Pipeline just for 2 relatively simple tasks (encompassing only 2 steps as in your example) is onerous / too labor-intensive.

I think a big part of the PipelineResources redesign is aiming directly at these kinds of use-cases. PipelineResources in their current incarnation effectively allow two Tasks' Steps to be merged into a single TaskRun (and therefore a single Pod), it's just that one of those Tasks is from an extremely constrained pool of options (git, gcs, image and cluster only).

For posterity it's also worth mentioning that we've explored this idea outside of PipelineResources as well, with a Pre/Post Step idea too. I think I've also heard discussion in passing of Pipelines being able to put all their Tasks in one Pod but I don't think we have that recorded as an issue anywhere so very much appreciate you getting this documented!

maybe a source code transformation step that copies the step from a shared git-clone Task into the start of the other Task - so we get reuse but can avoid the complexity of the Workspaces/PVCs if not totally necessary

I really like this idea. I would guess that this should be achievable/prototypable with kustomize? Might be an excellent way to prove out the concepts in play before committing to a specific implementation in the Pipelines controller. Might also be a great way to expose ourselves to potential bugs and pain points before writing any go code!

[jstrachan]

@sbwsg thanks for the heads up! let me read up on the PipelineResources redesign. I might have a little experiment with a source code transformation for a little while.

Incidentally I really love the tekton bundles approach. It would be super nice if we could nest the steps from a shared Task at a specific point in a Task's list of steps via the same kind of taskRef mechanism that can be used in a Pipeline...

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: my-simple-lint
spec:
  steps:  
  - taskRef:  # this step gets replaced by all the steps inside the shared git-clone Task
      name: git-clone
      bundle: docker.io/myrepo/mycatalog:v1.0.1
  - image: some/linter:1.2.3
    name: lint
    workingDir: /workspace/source

which is conceptually like having a Task inside a Task, but it just inlines the steps really - kinda like how the taskRef works in a Pipeline

[jlpettersson]

I was exploring some of these ideas in #2586
Sometimes I feel that it would be very tempting to run the whole PipelineRun within a single Pod. In #2586 the idea was that since the tasks needed to share a volume, it could be done within a Pod - or all pods should be scheduled to the same Node - to allow for parallel execution.

In discussions around that issue, we concluded that re-designing so all tasks would run within a single Pod is quite disruptive and big change. One concern I also had was that this means that all task would be run using the same ServiceAccount, so it would be harder in a security perspective as well - not always important, but sometimes.

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[jlpettersson]

This is now investigated in two TEPs

• Add TEP-0044: Decouple Task Composition from Scheduling ➰ community#316
• Add TEP-0046: Colocation of Tasks and Workspaces (formerly PipelineRun in a Pod) community#318

/remove-lifecycle stale

[vdemeester]

/remove-lifecycle stale

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale with a justification.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen with a justification.
Mark the issue as fresh with /remove-lifecycle rotten with a justification.
If this issue should be exempted, mark the issue as frozen with /lifecycle frozen with a justification.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.