Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add paketo builder option for buildpack #190

Merged
merged 3 commits into from
May 18, 2020
Merged

add paketo builder option for buildpack #190

merged 3 commits into from
May 18, 2020

Conversation

xiujuan95
Copy link
Contributor

@xiujuan95 xiujuan95 commented May 12, 2020

Fix issue:#147

  • Add gcr.io/paketo-buildpacks/builder:latest builder image option for buildpack;

In order to differ paketo and heroku, I add a suffix to the name, like build-heroku and build-paketo.

  • Add two e2e test to verify paketo, also e2e test passed
2020-05-13T02:16:24Z 1 Killing the local operator
 
 
Ran 10 of 15 Specs in 327.337 seconds
SUCCESS! -- 10 Passed | 0 Failed | 0 Pending | 5 Skipped
 
 
Ginkgo ran 1 suite in 5m35.11636469s
Test Suite Passed
  • There has some differences between paketo and heroku

First, for paketo, at prepare step, we should make user 1000 to own /tekton/home file first. Because the docker credential is stored here. If we don't do this, then when access this secret will produce permission denied issue.
And also we should run with user 1000 at export step. Because only user 1000 has the authority to access docker credential file instead of user 0.

Second, at export step, --helper parameter has beed removed for paketo
Because the version of lifecycle in paketo image is 0.7.3 and heroku is 0.6.1. For this 0.7.3 version, the docker credential helper has been removed, so this parameter also doesn't be needed.

@openshift-ci-robot
Copy link

Hi @xiujuan95. Thanks for your PR.

I'm waiting for a redhat-developer member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 12, 2020
Copy link
Member

@SaschaSchwarze0 SaschaSchwarze0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We just discussed in our scrum meeting. We think that we would not want to replace heroku with paketo but provide it as an alternative. Correct @sbose78 @qu1queee ?

This would mean that we relabel the old one's to be something like buildpacks-heroku, add new build strategies and builds and build runs for paketo, and add new specs in the e2e tests.

@xiujuan95
Copy link
Contributor Author

xiujuan95 commented May 12, 2020

@SaschaSchwarze0 Thanks for your comments!

If you all @qu1queee @sbose78 @zhangtbj agree with provide an paketo option for buildpack is better than replace it with paketo. Then I will modify my code. Please let me know the decision, thanks a lot!

And if provide an option, I plan to add a suffix to differ heroku and paketo. For example:
For heroku, these names should be like:

  • buildpack-nodejs-build-heroku
  • buildpack-nodejs-buildrun-heroku
  • buildpacks-v3-heroku

And for paketo, it should be:

  • buildpack-nodejs-build-paketo
  • buildpack-nodejs-buildrun-paketo
  • buildpacks-v3-paketo

Do you all agree?

@SaschaSchwarze0
Copy link
Member

SaschaSchwarze0 commented May 12, 2020

@xiujuan95 I agree, but let's also wait for @qu1queee and @sbose78 to confirm.

EDIT: just saw that @qu1queee did a +1 on my previous comment. So, he is probably okay as well.

@qu1queee
Copy link
Contributor

@xiujuan95 yes, lets keep paketo as an option not as a replacement. Also, I would be very interesting to see with which "runtimes" you tested this, e.g. could you say that this paketo works for golang, java, nodejs, etc?

@zhangtbj
Copy link
Contributor

zhangtbj commented May 13, 2020

Yes, for the first phase, I think two buildpacks strategies is ok for now.

But finally, we should JUST have one buildstrategy for buildpacks.

The different between heroku and paketo is just the builder image. It is total different with the case of Kaniko and buildah (different tools)

Two same buildpacks build strategies also make end-user confusion and I think most of the users don't know which is better or use, so we should provide a prefer one (then they may ask why google fans cannot choose google's buildpacks).

If we keep two, do we want to also keep the duplicate e2e tests for them? It will take e2e tests longer.

This is similar as the current case for heroku and cloudfoundry, we don't have two buildpacks buildstrategies for them, we just replaced from cloudfoundry to heroku and document in our buildstrategy document that we support two of them:
https:/redhat-developer/build/blob/master/docs/buildstrategies.md#buildpacks-v3


The buildpacks-v3 BuildStrategy/ClusterBuildStrategy uses a Cloud Native Builder (CNB) container image, and is able to implement lifecycle commands. The following CNB images are the most common options:

heroku/buildpacks:18
cloudfoundry/cnb:bionic

So I suggest, if we all agree and verify the paketo buildpacks, then replace and update the document to aovid the confusion.

@xiujuan95
Copy link
Contributor Author

xiujuan95 commented May 13, 2020

@xiujuan95 yes, lets keep paketo as an option not as a replacement. Also, I would be very interesting to see with which "runtimes" you tested this, e.g. could you say that this paketo works for golang, java, nodejs, etc?

Thanks @qu1queee ! Now, I can't say all runtimes work fine. Because before I just tested use nodejs runtime. But in theory, currently, paketo should support Java, Node.js, Go, PHP, .NET Core and NGINX.
I will use CATs app to confirm them next, and tell you the result.

@xiujuan95 xiujuan95 changed the title replace heroku with paketo buildpack builder add paketo builder option for buildpack May 13, 2020
@xiujuan95
Copy link
Contributor Author

xiujuan95 commented May 13, 2020

@qu1queee I verified nodejs/java/php/golang/dotNetCore runtime in cf-acceptance-test. And NGINX is unknown for me. I don't know which sample is suitable. So I don't verify it.

nodejs, java and php passed.

But golang and dotNetCore both have some issues. For golang, at detect step, it failed:

ERROR: No buildpack groups passed detection.
ERROR: Please check that you are running against the correct path.
ERROR: failed to detect: no buildpacks participating

And for dotNetCore, if failed at build step:

.NETCore Runtime Buildpack 0.0.135
no compatible versions found
ERROR: failed to build: exit status 102

I debug and try to understand lifecycle detect source code, but I still don't locate the root cause. I suspect go source code is not suitable for paketo. I am not sure, because I saw a message for go buildpack:

The buildpack supports building applications that use either the built-in Go modules feature or Dep for managing their dependencies. Support for each of these package managers is mutually-exclusive. There is no support for applications that do not use a package manager.

About above two issues, I have asked for paketo community guys. Waiting for answer from them, now.

@SaschaSchwarze0
Copy link
Member

@zhangtbj I do not think it is "just the replacement of the image". It is more and @xiujuan95 is outlining those in the description at the beginning:

First, for paketo, at prepare step, we should make user 1000 to own /tekton/home file first. Because the docker credential is stored here. If we don't do this, then when access this secret will produce permission denied issue.
And also we should run with user 1000 at export step. Because only user 1000 has the authority to access docker credential file instead of user 0.
Second, at export step, --helper parameter has beed removed for paketo
Because the version of lifecycle in paketo image is 0.7.3 and heroku is 0.6.1. For this 0.7.3 version, the docker credential helper has been removed, so this parameter also doesn't be needed.

From a build project perspective, the different buildpack strategies are just samples. Whether a consuming team is using all off them or decides for one is their decision.

@zhangtbj
Copy link
Contributor

Yep, the change which Zoe introduced is not the key difference for both builders, and I think if heroku is upgraded to the new version, we also need to change the parameters. All the steps are same just 1 or 2 configurations.

And if they are just samples, do we need to maintain and verify the tests both our them in our side?

The reason why we decide to hardcode the builder image in the buildstartegy (#157) is we don't want the end-user to choose from multiple builder images and end user may also don't know the difference of them.

And we also talked about in the meeting, some of the source code can be built by paketo correctly but fail by heruko. If we received some tickets about this case, should we ask end user to switch another buildpacks builder to retry? Or let us fix both of the build failures for heroku and paketo?

@SaschaSchwarze0
Copy link
Member

/ok-to-test

@openshift-ci-robot openshift-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 13, 2020
@xiujuan95
Copy link
Contributor Author

xiujuan95 commented May 13, 2020

I got an answer from communty guys:https://paketobuildpacks.slack.com/archives/CULAS8ACD/p1589352322139600?thread_ts=1589349077.139200&cid=CULAS8ACD

They also think this go sample is not correct for paketo. Because it's lacking of go.mod file or something. And in this go sample folder, these files don't exist:

======== Results ========
pass: paketo-buildpacks/[email protected]
skip: paketo-buildpacks/[email protected]
skip: paketo-buildpacks/[email protected]
skip: paketo-buildpacks/[email protected]
pass: paketo-buildpacks/[email protected]
pass: paketo-buildpacks/[email protected]
pass: paketo-buildpacks/[email protected]
pass: paketo-buildpacks/[email protected]
skip: paketo-buildpacks/[email protected]
skip: paketo-buildpacks/[email protected]
skip: paketo-buildpacks/[email protected]
skip: paketo-buildpacks/[email protected]
skip: paketo-buildpacks/[email protected]
skip: paketo-buildpacks/[email protected]
skip: paketo-buildpacks/[email protected]
Resolving plan... (try #1)
skip: paketo-buildpacks/[email protected] requires jvm-application
skip: paketo-buildpacks/[email protected] requires jvm-application
skip: paketo-buildpacks/[email protected] requires jvm-application
skip: paketo-buildpacks/[email protected] requires jvm-application
fail: paketo-buildpacks/[email protected] provides unused jdk
Resolving plan... (try #2)
skip: paketo-buildpacks/[email protected] requires jre
skip: paketo-buildpacks/[email protected] requires jre
skip: paketo-buildpacks/[email protected] requires jre
skip: paketo-buildpacks/[email protected] requires jvm-application
skip: paketo-buildpacks/[email protected] requires jvm-application
skip: paketo-buildpacks/[email protected] requires jvm-application
skip: paketo-buildpacks/[email protected] requires jvm-application
fail: paketo-buildpacks/[email protected] provides unused jdk
Resolving plan... (try #3)
skip: paketo-buildpacks/[email protected] requires jvm-application
skip: paketo-buildpacks/[email protected] requires jvm-application
skip: paketo-buildpacks/[email protected] requires jvm-application
skip: paketo-buildpacks/[email protected] requires jvm-application
skip: paketo-buildpacks/[email protected] provides unused jre
fail: no viable buildpacks in group
======== Output: paketo-buildpacks/[email protected] ========
failed
======== Results ========
pass: paketo-buildpacks/[email protected]
fail: paketo-buildpacks/[email protected]
======== Output: paketo-buildpacks/[email protected] ========
failed
======== Results ========
pass: paketo-buildpacks/[email protected]
fail: paketo-buildpacks/[email protected]
======== Output: paketo-buildpacks/[email protected] ========
no "go.mod" found at: /workspace/source/assets/golang/go.mod
======== Results ========
pass: paketo-buildpacks/[email protected]
fail: paketo-buildpacks/[email protected]
======== Output: paketo-buildpacks/[email protected] ========
failed detection: no Gopkg.toml found at root level
======== Results ========
pass: paketo-buildpacks/[email protected]
fail: paketo-buildpacks/[email protected]
======== Output: paketo-buildpacks/[email protected] ========
no proj file found
======== Output: paketo-buildpacks/[email protected] ========
*.runtimeconfig.json file not found and expecting only a single *.csproj file in the app directory
======== Results ========
pass: paketo-buildpacks/[email protected]
pass: paketo-buildpacks/[email protected]
pass: paketo-buildpacks/[email protected]
pass: paketo-buildpacks/[email protected]
pass: paketo-buildpacks/[email protected]
skip: paketo-buildpacks/[email protected]
fail: paketo-buildpacks/[email protected]
======== Output: paketo-buildpacks/[email protected] ========
no "composer.json" found in the following locations: [/workspace/source/assets/golang/composer.json /workspace/source/assets/golang/htdocs/composer.json]
======== Results ========
pass: paketo-buildpacks/[email protected]
pass: paketo-buildpacks/[email protected]
pass: paketo-buildpacks/[email protected]
skip: paketo-buildpacks/[email protected]
fail: paketo-buildpacks/[email protected]
======== Results ========
pass: paketo-buildpacks/[email protected]
Resolving plan... (try #1)
fail: paketo-buildpacks/[email protected] provides unused httpd
======== Results ========
pass: paketo-buildpacks/[email protected]
Resolving plan... (try #1)
fail: paketo-buildpacks/[email protected] provides unused nginx
======== Results ========
fail: paketo-buildpacks/[email protected]
ERROR: No buildpack groups passed detection.
ERROR: Please check that you are running against the correct path.
ERROR: failed to detect: no buildpacks participating

They suggest me to use this one. I verified and passed. So now, I can say paketo also supports golang runtime.

@SaschaSchwarze0
Copy link
Member

@zhangtbj I agree that from an end user perspective, at the end, just "buildpack" should be exposed. Whatever builder is below should not matter. But, at this point in time, I think we do not yet know which one this will be: we also want to investigate the Google Buildpacks, we might even end up to use the one or other builder depending on the programming language.

BTW: I think kaniko vs buildah is the same. These are different tools yes, but similarly to the different builders in buildpack, they are solving the same problem which is doing a Docker build. The end user at the end should not care = he must not know the details about these tools. Still it makes sense that this project here contains samples for both.

@zhangtbj
Copy link
Contributor

Yes, that is the point.

That is why I agree we can have two co-existing buildpacks buildstrategies just now if all agree.

And then we also continue investigating the Paketo and even Google buildpacks.

But if we ALL agree any one is best. I think we should choose one as the buildpacks strategy, just one.

We cannot provide many buildpacks buildstraties in the repo when there is a new buildpacks providers, maybe there is heroku, paketo, google, ibm, aws......

@sbose78
Copy link
Member

sbose78 commented May 15, 2020

Even though Kaniko and Buildah builds Dockerfiles, the internals are fairly different and users are opinionated on which ones they would want to use. That's why I have kept them distinct :) we could revisit that.

However , in case of Buildpacks, as a project, we should make something our "default" buildpacks strategy. It's more of a 'opinion' the project should stand by :)

If we feel that Paketo makes more sense, we should have Paketo as the "buildpacks-v3" strategy and the other one named as "heroku-buildpacks-v3" strategy.

This is (somewhat) similar to the way "source-to-image" strategy and the "redhat-source-to-image" strategy has been added.

@zhangtbj
Copy link
Contributor

zhangtbj commented May 15, 2020

Good idea and agree

And all of our test and document should just use the "default" buildpacks strategy.
(And we don't need to maintain for two)

We have verified that Paketo makes more sense.

If all agree, we can modify this PR like this style

@qu1queee
Copy link
Contributor

@zhangtbj I agree, but this does not mean heroku needs to be removed, it just means heroku is not longer the standard for the project, therefore when we speak about "buildpacks-v3" we mean paketo, and that would be all. @xiujuan95 I think your PR is almost good, I will review now to add some comments. I already fixed the contextDir in another PR, just be aware of that.

Copy link
Contributor

@qu1queee qu1queee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work @xiujuan95 , very close to merge this.

README.md Outdated Show resolved Hide resolved
@sbose78
Copy link
Member

sbose78 commented May 15, 2020

Correct @qu1queee !

@qu1queee qu1queee self-requested a review May 18, 2020 08:52
@SaschaSchwarze0
Copy link
Member

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label May 18, 2020
@qu1queee
Copy link
Contributor

/approve

@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qu1queee

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 18, 2020
@openshift-merge-robot openshift-merge-robot merged commit 28b684c into shipwright-io:master May 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants