Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(v2): generate cache key and hash key #5849

Merged
merged 5 commits into from
Jun 17, 2021

Conversation

capri-xiyue
Copy link
Contributor

Description of your changes:
Fixed #5816, generate cache key and hash key

Checklist:

@google-cla google-cla bot added the cla: yes label Jun 11, 2021
@capri-xiyue capri-xiyue requested review from neuromage and Bobgy and removed request for Ark-kun and Bobgy June 11, 2021 21:15
Copy link
Contributor

@Bobgy Bobgy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we move cache related code to a new cache folder? We will want to reuse them, and launcher already includes too much.

I feel a little reluctant for skipping unit testing. E.g. the unordered item issue is subtle and may be hard to catch.

}

func (l *Launcher) generateFingerPrint(cacheKey cacheKey) (string, error) {
b, err := json.Marshal(cacheKey)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this guarantee unordered items have a stable order? e.g. different input parameter names

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The json package always orders keys when marshalling. Specifically:

  1. Maps have their keys sorted lexicographically
  2. Structs keys are marshalled in the order defined in the struct

FYI: https://stackoverflow.com/questions/18668652/how-to-produce-json-with-sorted-keys-in-go and https://golang.org/pkg/encoding/json/#Marshal.

Copy link
Contributor

@Bobgy Bobgy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, you can use this executor input as reference

want: &pipeline_spec.ExecutorInput{

@capri-xiyue
Copy link
Contributor Author

capri-xiyue commented Jun 14, 2021

Shall we move cache related code to a new cache folder? We will want to reuse them, and launcher already includes too much.

I feel a little reluctant for skipping unit testing. E.g. the unordered item issue is subtle and may be hard to catch.

The json marshall package handles the unordered item issue. I don't think we need UT for now to verify the functionality of json marshall package. In addition, same logic are used in KFP V1

func generateCacheKeyFromTemplate(template string) (string, error) {
.
I'm fine to add UT if we want to verify the logic of generate cacheKey struct based on the runtime info(adding a UT for this method
func (l *Launcher) generateCacheKey(
)

Copy link
Contributor

@Bobgy Bobgy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

v2/component/launcher.go Outdated Show resolved Hide resolved
v2/cache_utils/cache.go Outdated Show resolved Hide resolved
@Bobgy
Copy link
Contributor

Bobgy commented Jun 17, 2021

/lgtm
/approve
Looks great!

@google-oss-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bobgy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-robot google-oss-robot merged commit 6f96e74 into kubeflow:master Jun 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[v2compat] generate cache key and calculate hash of the cache key
3 participants