omitempty corrections

[Tom-Newton]

Purpose of this PR

Some corrections to make a couple of required fields required and optional fields genuinely optional.

Proposed changes:

• Fix [BUG] minor: Several sparkUIOptions have become required #2215. This problem was caused by some optional fields in the CRD not using omitempty. That meant that valid SparkApplication configs got nulls added were they weren't allowed during some encode and decode in the SparkApplication default webhook.
• Use regex \+optional\n^(?!.*omitempty).*$ to find all the optional parameters that don't use omitempty and add omitempty to them, to avoid the same problem in other places.
• Remove omitempty on metadata and spec fields of SparkApplication and ScheduledSparkApplication. I think these should be required and they used to be in the past. and I suspect this may have been changed accidentally. Obviously let me know if there is good reason to add omitempty on these.
• Write unittests for both of the above. I wasn't really sure how to do this but I came up with something. I don't know if its a good way to do it, but it adds the required coverage. My approach was to read the openapiv3 schema from the CRD and validate against that.

Change Category

Indicate the type of change by marking the applicable boxes:

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Checklist

Before submitting your PR, please review the following:

• I have conducted a self-review of my own code.
• I have updated documentation accordingly - I don't think its applicable
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

I'm still a golang noob.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chenyi015 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ChenYi015]

Remove omitempty on metadata and spec fields of SparkApplication and ScheduledSparkApplication. I think these should be required and they used to be in the past. and I suspect this may have been changed accidentally. Obviously let me know if there is good reason to add omitempty on these.

@Tom-Newton Since v2, we use kubebuilder to create api for SparkApplication/SchedulerSparkApplication with the command like follows:

kubebuilder create api --version v1beta2 --kind SparkApplication

and the omitempty annotation exists by default.

[Tom-Newton]

@Tom-Newton Since v2, we use kubebuilder to create api for SparkApplication/SchedulerSparkApplication with the command like follows:

kubebuilder create api --version v1beta2 --kind SparkApplication

and the omitempty annotation exists by default.

Thanks for the info. Does that mean that mean we should keep them as they were? or is there somewhere else that is source of truth to mark them as required when running kubebuilder?