-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support MUST / MAY / SHOULD responses #153
Labels
Comments
This was referenced Aug 29, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
A spec might allow for multiple responses depending on the gateway's capability.
For example, the specs for path gateway - range request header allows a gateway to return either:
We solve this by running a first check that requests an endpoint, stores the response's header through a side-effect, then uses an
if / else
flow to test between these capabilities (code).@aschmahmann is working on adding more of these tests across the suite; before implementing many more, we might want to support this natively through the sugar API.
Also, since two gateways might pass the specs with very different feature sets, it would make sense to signal which features are supported in the conformance output.
Problem 1: Support many possible responses
At the moment, you may define a test with:
We could add
AnyOf
operator:That would pass if the actual response matches any of the responses.
We could implement this as a /simple/ "Or" operator.
This opens up more complex operators like
Problem 2: Surface MUST / MAY / SHOULD features
Rough idea:
This will signal to users/implementers which features are supported or not.
It might be subtle to represent and evaluate these in go code: some of these would be "additive", this is how specs are written; we'd define three expectations, and the gateway MUST pass some, SHOULD pass some, and MAY pass others.
But in the case of range requests, the expected responses are mutually exclusive; if the gateway returns the minimal dag that matches the range requests (SHOULD), then the response will be very different from the gateway that returns the whole dag (MUST). So the
AnyOf
operator should take these into account during evaluation:Early exit
We'll have to be careful with
AND/AllOf
and early exit: if the first predicate fails, it's intuitive to exit the whole evaluation early, BUT in the case of testing, we should continue running all the tests in order to preserve the test counts.Related
The text was updated successfully, but these errors were encountered: