This document mostly gathers thoughts from the current CNCF OKR’s around apisnoop and istio, knative conformance.
There are three tracks in our cncf okr project board that I will have strong involvement in: APISnoop, Istio, Knative. APISnoop relates to the site and testing framework we use for kubernetes conformance. Istio, and Knative are CNCF projects that are run on top of kubernetes, and are also looking to start a conformance project.
We want to extend our work on APISnoop to include Istio and Knative conformance. We also want to extend the ownership of APISnoop to the larger k8s community, so that we can donate the project, ultimately, to kubernetes.
I think our work will be strongest if we remember the intent of APISnoop as it related to the workflow and goals of kubenrets confomrance. Then, customize our tools to serve the same intent for these new conformance projects, customized to their unique release cycles and develpment flows.
I believe we will end up using the components of our current APISnoop, but in new configurations that better serve these communities.
Technically, Snoop provides a db used for test-writing and querying the state of kubernetes conformance and a website for displaying these results.
Non-technically, it provided measurements, goals, and a timer for the kubernetes conformance.
One of its biggest values is the distinction between kubernetes versions, and always showing the latest dev version of kubernetes. It is simple, but it meant that whenever you visited the homepage you saw the work that was needed before the next code freeze. It added to the urgency of the conformance goals.
Another huge value in it was the conformance progress graph, showing the technical debt, the progress we made, and how the velocity of that progress was increasing. It was essentially a graph that summarized why the project should be funded.
If we simply extended the db so that it showed various CRD endpoints, or extended the site so someone could see the sunburst with istio endpoints on it, we would miss onthe big values we could bring to Istio or Knative. Both projects need tools that fit the rhythms and goals of their project. The test coverage should be based on the project’s release cycles, and the progress should be based on their own goals. Do they operate with the same rhythms as kubernetes in general, with the same notions of code and test freezes?
More generally, I think our experience with APISnoop helps us see the components needed for a conformance project: the shape of the data for the test suites, the test writing environment, the way check-ins are structured, how to measure success and velocity, etc. We can use this knowledge to give Istio and Knative the best start possible.
The primary goal for the APISnoop handoff should be to simplify the tool so it is a generic framework for kubernetes conformance projects.
We can define the components of the site and tool that are important, and try to make them self-contained and easier to reconfigure.
The three streams of work: Apisnoop, istio conformance, and knative conformance, can happen simultaneously to each other, with the cleanup work in APISnoop making it easier to build something for Istiosnoop, and our learnings from the istio community helping guide how we simplify snoop.
Practically speaking, what this would look like is a hardcore cleanup of the APISnoop repo, removing some historical org files and various deployment experiments until we have a simple, unified way to develop and deploy it. Then, a walk through the codebase to make sure its existing components have generic inputs, with nothing hard-coded. This will likely result in a config file that is passed to the db or site on deployment, that say what datasources to pull from and how things should be labelled, etc.
Similarly, we’ll analyze the existing site to better define the necessary components, and rearrange the codebase around these components. The goal is to be able to make sites (or pages within the existing site) that are tailored to these other projects, but without having to build a bunch of new tools to do so. I imagine the KnativeSnoop and IstioSnoop will look familiar, but be meaningfully different from APISnoop, as they are designed around different releases and cycles.
Our work with istio/knative will require some collaboration with their community, interviews with folks to understand what they are trying to achieve iwth conformance, what their tests look like now, etc. We want to find our target users in each community, then use them as a guide and feedback for our work. We can also consult on how to set up some sort of test suite so that we have continual data to pull from for our work.
Lastly, I think there should be work put into making a more generic, accessible entrance into the test-writing side of snoop. Specifically, a web frontend associated to a space that shows the results of the current test alongside useful metadata around the API. It would be a mix of the queries we have now in org-mode, plus Stephen’s cool web-based org workflow, plus a version of the APISnoop page that uses similar components but is scoped to that particular instance and test.
With this holistic approach, I think we can end up with something special that can be altered and deployed for cloud-based projects beyond Istio and Knative, maybe some that don’t exist yet, but will soon, and will need confomrance.
Both projects follow the same feature lifecycle as kubernetes, with endpoints graduating from experimental to stable, with the endpoints naming following the same structure as kubernetes.
- [Istio Fe]
- Istio has the same feature release cycle as k8s, same versioning and same feature freeze prior to release. The release flow is outlined here:
https:/istio/istio/blob/master/RELEASE_BRANCHES.md
- Their release flow is outlined in their wiki, with ecah release getting its own wiki page: https:/istio/istio/wiki/Istio-Release-Management
- Releases happen roughly every three months
- Releases are managed, generally, by the Technical Oversight Committee: https:/istio/community/blob/master/TECH-OVERSIGHT-COMMITTEE.md
- TOC assigns new release managers for each release. It is a transient role, but a person may be chosen multiple times: https:/istio/community/blob/master/ROLES.md#release-manager
- The technical oversight committee meets regularly
- agenda here: https://docs.google.com/document/d/13lxJqtlaQhmV2EwsNnS6h-_O4pobZQZuMjrzOeMgVI0/edit#heading=h.o8pz6aqnzzgk)
- They meet monthly, roughly third week of the month, 4,5am Tuesday NZT time.
- next meeting, 15 August
- Knative has a release repo for tracking the steps needed for a release: https:/knative/release
- they release quarterly, on the 4th week of January, April, July, October
- release leads are also rotating, handled via a self-nominating pr
- their procedures are outlined here: https:/knative/release/blob/main/PROCEDURES.md
- Knative harder to track, it’s microservices all the way down and so there isnt’ one knative/knative repo. Key repos are eventing and serving.
- The best meeting to attend is likely the community meeting, but they have a set of WG meetings too, and a TOC overseeing the whole project https:/knative/community#meetings-and-work-groups
- All meetings are roughly “pleasantly in the morning, Pacific time” and so around 4 or 5 am NZT.