Refine Release Team Selection Process #1714
Replies: 17 comments 28 replies
-
|
Just for giving context: I have applied three times and I've been in both status of applying for being a Shadow: been accepted and not. And talking about those specific points mentioned above, here are my thoughts/proposals: Reviewing hundreds of applications
Tracking applications
About the other points, I don't have enough experience or more ideas for now but I'll submit ones during this Shadow path. |
Beta Was this translation helpful? Give feedback.
-
Would it help to have a more stable pool of people within the release team? For example, we could achieve that by maintaining a stable release team roster while evaluating at the end of the cycle if the members are still interested to be part of it. Shadows could become either members of that roster or work on something else after a cycle ends. One of the main tasks of the roster could be to work on the retrospective items proactively or onboarding new folks and shadows. The leads would have more responsibility to improve the actual release process while being able to delegate more tasks to the other members and shadows of the roster. The roster itself is not bound to a particular release any more, but the lead as well as shadow selection is. |
Beta Was this translation helpful? Give feedback.
-
|
I didn't want to solutioneer, and propose something before discussing the problems we're trying to solve, but yes I suspect that some kind of stable team roster is a sensible move. Something that supports not only people that want to do release team work every cycle, but people that want to take a break for a cycle and come back without worrying about going through the shadow application again. I wonder what the effect on the role subteams would be though. If each subteam becomes a role lead with a few "roster" folk and a shadow or two each cycle, then is the emphasis on the lead for each role still required if you have 3 or four people in each role team that could lead, or at that point is it a question of the roster people deciding "who wants to be the one to turn up to the leads syncs". |
Beta Was this translation helpful? Give feedback.
-
I think "hard" selection criteria are difficult to define, the questions if they have contrib experience is the closest one but also here it is not a real criteria. Tracking applications I'm a fan of: My thought was (and there is somewhere from me a slightly over engineered picture to it) that we have a kind of scoring system. E.g. every applicant has like 3 points, with every application and no selection this goes down by one. If reached 0 the applicants are filtered out for backup but no active consideration. (Hard selection criteria?) Why that? If an applicant didn't got selected within 3 releases, at least 3 (max. 9/12/) leads considered the application as a non fit. That maybe feels hard but I believe we have to focus.
I like the idea of the roster, but I'm not sure of the tasks. I would see it more like an leveraged step between releases: application -> shadow -> roster -> shadow/lead -> roster etc. This way, people who are tracked on the roster are more visible for upcoming release but also have to Opt-In for it. In addition this can allow people who maybe can't support a release (new job/family/rest) to be re-considere the release cycle after the next. While writing this, then they still could carry out the tasks from retro, onboarding etc. Back to the main points: Also somewhat over engineered, I guess we have to have history overview of the applicants, based on this data we can slice rosters, etc. I believe to create some scripts like in google sheets shouldn't be an issue. To be honest, I'm not a fan of up & out. Maybe we have to discuss some criterias here like a minimum 4 releases before entering the release lead team or must have of two times leading a sub team of the release. This can also help the future lead as the person has seen a lot of different situations and knows for example at least 2 roles inside out. |
Beta Was this translation helpful? Give feedback.
-
What would be the benefit of going back to shadow when being already part of the roster? Is it:
Because then I see no reason why not being part of the roster, when considering to come back in 1-2 releases. So the general idea is: Less rules, but 1 more role to increase the amount of people taking responsibility for the release.
I think that's a great idea and would be totally achievable if we imagine that the team size increases as well as the stability of people participating in the process. |
Beta Was this translation helpful? Give feedback.
-
|
I see the roster as a group of advice people who were part of a release and are interested in a) being part of the next releases or b) supporting the release but can't fulfill the time schedules. Having the "roster" tag means someone with experience of SIG release. But maybe I don't get the whole roster thing and the role/tasks need to be more sorted out. |
Beta Was this translation helpful? Give feedback.
-
|
Pinging the previous release team leads and emeritus advisors and kindly asking for their input: @claurence @guineveresaenger @alejandrox1 @lachie83 @onlydole @tpepper @jeremyrickard @palnabarun @savitharaghunathan @reylejano |
Beta Was this translation helpful? Give feedback.
-
|
Dropped a note to k-dev as well: https://groups.google.com/g/kubernetes-sig-release/c/pebN2IY-ZIs/m/KFIY-uVKAwAJ |
Beta Was this translation helpful? Give feedback.
-
|
I am overall +1 for stable release team roster + some dedicated teams. For posterity here is a link to the slack thread with some discussion on it from last week: For each "team" there could be at 3-4 (eventually more) dedicated people (This should be a defined role/team). One of them could take point for each release to function as the "lead" for that release cycle, while the others are free to work on automating and improving the process for their area - this could include working with previous shadows to take on those tasks. Some general benefits:
This does mean that the shadow program might take a hit in growing new folk, but we shouldn't perpetuate a process that should be improved/automated for the sake of bringing new contributors. (That probably sounds bad coming from a contribex lead XD but its true) One other thing I want to see for the shadows is actual accountability. If they are not an org member they are granted it without having to go through our standard process of contributing first. There should be an expectation that if they do not fulfill their duties as a shadow (e.g. just stop showing up / communicating with lead), they are removed from the org and have to work towards being added again. |
Beta Was this translation helpful? Give feedback.
-
I think this would be good - during my time on the release team I observed this multiple times. |
Beta Was this translation helpful? Give feedback.
-
I like this idea, from a longer term sustainability perspective. I agree that this would provide a little more consistency on the automating/improving front, which seems to really struggle from cycle to cycle, and gives better perspective across multiple releases. Somethings, like CI Signal, probably also would really benefit from longer term pattern observation and more in depth knowledge of tracking down flakes/failures. I also think it also makes it a little easier to align with some of these functions in other SIGs:
In terms of shadowing, perhaps something more like the release manager associate model where we onboard people for a longer haul and then re-evalute through surveys and off board folks when their availability/wants change, then onboard new folks. And perhaps these longer term things could be a good two way pipeline? Shadows could come from the SIGs (maybe this is one source, and we solicit new contributors as well) and we could also build folks into longer term contributors for those areas. |
Beta Was this translation helpful? Give feedback.
-
+1 for this. There needs to be some level of accountability. It also helps with avoiding other members of the team from getting burnt out. |
Beta Was this translation helpful? Give feedback.
-
|
It's better to restrict who can apply than to aggressively kick people off. I'm not sure that org membership should be a requirement, but we could do other things to narrow the eligible pool in a way that makes it more likely that the shadows will succeed. Making the RT positions interview replacement Shadows through the whole release cycle would not be an improvement. |
Beta Was this translation helpful? Give feedback.
-
IssuesFor the 1.23 release cycle we had several issues:
Typically, Kubernetes org membership is earned based on multiple contributions and sponsored by 2 reviewers. K8s org membership is granted because some teams require org membership e.g. for milestone-maintainers or website-milestone-maintainers. Comments/thoughts on possible solutionsIncrease gender diversity in shadow applicantsWe need to be more inclusive and extend an invitation by:
Lack of qualified shadows for succession / lack of shadow contributionThe shadow program is a two-way street meaning that the Lead, Lead Shadows, Role Leads provide mentorship for shadows to succeed but shadows have duties to fulfill. Shadows should be held accountable for not fulfilling duties. I agree with Bob's comment:
I suggest not granting K8s org membership (and maybe not grant formal release team membership) to new shadows until their contributions have earned org membership. I know some teams require org +1 to Bob's comment:
Teams that do not require org membership can include shadow positions where shadows can contribute and earn K8s org membership based on contributions. Evaluating shadows' contributions for org membership keeps the standard of K8s org membership that all org applicants are subject to. There still needs to be accountability when shadows do not fulfill their duties. A release team can be more dynamic and shadows can be removed and replaced by another applicant. The risk is that we have a revolving door within a release. Aside from the Emeritus Advisor, Lead and Role Leads, I suggest to formally announce the release team at the end of the release to formally state, these are the contributors that worked on the release. For new shadows, there should be a clear path to leadership or to be on the stable release team roster / dedicated team. Having a stable release team roster/dedicated team will reduce the number of shadow positions. This will increase the barrier to entry for new contributors. Part of the SIG Release charter is to ensure a consistency of community members for releases. I believe people took advantage of the lower barrier to entry to benefit their own agenda and did not contribute to releases which reduced the consistency of community members to support releases. For reference, in the SIG Release charter under Scope:
Having a stable release team roster/dedicated team will increase consistency of community members to support releases. Difficult shadow selection processFor 1.23, we had 185 shadow applicants for ~26 shadow positions. Selecting shadows is one of the most difficult tasks for Role Leads. There are applicants who have applied multiple times but have not been selected. Applicants who were not selected may benefit from a having a section on What if you were not selected and how to contribute to increase your chance to be selected next time in the release-team-selection.md to strengthen their future application. I believe tracking applicants would be beneficial but takes effort to initially create or apply a solution. |
Beta Was this translation helpful? Give feedback.
-
|
Hey, all: Based on my convo with Eddie at Kubecon, one thing which apparently got dropped from the shadow selection process sometime in the last several releases is doing video interviews of the final pool of shadow applicants. That was part of the process through at least 1.14, and is essential to making sure that you've selected shadows who are actually interested in doing the work. So whatever revised process you come up with must include final video interviews, or you'll keep picking shadows who bail two weeks into the cycle.
No. No, no, no, no. This is what SIG-Release is for. If this isn't what SIG-Release is doing, then that's a problem with SIG-Release and not with the Release Team. And if it's not what SIG-Release is doing, then what is it doing? The whole reason we created the rotational structure of the release team is that burnout was a very real thing on the early release teams. Do you see Anthony around anymore? So having anyone "permanently" on a Release team means committing to burning someone out. Dedicated teams for other things sounds like a fine idea, things like release instrumentation. But even these teams should be oriented towards self-replacement, just over a longer period of time. Like Release Engineering, those teams should have their own shadows. |
Beta Was this translation helpful? Give feedback.
-
|
I was a part of the RT cycle from 1.18-1.22 and wasn't aware of the video interviews concept at all. I think that will definitely help with the shadow selection. +1 to bringing it back. |
Beta Was this translation helpful? Give feedback.
-
Great point, Josh! 100% agree on this. Sometimes being on a release team could feel like a full time job, especially for the sub team leads,RT lead + RT lead shadows, and enhancements team. This could cause severe burnout. |
Beta Was this translation helpful? Give feedback.
-
|
To my mind the idea is not "construct a team of people, who shall forever more do release work". However I do feel that the release team as a whole is too focused on self-replacement, leading to problems where it can be hard to "feel at home" on the team or build expertise. Can there be a happy medium between those extremes? A structure that allows contributors to remain on the release team for multiple cycles easily, while taking breaks if they wish and still bringing new people in? More generally, isn't the release team the only community position that operates in this way? Release managers, SIG tech leads, SIG chairs, even positions with formal elections like TOC and steering committee don't have such an aggressive churn right? |
Beta Was this translation helpful? Give feedback.
-
|
Event teams have the same setup, and for the same reasons. During various periods in the release, particularly the last three weeks, the workload for most of the RT is such that it's not possible to do anything else -- one of the things we at least used to tell shadows was that they needed permission from their boss to be on the RT, because it was going to affect their availability for their job. Lots of people can do that for one release cycle. If we ask people to do that for multiple release cycles per year, we narrow down the pool considerably, mostly just to staff of the biggest vendors. And, based on real-world experience, some people voluteer to do that despite it being a bad idea and burn out. We should not be encouraging burnout. Again the idea of the Release Team "churn" was that folks were supposed to peel off the release team and the spend time working on other priorities for SIG-Release, often ones that were more long-term and less intensive. For example, I went from Release Lead to WG-LTS, which we then worked on for 1.5 years after that. Apparently that system is not working anymore, but if it's not working, why? The idea of "permanent RT members" really sounds like a problem with SIG-Release that folks are trying to fix by tinkering with the Release Team. At the very least, there should be discussion of SIG-Release as a whole, and not treating the RT like it's completely separate. |
Beta Was this translation helpful? Give feedback.
-
|
We should also expect that people peel off into other SIGs, not just SIG Release. The release team gives a newcomer a chance to see some breadth in the project and find a place for which they have an affinity. Still I agree that between SIG Release, Release Management subproject, and Release Team(s) subproject(s...if you think of each as a project) should together represent continuity versus installing some level of permanence in the Release Team: I like that the release teams are "up and out", in that it has the potential to be an entry point into the project. But I also dislike it. IMO there can't be a pre-ordained pace through which one enters and departs. It can't be just that people are up and out into SIG Release specifically. Ideally in my mind too it's more than just 3-6 months. It takes some time to learn. It takes some time to build ones replacement (directly 1:1 and in small groups and through the docs they will leverage). And wedged in between those too hopefully is some duty to contributing to continuity. And at the same time if some folks aren't in for a tad longer period period or are not up and out into specifically SIG Release, then there is insufficient continuity. |
Beta Was this translation helpful? Give feedback.
-
|
Based on @savitharaghunathan history, current "release team tenure" is around 1.5 years. |
Beta Was this translation helpful? Give feedback.
-
|
I think there is some tension between the release team as a way of onboarding new people, and the release team as a way of providing a quality experience to other contributors. To my mind, we're focused too much on the former. Exactly how we formulate it (e.g., more in SIG Release vs a consistent release team) I'd like to enable people to stick around and build expertise, and I don't feel the current system encourages that. |
Beta Was this translation helpful? Give feedback.
-
|
As for the flood of Shadow applications, one solution is to require applicants to made Kubernetes contributions before applying to the RT. Minor stuff, like helping with issue triage, for example. We didn't do that in 2019 because getting enough shadow applications was the challenge then, but if we have an excess of shadow applicants and not enough people doing things like triage ... then that seems like a reasonable transfer of recruits. SIG-Contribex would need to be involved, of course, to make that work, but that is what we're supposed to be doing in ContribEx. The RT is getting this flood of applicants because it's Kubernetes' only clear on-ramp to becoming an accredited contributor that has an application form. Before COVID we were looking at other areas of the project where we could do the same, and maybe it's time to revive that. |
Beta Was this translation helpful? Give feedback.
-
Believe it or not, I'm going to suggest that the solution to this is having fewer shadows. I've spoken up about this before; I really don't think that in most roles (there are exceptions) one role lead can adequately mentor 4-5 shadows. I know I personally can't mentor 5 people at once well. I really think the default should be 2 shadows per lead, because with two shadows you can't "lose track" of either of them. If having a "bench roster" lets the RT have fewer shadows with confidence, then that seems like a good idea, and it's easily assembled out of all of the folks who have held that position before. |
Beta Was this translation helpful? Give feedback.
-
|
This resonates with me because it was my experience. @jberkus reached out and said hey we need people like you. It was the middle-ish of the 1.9 cycle. He said show up. Start learning. Contribute. So I did. Next cycle I shadowed formally, but I brought myself to the bench ahead of that. Because Josh led me to it. This is sometimes called sponsorship, a more active thing than just information sharing in mentorship. It's not a scalable process in itself. But if each of us does it once or twice a year, it's VERY impactful. In a few months I went from "zomg this project feels impenetrable" to being seriously in it. It's critical to have an open process and not just have selection bias for those we know. But how much is SIG Release doing sponsorship today? We should be spotting the repeat presences on the candidate list, bringing them to the bench out of cycle, helping the bench build the contribution history which makes the individuals the obvious choice for a specific role. And if not chosen...surprise in the meantime the individuals are active leaders in SIG Release, skipping right past the release team member titles. |
Beta Was this translation helpful? Give feedback.
-
|
@jberkus How do you feel about having only 2 (ish) shadows on each team, two "experienced" members, and one lead? Which is realise a codification of how we typically operate right now (usually people don't select four brand new people as shadows, but will select a couple of regulars and a couple of new people.) |
Beta Was this translation helpful? Give feedback.
-
|
@JamesLaverack that's pretty much what we're already doing. That said, shadowing the same position for more than one cycle should be the exception, not the rule. If someone actually knows the job, they are a team member and not a shadow. I know there are times when someone doesn't quite "get" the job on the first runthrough, but I'd see an end result where people are perpetual shadows as a failure of the process. It's an apprenticeship. Which raises another idea: for positions where there is enough work, there's no reason why the team needs to be one person. Consider making some positions, like Enhancements or Release Notes, more of a "team" with a lead, a couple members, and a couple shadows, instead of just lead/shadows. This doesn't help, of course, for teams where there pretty much only is one experienced persons' worth of work. |
Beta Was this translation helpful? Give feedback.
-
@jberkus what's the difference between this and #1714 (comment) ? |
Beta Was this translation helpful? Give feedback.
-
|
Because that team doesn't address the difference between role leads and shadows ... it still assumes just one role lead. Having some kind of roster of folks who have RT experience is orthagonal to how many people get to be leads as opposed to shadows. |
Beta Was this translation helpful? Give feedback.
-
|
I agree that as a way to get org membership, the release team is... too easy? It's a bit ironic for me to say that considering that it's how I got my org membership. I'm supportive of the idea of not granting day one org membership. This might mean we need Prow changes (e.g., to enable it to accept commands from release team shadows who are not org members) for some roles (e.g., enhancements). Instead I'd like to see it as a path to satisfying the normal org membership requirements. |
Beta Was this translation helpful? Give feedback.
-
|
I didn't even realize that RT shadows got day 1 org membership. This seems contradictory to our org membership guidelines, which state that org members should have a sustained history of contributions, which is not the case for a new RT member who has not previously contributed. |
Beta Was this translation helpful? Give feedback.
-
|
@ehashman not necessarly day 1, but certainly week 2. There's several roles on the RT which aren't possible to perform without being able to send commands to automation -- such as Enhancements. However, this isn't true of all roles. Also, we could handle this with a "brevet": that is, someone gets their org membership on week 2, but they go on a list of org memberships to revert after week 16 ... unless they finish out the release, in which case they get to keep it. |
Beta Was this translation helpful? Give feedback.
-
This hasn't been happening =/ and really needs to be codified/enforced. |
Beta Was this translation helpful? Give feedback.
-
|
It was never a rule before, hence it was never implemented. Drop-outs back in 2018 were < 20% of the shadows, and those got dealt with individually. It's also not the first thing I'd do: see comments above about being more restrictive about who's on the RT in the first place, and reducing the number of shadows. |
Beta Was this translation helpful? Give feedback.
-
|
Could we modify the automation to accept commands from release team shadows that are not org members? Which would allow shadows to perform their roles without needing org membership. |
Beta Was this translation helpful? Give feedback.
-
|
With 1.23 scheduled to release in a few weeks, if we'd like to try changing things for 1.24 we should start to formalise a plan. Reading through the various discussions, there seem to be a handful of common themes:
Points 2 (long-term applicant tracking) and 4 (proactively increase diversity) seem relatively non-controversial (at least, I don't think I saw any arguments against) and I don't think would require a process KEP, so maybe we can just do those? With the other points, I think I've seen arguments for and against all of them in the threads above. I think 3 (stable team roster) is probably the most controversial, and I suspect probably would require a process KEP. Let me know if I've missed or misrepresented anything 😅. But hopefully we can figure out a path forward, even if that path is "no change at this time". |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for summarizing the outcome of the discussion @JamesLaverack! I agree, focusing on the shadow selection process for now seems to be the most valuable path forward for now. Let's do that and start by putting everything into documentation rather than writing a KEP. 👍 |
Beta Was this translation helpful? Give feedback.
-
|
In particular for 1.24's shadow selection (cc @jeremyrickard @jrsapi), it'd say we can roll in the non-controversial points from above. The long-term applicant tracking probably needs to happen at the SIG level across multiple releases, but we can think about how we can proactively increase the diversity of the team by ourselves. Are there things about the survey questions that we can improve? Can we advertise it anywhere in particular? In terms of the other points. Setting up a "roster" seems like a long-term goal if we are going to do it (and it's clear from the discussion we might not). I'd still like to make things easier for returning release team members. Adding to this we already suggest that role leads take two experienced shadows and two inexperienced shadows. I wonder if a first step is just to separate the entry to the team into two surveys: One for experienced team members and one for brand new shadows? This way we can ask different questions on each (i.e., the returning member survey can ask about previous releases more), and role leads can use the two lists to make selection easier (e.g., knowing they should probably pick two from each list). |
Beta Was this translation helpful? Give feedback.
-
|
FWIW, I don't think the roster requires a KEP, just an official decision by SIG-Release. |
Beta Was this translation helpful? Give feedback.
-
|
FYI we're discussing starting to design the 1.25 shadow survey ahead of time: #1814 |
Beta Was this translation helpful? Give feedback.
-
|
A few of us from SIG Release discussed this topic at the KubeCon EU 2022 Valencia Contributor Summit. You can see the notes from that discussion here. In that meeting I felt we had agreement amoung those present that we need to change something for 1.26 and beyond. I feel we should pick one problem and implement a solution, with the expectation that we will continue to iterate. The area I'd like to focus on is a more stable group of people on the release team. Both to provide release team members more continuity, and to help the rest of the community (particularly KEP authors) with a more consistent group of people to build relationships with. We've talked about a few solutions before:
I'd suggest a combination of these appraoches would be best, and is acchiveable in time for 1.26. To be clear I'm not suggesting we change anything for 1.25, as the team is already being selected. As a solid suggestion, what if we did the following:
I'm sure there are many issues, and things I haven't thought of, but i'd love to hear if folk would like to move in this kind of derection or think this is totally the wrong way. |
Beta Was this translation helpful? Give feedback.
-
|
I was in favor of this general direction in my first comment (#1714 (comment)), where I also see the different opinions like keeping the rotation to avoid burning out people (#1714 (comment)). I'm happy to outline the process changes in a KEP together with a few other volunteers, because this seems a reasonable next step for me. |
Beta Was this translation helpful? Give feedback.
-
|
I've added an enhancement for this kubernetes/enhancements#3344, and opened a work in progress pull request for the KEP kubernetes/enhancements#3347. (We got the nice memorable KEP number 3344 :D) |
Beta Was this translation helpful? Give feedback.
-
|
Note that, in the current RT, for someone to make RL they will probably be on the release team for 1.5-2 years. I'm not sure making that process longer, particularly during the shadow phase, is beneficial or justified. I was at that meeting too and I don't think I got out of the the same things you did. |
Beta Was this translation helpful? Give feedback.
-
|
Lemme back up here. What specific problems would having "permanent RT members" address? What problems would having people shadow longer address? I'm clearly not comprehending the motivations for the suggested change, which is why I keep arguing against it. |
Beta Was this translation helpful? Give feedback.
-
This discussion opened following a topic in the Tuesday 21st September 2021 SIG Release meeting.
In the most recent release cycle (1.23) the release team shadow program received 185 applicants for approximately 26 available shadow roles. The shadow program itself is, in my view, highly useful for onboarding new members to the community. However at this scale it is becoming unsustainable.
A few issues noted:
To give some data on this, if you compare the 1.23 release team to 1.20 a year earlier there are 8 out of 40 members who were on both: Rey, Jeremy, Nabarun, Max, Menna, Joseph, Xander, and myself. None of whom are shadows (I'm not counting lead shadows or branch manager shadows as "shadows" for this purpose as they don't use the same application process).
If you compare to the 1.16 release team two years ago there is only one shared member: Karen.
To address these issues, I'd like to start considering how we could improve the release team selection process. 1.24 is likely to start selection in December or January so we have time to consider options (although I don't want to rush into changes either).
I'd really love opinions on this from anyone in the community, but in particular I'd welcome input from current or former shadows, or people who have applied to be shadows multiple times (and been successful or not).
Beta Was this translation helpful? Give feedback.
All reactions