<!-- **Note:** When your KEP is complete, all of these comment blocks should be removed. To get started with this template: - [ ] **Pick a hosting SIG.** Make sure that the problem space is something the SIG is interested in taking up. KEPs should not be checked in without a sponsoring SIG. - [ ] **Create an issue in kubernetes/enhancements** When filing an enhancement tracking issue, please make sure to complete all fields in that template. One of the fields asks for a link to the KEP. You can leave that blank until this KEP is filed, and then go back to the enhancement and add the link. - [ ] **Make a copy of this template directory.** Copy this template into the owning SIG's directory and name it `NNNN-short-descriptive-title`, where `NNNN` is the issue number (with no leading-zero padding) assigned to your enhancement above. - [ ] **Fill out as much of the kep.yaml file as you can.** At minimum, you should fill in the "Title", "Authors", "Owning-sig", "Status", and date-related fields. - [ ] **Fill out this file as best you can.** At minimum, you should fill in the "Summary" and "Motivation" sections. These should be easy if you've preflighted the idea of the KEP with the appropriate SIG(s). - [ ] **Create a PR for this KEP.** Assign it to people in the SIG who are sponsoring this process. - [ ] **Merge early and iterate.** Avoid getting hung up on specific details and instead aim to get the goals of the KEP clarified and merged quickly. The best way to do this is to just start with the high-level sections and fill out details incrementally in subsequent PRs. Just because a KEP is merged does not mean it is complete or approved. Any KEP marked as `provisional` is a working document and subject to change. You can denote sections that are under active debate as follows: ``` <<[UNRESOLVED optional short context or usernames ]>> Stuff that is being argued. <<[/UNRESOLVED]>> ``` When editing KEPS, aim for tightly-scoped, single-topic PRs to keep discussions focused. If you disagree with what is already in a document, open a new PR with suggested changes. One KEP corresponds to one "feature" or "enhancement" for its whole lifecycle. You do not need a new KEP to move from beta to GA, for example. If new details emerge that belong in the KEP, edit the KEP. Once a feature has become "implemented", major changes should get new KEPs. The canonical place for the latest set of instructions (and the likely source of this file) is [here](/keps/NNNN-kep-template/README.md). **Note:** Any PRs to move a KEP to `implementable`, or significant changes once it is marked `implementable`, must be approved by each of the KEP approvers. If none of those approvers are still appropriate, then changes to that list should be approved by the remaining approvers and/or the owning SIG (or SIG Architecture for cross-cutting KEPs). -->

KEP-2420: Reducing Kubernetes Build Maintenance

<!-- This is the title of your KEP. Keep it short, simple, and descriptive. A good title can help communicate what the KEP is and should be considered as part of any review. --> <!-- A table of contents is helpful for quickly jumping to sections of a KEP and for highlighting any additional information provided beyond the standard KEP template. Ensure the TOC is wrapped with <code>&lt;!-- toc --&rt;&lt;!-- /toc --&rt;</code> tags, and then generate with `hack/update-toc.sh`. --> <!-- toc -->

• Release Signoff Checklist
• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • Risks and Mitigations
• Design Details
  • Test Plan
  • Graduation Criteria
  • Upgrade / Downgrade Strategy
  • Version Skew Strategy
• Production Readiness Review Questionnaire
  • Feature Enablement and Rollback
  • Rollout, Upgrade and Rollback Planning
  • Monitoring Requirements
  • Dependencies
  • Scalability
  • Troubleshooting
• Implementation History
• Drawbacks
• Alternatives
• Infrastructure Needed (Optional)

<!-- /toc -->

§ Release Signoff Checklist

<!-- **ACTION REQUIRED:** In order to merge code into a release, there must be an issue in [kubernetes/enhancements] referencing this KEP and targeting a release milestone **before the [Enhancement Freeze](https://git.k8s.io/sig-release/releases) of the targeted release**. For enhancements that make changes to code or processes/procedures in core Kubernetes—i.e., [kubernetes/kubernetes], we require the following Release Signoff checklist to be completed. Check these off as they are completed for the Release Team to track. These checklist items _must_ be updated for the enhancement to be released. -->

Items marked with (R) are required prior to targeting to a milestone / release.

• (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
• (R) KEP approvers have approved the KEP status as implementable
• (R) Design details are appropriately documented
• (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
• (R) Graduation criteria is in place
• (R) Production readiness review completed
• (R) Production readiness review approved
• "Implementation History" section is up-to-date for milestone
• User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
  • N/A, this is not user facing
• Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

<!-- **Note:** This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone. -->

§ Summary

<!-- This section is incredibly important for producing high-quality, user-focused documentation such as release notes or a development roadmap. It should be possible to collect this information before implementation begins, in order to avoid requiring implementors to split their attention between writing release notes and implementing the feature itself. KEP editors and SIG Docs should help to ensure that the tone and content of the `Summary` section is useful for a wide audience. A good summary is probably at least a paragraph in length. Both in this section and below, follow the guidelines of the [documentation style guide]. In particular, wrap lines to a reasonable length, to make it easier for reviewers to cite specific portions, and to minimize diff churn on updates. [documentation style guide]: https://github.com/kubernetes/community/blob/master/contributors/guide/style-guide.md -->

Kubernetes currently maintains multiple build systems, an ongoing burden and a source of contributor friction and confusion. Much has changed since Bazel was first introduced as an additional build system, upon re-evaluating the project it is clear that we should dedupe this. More details on why can be found in Motivation and Drawbacks.

§ Motivation

<!-- This section is for explicitly listing the motivation, goals, and non-goals of this KEP. Describe why the change is important and the benefits to users. The motivation section can optionally provide links to [experience reports] to demonstrate the interest in a KEP within the wider Kubernetes community. [experience reports]: https://github.com/golang/go/wiki/ExperienceReports -->

Goals

<!-- List the specific goals of the KEP. What is it trying to achieve? How will we know that this has succeeded? -->

• Remove the toil of maintaining multiple build-systems from the Kubernetes repo maintainers
• Eliminate the friction of generating BUILD files from Kubernetes contributors using Go natively
• Simplify Golang upgrades for SIG Release
• Remove qualification gaps caused by duplicate-but-slightly-different binary builds
• Remove testing gaps caused by duplicate-but-slightly-different test-invocation methods
• Empower broader community maintenance of tests by converging on Golang testing standards

Non-Goals

<!-- What is out of scope for this KEP? Listing non-goals helps to focus discussion and make progress. -->

• Support importing github.com/kubernetes/kubernetes as a library
  • This has never been supported, and is orthogonal to the decision of maintaining duplicate build systems
  • Code under kubernetes/kubernetes/src/staging/k8s.io/foo is intended to be imported as k8s.io/foo from the staged copies at github.com/kubernetes/foo; anything else is internal-only and not supported for import, bazel or otherwise.
• Removing Bazel from sub-projects other than the core repo
  • Bazel comes with tradeoffs, each subproject can make its own decision in this regard. This KEP only covers the main Kubernetes repository, and nothing else.
• Improving the previously existing make build
  • We should strongly consider improving the implementation and behavior of this build in the future, but this is largely orthogonal to whether we should consider maintaining two build systems. An anticipated outcome of this KEP is increased bandwidth available to improve our single build system.

§ Proposal

<!-- This is where we get down to the specifics of what the proposal actually is. This should have enough detail that reviewers can understand exactly what you're proposing, but should not include things like API designs or implementation. What is the desired outcome and how do we measure success?. The "Design Details" section below is for the real nitty-gritty. -->

For the development (main/master) branch only, NOT the existing release branches:

1. Switch remaining CI usage (mostly a few presubmits) to use the make build.

• Most of CI already uses the make builds, excluding some presubmits, we will need to switch these (generally a flag flip in the CI configuration).
• Most of periodic testing consumes pre-uploaded binaries from the make builds, and does not build at all. These will require no changes.
• In areas where the make build generates fewer artifacts or exercises fewer paths than bazel, we will err on the side of parity with artifacts that end up in a kubernetes/kubernetes release

2. Remove the bazel build and associated tooling.

• There are multiple scripts and LOTS of files related wholly to the bazel build in Kubernetes. Once we are confident that CI is no longer reliant on them we can remove these and relieving the maintenance toil.

No changes should be made to the release branches or their CI.

Risks and Mitigations

<!-- What are the risks of this proposal, and how do we mitigate? Think broadly. For example, consider both security and how this will impact the larger Kubernetes ecosystem. How will security be reviewed, and by whom? How will UX be reviewed, and by whom? Consider including folks who also work outside the SIG or subproject. -->

§ Design Details

<!-- This section should contain enough information that the specifics of your change are understandable. This may include API specs (though not always required) or even code snippets. If there's any ambiguity about HOW your proposal will be implemented, this is the place to discuss them. -->

Test Plan

Non-blocking “make build” equivalents to kubernetes/kubernetes “bazel” CI jobs will be introduced (if they don’t already exist). When the new jobs provide equivalent signal, they will be moved to blocking, and the old jobs will be retired.

This is relevant for at least the following jobs release-blocking and merge-blocking:

• pull-kubernetes-bazel-test (this can be converted to ~ make test)
• pull-kubernetes-bazel-build (this largely overlaps with other presubmits, if not for testing ~bazel build //... and can likely be removed)
• periodic-bazel-build-<branch> (this can likely already be removed in favor of ci-kubernetes-build-<branch>)
• periodic-bazel-test-<branch>
• post-kubernetes-bazel-build (this can likely already be removed, it’s unclear what depends on this job)

Again, this should not apply to existing release branches.

<!-- **Note:** *Not required until targeted at a release.* Consider the following in developing a test plan for this enhancement: - Will there be e2e and integration tests, in addition to unit tests? - How will it be tested in isolation vs with other components? No need to outline all of the test cases, just the general strategy. Anything that would count as tricky in the implementation, and anything particularly challenging to test, should be called out. All code is expected to have adequate tests (eventually with coverage expectations). Please adhere to the [Kubernetes testing guidelines][testing-guidelines] when drafting this test plan. [testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md -->

Graduation Criteria

<!-- **Note:** *Not required until targeted at a release.* Define graduation milestones. These may be defined in terms of API maturity, or as something else. The KEP should keep this high-level with a focus on what signals will be looked at to determine graduation. Consider the following in developing the graduation criteria for this enhancement: - [Maturity levels (`alpha`, `beta`, `stable`)][maturity-levels] - [Deprecation policy][deprecation-policy] Clearly define what graduation means by either linking to the [API doc definition](https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-versioning) or by redefining what graduation means. In general we try to use the same stages (alpha, beta, GA), regardless of how the functionality is accessed. [maturity-levels]: https://git.k8s.io/community/contributors/devel/sig-architecture/api_changes.md#alpha-beta-and-stable-versions [deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/ Below are some examples to consider, in addition to the aforementioned [maturity levels][maturity-levels]. #### Alpha -> Beta Graduation - Gather feedback from developers and surveys - Complete features A, B, C - Tests are in Testgrid and linked in KEP #### Beta -> GA Graduation - N examples of real-world usage - N installs - More rigorous forms of testing—e.g., downgrade tests and scalability tests - Allowing time for feedback **Note:** Generally we also wait at least two releases between beta and GA/stable, because there's no opportunity for user feedback, or even bug reports, in back-to-back releases. #### Removing a Deprecated Flag - Announce deprecation and support policy of the existing flag - Two versions passed since introducing the functionality that deprecates the flag (to address version skew) - Address feedback on usage/changed behavior, provided on GitHub issues - Deprecate the flag **For non-optional features moving to GA, the graduation criteria must include [conformance tests].** [conformance tests]: https://git.k8s.io/community/contributors/devel/sig-architecture/conformance-tests.md -->

This will be declared stable/GA when:

• All kubernetes/kubernetes master branch CI jobs use the preexisting make build system

<!-- TODO: as old jobs are rotated out each release anyhow, it may be acceptable to reduce the stable target to just the current development branch -->

• Bazel-related source files and related tooling are removed from the kubernetes/kubernetes repository on currently supported release branches and the current development branch
  • This will only happen for release branches as we phase out support for older releases, rotating in new supported releases that never contained the Bazel build
• Bazel-related configuration/presets are removed from kubernetes/kubernetes CI jobs in kubernetes/test-infra

As bazel-built artifacts are not built or distributed as part of a kubernetes/kubernetes release, there is no deprecation window required.

Upgrade / Downgrade Strategy

<!-- If applicable, how will the component be upgraded and downgraded? Make sure this is in the test plan. Consider the following in developing an upgrade/downgrade strategy for this enhancement: - What changes (in invocations, configurations, API use, etc.) is an existing cluster required to make on upgrade, in order to maintain previous behavior? - What changes (in invocations, configurations, API use, etc.) is an existing cluster required to make on upgrade, in order to make use of the enhancement? -->

n/a. Not relevant to upgrades. Existing release builds and upgrade CI use make.

Version Skew Strategy

<!-- If applicable, how will the component handle version skew with other components? What are the guarantees? Make sure this is in the test plan. Consider the following in developing a version skew strategy for this enhancement: - Does this enhancement involve coordinating behavior in the control plane and in the kubelet? How does an n-2 kubelet without this feature available behave when this feature is used? - Will any other components on the node change? For example, changes to CSI, CRI or CNI may require updating that component before the kubelet. -->

n/a.

§ Production Readiness Review Questionnaire

<!-- Production readiness reviews are intended to ensure that features merging into Kubernetes are observable, scalable and supportable; can be safely operated in production environments, and can be disabled or rolled back in the event they cause increased failures in production. See more in the PRR KEP at https://git.k8s.io/enhancements/keps/sig-architecture/1194-prod-readiness. The production readiness review questionnaire must be completed and approved for the KEP to move to `implementable` status and be included in the release. In some cases, the questions below should also have answers in `kep.yaml`. This is to enable automation to verify the presence of the review, and to reduce review burden and latency. The KEP must have a approver from the [`prod-readiness-approvers`](http://git.k8s.io/enhancements/OWNERS_ALIASES) team. Please reach out on the [#prod-readiness](https://kubernetes.slack.com/archives/CPNHUMN74) channel if you need any help or guidance. -->

Feature Enablement and Rollback

This section must be completed when targeting alpha to a release.

• How can this feature be enabled / disabled in a live cluster?

  N/A

• Does enabling the feature change any default behavior?

  N/A

• Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

  N/A

• What happens if we reenable the feature if it was previously rolled back?

  N/A

• Are there any tests for feature enablement/disablement?

  N/A

Rollout, Upgrade and Rollback Planning

This section must be completed when targeting beta graduation to a release.

• How can a rollout fail? Can it impact already running workloads?

  N/A

• What specific metrics should inform a rollback?

  N/A

• Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

  N/A

• Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

  N/A

Monitoring Requirements

This section must be completed when targeting beta graduation to a release.

• **How can an operator determine if th