RuntimeClass Support for Windows

§ Table of Contents

<!-- toc -->

• Release Signoff Checklist
• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • User Stories
    • Story 1 - Easy selection of Windows Server releases
    • Story 2 - Forward compatibility with Hyper-V
    • Story 3 - Choosing a specific multi-arch image
  • Implementation Details/Notes/Constraints
    • Adding new label node.kubernetes.io/windows-build (done)
    • Adding annotations to ImageSpec
      • ImageSpec changes
      • ImageSpec as part of the Image struct
      • Scenarios
  • Risks and Mitigations
    • Adding new node label
    • Annotations in ImageSpec
• Design Details
  • Test Plan
    • E2E Testing with CRI-ContainerD and Kubernetes
    • Unit testing with CRITest
  • Graduation Criteria
    • Alpha
    • Alpha -> Beta Graduation
    • Beta -> GA Graduation
      • Removing a deprecated flag
  • Upgrade / Downgrade Strategy
• Implementation History
• Alternatives
  • Support multiarch os/arch/version in CRI
  • Make the scheduler aware of Multi-arch images
  • Create a multi-arch Mutating admission controller
• Future Considerations
  • Pod Overhead
  • RuntimeClass Parameters
• Reference & Examples
  • Multi-arch container image overview

<!-- /toc -->

§ Release Signoff Checklist

ACTION REQUIRED: In order to merge code into a release, there must be an issue in kubernetes/enhancements referencing this KEP and targeting a release milestone before Enhancement Freeze of the targeted release.

For enhancements that make changes to code or processes/procedures in core Kubernetes i.e., kubernetes/kubernetes, we require the following Release Signoff checklist to be completed.

Check these off as they are completed for the Release Team to track. These checklist items must be updated for the enhancement to be released.

Items marked with (R) are required prior to targeting to a milestone / release.

• (R) kubernetes/enhancements issue in release milestone
• (R) KEP approvers have set the KEP status to implementable
• (R) Design details are appropriately documented
• (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
• (R) Graduation criteria is in place
• (R) Production readiness review completed
• Production readiness review approved
• "Implementation History" section is up-to-date for milestone
• User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
• Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Note: Any PRs to move a KEP to implementable or significant changes once it is marked implementable should be approved by each of the KEP approvers. If any of those approvers is no longer appropriate than changes to that list should be approved by the remaining approvers and/or the owning SIG (or SIG-arch for cross cutting KEPs).

Note: This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone.

§ Summary

RuntimeClass can be used to make it easier to schedule Pods onto appropriate nodes based on OS, OS Version, CPU Architecture and variant. These are all supported in container distribution & runtimes as part of Multi-arch container images today.

With Hyper-V available, Windows can run containers multiple Windows OS versions today, and Linux containers may be available in the future. This document proposes controlling those features through RuntimeClass as well, rather than adding new fields to the Pod API and changing Kubernetes scheduling behavior.

§ Motivation

There are a few related customer experience problems when running mixed (multiple-OS and/or multiple-CPU-arch) clusters today. Kubernetes scheduling is based solely on what's declared in the deployment as part of the Pod spec. A Pod could be scheduled to an incompatible node and fail to pull. This error causes a retry and backoff loop, and doesn't automatically fail over to another node.

In addition to the stuck pull failure case, there's also a question of user intent. Some containers are published for multiple OS's and/or architectures - this often referred to as multi-arch but the OCI distribution spec actually includes the OS, OS version, CPU architecture and variant. If a node can run multiple permutations of these, then there's not a deterministic way to know what the user intended to run. Running a container under CPU emulation may have performance penalties, or cost more in terms of leased compute time.

Today with only a single supported Windows version (10.0.17763) & runtime (Docker), the problem is easily mitigated with NodeSelectors or Taints. This is documented today in [Guide for scheduling Windows containers in Kubernetes]. Moving these to RuntimeClasses simplifies the experience further, and can simplify deployment YAML for users in Kubernetes v1.16.

Next, SIG-Windows is adding support for [Windows CRI-ContainerD], Windows nodes will be able to handle running multiple Windows OS versions using Hyper-V isolation. This technology could be used to run Linux containers as well, leading to more ambiguity. This KEP aims to resolve the ambiguity of pulling and running a container image and pod when multiple os/versions/architecture/variants may be supported on a single node.

Goals

• Schedule a Windows Pod to a compatible node
  • Allow matching Windows versions without Hyper-V isolation
  • Allow opt-in on a per-Pod basis to run containers using existing Windows versions with backwards-compatibility provided by Hyper-V on a new Windows OS version node
• Be able to schedule a specific image from a multi-arch manifest on a given node
• Provide a simpler experience (fewer lines of YAML) than adding os and version nodeSelector and tolerations to each Pod

Non-Goals

• Linux container support on Windows is not a requirement or test target for Kubernetes 1.17, but it's not specifically excluded.
• Running newer Windows OS version containers (Windows Server version 1903) on an older OS version host (Windows Server version 1809). This is not supported today with or without Hyper-V isolation (see [Windows container version compatibility]).

§ Proposal

For Kubernetes 1.17, we're proposing three key things to improve the experience deploying Windows containers across multiple versions, and enable experimentation with Hyper-V while ContainerD support is in alpha/beta stages.

1. Add node.kubernetes.io/windows-build label using Windows kubelet
2. Add RuntimeHandler to the CRI pull API
3. Use RuntimeHandler and separate ContainerD configurations to control runtime specific parameters such as overrides for OS/version/architecture and Hyper-V isolation.

As a fallback plan, steps 2/3 could use annotations instead during alpha, but this would make things more difficult for users trying out the new features as the Pod specs would change between versions. Comparable solutions based on ContainerD + Kata on Linux are moving away from annotations and to RuntimeClass, so we want to follow the same approach for ContainerD + Hyper-V.

User Stories

Story 1 - Easy selection of Windows Server releases

As of Kubernetes 1.16, [RuntimeClass Scheduling] is in beta and can be used to simplify setting nodeSelector & tolerations. This makes it easier to steer workloads to a suitable Windows or Linux node using the existing labels. With the addition of a new windows-build label it can also distinguish between multiple Windows version in the same cluster. For more details on how and why build numbers are used, read Adding new label node.kubernetes.io/windows-build below.

Note: There's an open PR website#16697 to add RuntimeClass examples to existing documentation for Kubernetes 1.16.

• Example of a RuntimeClass that would restrict a pod to Windows Server 2019 / 1809 (10.0.17763)

  apiVersion: node.k8s.io/v1beta1
  kind: RuntimeClass
  metadata:
    name: windows-1809
  handler: 'docker'
  scheduling:
    nodeSelector:
      kubernetes.io/os: 'windows'
      kubernetes.io/arch: 'amd64'
      node.kubernetes.io/windows-build: '10.0.17763'
    tolerations:
    - effect: NoSchedule
      key: windows
      operator: Equal
      value: "true"
  

• Example of a RuntimeClass that would restrict a pod to Windows Server version 1903 (10.0.18362)

  apiVersion: node.k8s.io/v1beta1
  kind: RuntimeClass
  metadata:
    name: windows-1903
  handler: 'docker'
  scheduling:
    nodeSelector:
      kubernetes.io/os: 'windows'
      kubernetes.io/arch: 'amd64'
      node.kubernetes.io/windows-build: '10.0.18362'
    tolerations:
    - effect: NoSchedule
      key: windows
      operator: Equal
      value: "true"
  

Story 2 - Forward compatibility with Hyper-V

Once a new version of Windows Server is deployed using [Windows CRI-ContainerD], Hyper-V can be enabled to provide backwards compatibility and run Windows containers based on the previous OS version. Cluster admins can pick between two different approaches to move applications forward. Currently Windows Server has only supported backwards compatibility with Hyper-V, for example running a 1809 container on 1903. Therefore the node OS version should be the same or ahead of the container OS version used (see: [Windows container version compatibility]).

1. Create a new RuntimeClass that Pods can use to try out Hyper-V isolation on the new version of Windows.

   apiVersion: node.k8s.io/v1beta1
   kind: RuntimeClass
   metadata:
     name: windows-1809-hyperv
   handler: 'containerd-hyperv-17763'
   scheduling:
     nodeSelector:
       kubernetes.io/os: 'windows'
       kubernetes.io/arch: 'amd64'
       node.kubernetes.io/windows-build: '10.0.18362'
     tolerations:
     - effect: NoSchedule
       key: windows
       operator: Equal
       value: "true"
   

2. Once sufficient testing is done, the RuntimeClass from Story 1 could be updated. This would cause new deployments to go to these nodes, without having to update them individually. The nodes running the previous Windows version could be drained and removed from the cluster. As the pods running on them are rescheduled, the new RuntimeClass will be applied.

   apiVersion: node.k8s.io/v1beta1
   kind: RuntimeClass
   metadata:
     name: windows-1809
   handler: 'containerd-hyperv-17763'
   scheduling:
     nodeSelector:
       kubernetes.io/os: 'windows'
       kubernetes.io/arch: 'amd64'
       node.kubernetes.io/windows-build: '10.0.18362'
     tolerations:
     - effect: NoSchedule
       key: windows
       operator: Equal
       value: "true"
   

Another RuntimeClass could still be run on the same hosts to use updated containers without Hyper-V isolation.

apiVersion: node.k8s.io/v1beta1
kind: RuntimeClass
metadata:
  name: windows-1903
handler: 'default'
scheduling:
  nodeSelector:
    kubernetes.io/os: 'windows'
    kubernetes.io/arch: 'amd64'
    node.kubernetes.io/windows-build: '10.0.18362'
  tolerations:
  - effect: NoSchedule
    key: windows
    operator: Equal
    value: "true"

Story 3 - Choosing a specific multi-arch image

Starting from story 2, the RuntimeClass handler field could also be used as a means os/version/architecture/variant used by the container runtime. The handler is matched up with a corresponding section in the ContainerD configuration file.

Here's an example of what corresponding ContainerD configurations could look like for the runtimes above. The first one would force Hyper-V for compatibility with containers needing 10.0.17763. The second one would be the default for the current version where no Hyper-V isolation is required for compatibility.

        [plugins.cri.containerd.runtimes.containerd-hyperv-17763]
          runtime_type = "io.containerd.runhcs.v1"
          [plugins.cri.containerd.runtimes.containerd-hyperv-17763.options]
            SandboxImage = "{{WINDOWSSANDBOXIMAGE}}"
            SandboxPlatform = "windows/amd64"
            SandboxOsVersion = "10.0.17763"
            SandboxIsolation = 1

        [plugins.cri.containerd.runtimes.default]
          runtime_type = "io.containerd.runhcs.v1"
          # No version is specified for process isolation, the node OS version is used

Implementation Details/Notes/Constraints

There were multiple options discussed with SIG-Node & SIG-Windows on October 8 2019 prior to filing this KEP. That discussion & feedback were captured in [Difficulties in mixed OS and arch clusters]. If you're looking for more details on other approaches excluded, please review that document.

Adding new label node.kubernetes.io/windows-build (done)

Done. This label was added in Kubernetes 1.17

In [Bounding Self-Labeling Kubelets], a specific range of prefixes were reserved for node self-labeling - [*.]node.kubernetes.io/*. Adding a new label within that namespace won't require any changes to NodeRestriction admission. As a new field it also won't require changes to any existing workloads.

Build numbers will be used instead of product names in the node labels for a few reasons:

• The same OS version may be marketed under two different names due to support / licensing differences. For example, Windows Server 2019 and Windows Server version 1809 are the same build number. The actual binary compatibility is based on build number and the same container can run on either.
• Windows product names may not have been announced when the Kubernetes project contributors start building and testing against it. Using build numbers instead avoids needing to change Kubernetes source once a name has been announced.

Here are the current product name to build number mappings to illustrate the point:

Product Name	Build Number(s)
Windows Server 2019	10.0.17763
Windows Server version 1809	10.0.17763
Windows Server version 1903	10.0.18362
Windows Server version 1909	10.0.18363

[Windows Update History] has a full list of version numbers by release & patch. Starting from an example of 10.0.17763.805 - the OS major, minor, and build number - 10.0.17763 - should match for containers to be compatible. The final .805 refers to the monthly patches and isn't required for compatibility. Therefore, a value such as node.kubernetes.io/windows-build = 10.0.17763 will be used. Each one of these Windows Server version will have corresponding containers released - see [Container Base Images] and [Windows Insider Container Images] for details.

This will pass the regex validation for labels (references: src1 src2). Even the most specific identifier in the Windows registry BuildLabEx, for example 18362.1.amd64fre.19h1_release.190318-1202 is within the allowed length and characters to pass the existing validation, although we're not planning to use that whole string. Instead, the kubelet will shorten to just what's needed similar to what's returned today in the output from kubectl describe node (src)

To make this easier to consume in the Kubelet and APIs, it will be updated in multiple places:

• Add a well-known label in
  • k8s.io/api/core/v1/well_known_labels.go
  • pkg/kubelet/apis/well_known_labels.go
• Set & update it in Kubelet
  • pkg/kubelet/kubelet_node_status.go

Adding annotations to ImageSpec

Current proposal is to change the ImageSpec to incorporate annotations for runtime class. These can be passed down to the container runtimes where various actions can be taken as per the runtime class specified. Also proposed is the change to Image struct to include the ImageSpec and thereby include the annotations.

ImageSpec changes

Currently ImageSpec contains just a string.

type ImageSpec struct {
  Image string
}

The proposal is to add Annotations into the ImageSpec:

type ImageSpec struct {
  Image string
  Annotations []Annotation
}

The runtimeHandler annotation will be based on the Runtime Class specified by the user:

“kubernetes.io/runtimehandler”: “<corresponding values>”

We could potentially also add the kubernetes specification annotations for consideration of the runtime:

“kubernetes.io/arch”: “amd64”
“kubernetes.io/os”: ”linux”

Note that these are currently derived from from GOARCH and GOOS at runtime, which does not reflect the image which needs to be pulled but instead corresponds to the system on which kubelet is running. The runtime handler specified in the annotation will provide more accurate indication of the user intent to the runtime.

ImageSpec as part of the Image struct

ListImage returns Image struct from the runtime to Kubelet which currently does not include the ImageSpec:

type ImageService interface {
  // PullImage pulls an image from the network to local storage using the supplied
  // secrets if necessary. It returns a reference (digest or ID) to the pulled image.
   PullImage(image ImageSpec, pullSecrets []v1.Secret, podSandboxConfig *runtimeapi.PodSandboxConfig) (string, error)
  // GetImageRef gets