Windows node support

§ Table of Contents

<!-- toc -->

• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • What works today
  • Windows Node Roadmap (post-GA work)
    • Custom DNS updates for CNI plugins
  • What will never work
  • Windows Container Compatibility
  • Relevant resources/conversations
  • Risks and Mitigations
    • Ensuring OS-specific workloads land on appropriate container host
    • Memory Overprovisioning
• Graduation Criteria
• Implementation History
• Testing Plan
  • Test Dashboard
  • Test Environment
  • Test Approach
    • Adapting existing tests
    • Substitute test cases
    • Windows specific tests
• Conformance Testing
• API Reference
  • V1.Container
  • V1.Pod
  • V1.PodSecurityContext
• Other references

<!-- /toc -->

§ Summary

There is strong interest in the community for adding support for workloads running on Microsoft Windows. This is non-trivial due to the significant differences in the implementation of Windows from the Linux-based OSes that have so far been supported by Kubernetes. This KEP will allow Windows nodes to be added to a Kubernetes cluster as compute nodes. With the introduction of Windows nodes, developers will be able to schedule Windows Server containers and run Windows-based applications on Kubernetes.

§ Motivation

Windows-based workloads still account for a significant portion of the enterprise software space. While containerization technologies emerged first in the UNIX ecosystem, Microsoft has made investments in recent years to enable support for containers in its Windows OS. As users of Windows increasingly turn to containers as the preferred abstraction for running software and modernizing existing applications, the Kubernetes ecosystem stands to benefit by becoming a cross-platform cluster manager.

Goals

• Enable users to schedule Windows Server containers in Kubernetes through the introduction of support for Windows compute nodes
• Document the differences and limitations compared to Linux
• Create a test suite in testgrid to maintain high quality for this feature and prevent regression of functionality

Non-Goals

• Adding Windows support to all projects in the Kubernetes ecosystem (Cluster Lifecycle, etc)
• Enable the Kubernetes master components to run on Windows
• Support for LCOW (Linux Containers on Windows with Hyper-V Isolation)

§ Proposal

As of 29-11-2018 much of the work for enabling Windows nodes has already been completed. Both kubelet and kube-proxy have been adapted to work on Windows Server, and so the first goal of this KEP is largely already complete.

What works today

• Windows-based containers can be created by kubelet, provided the host OS version matches the container base image. Microsoft will distribute the operating system-dependent pause image (mcr.microsoft.com/k8s/core/pause:1.0.0).
  • Pod
    • Single or multiple containers per Pod with process isolation
    • There are no notable differences in Pod status fields between Linux and Windows containers
    • Readiness and Liveness probes
    • postStart & preStop container lifecycle events
    • ConfigMap, Secrets: as environment variables or volumes (Volume subpath does not work)
    • EmptyDir
    • Named pipe host mounts
    • Volumes can be shared between containers in a Pod
    • Resource limits
  • Services types NodePort, ClusterIP, LoadBalancer, and ExternalName. Service environment variables and headless services work.
    • Cross operating system service connectivity
  • Workload controllers ReplicaSet, ReplicationController, Deployments, StatefulSets, DaemonSet, Job, CronJob
  • Scheduler preemption
  • Pod & container metrics
  • Horizontal Pod Autoscaling using all metrics
  • KubeCtl Exec
  • Resource Quotas
• Windows Server 2019 is the only Windows operating system we will support at GA timeframe. Note above that the host operating system version and the container base image need to match. This is a Windows limitation we cannot overcome.
• Customers can deploy a heterogeneous cluster, with Windows and Linux compute nodes side-by-side and schedule Docker containers on both operating systems. Of course, Windows Server containers have to be scheduled on Windows and Linux containers on Linux
• Out-of-tree Pod networking with Azure-CNI, OVN-Kubernetes, two CNI meta-plugins, Flannel (VXLAN and Host-Gateway)
• Dockershim CRI
• Many<sup id="a1">[1]</sup> of the e2e conformance tests when run with alternate Windows-based images which are being moved to kubernetes-sigs/windows-testing
• Persistent storage: FlexVolume with SMB + iSCSI, and in-tree AzureFile and AzureDisk providers
• Kube-Proxy support for L2Bridge and Overlay networks

Windows Node Roadmap (post-GA work)

• Group Managed Service Accounts, a way to assign an Active Directory identity to a Windows container, is forthcoming with KEP Windows Group Managed Service Accounts for Container Identity. This work will be released as alpha in v1.14 and is already merged.
• kubectl port-forward hasn't been implemented due to lack of an nsenter equivalent to run a process inside a network namespace.
• CRIs other than Dockershim: CRI-containerd support is forthcoming
• Some kubeadm work was done in the past to add Windows nodes to Kubernetes, but that effort has been dormant since. We will need to revisit that work and complete it in the future.
• Calico CNI for Pod networking
• Hyper-V isolation (Currently this is limited to 1 container per Pod and is an alpha feature)
  • This could enable backwards compatibility likely as a RuntimeClass. This would allow running a container host OS that is newer in version than the container OS (visit this link for additional compatibility definitions https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility)
• It is unclear if the RuntimeClass proposal from sig-node will simplify scheduled Windows containers. We will work with sig-node on this.
• Properly implement terminationGracePeriodSeconds for Windows (https://github.com/moby/moby/issues/25982 and https://github.com/kubernetes/kubernetes/issues/73434)
• Single file mapping and Termination message will work when we introduce CRI containerD support in Windows
• Design and implement --enforce-node-allocatable, hard/soft eviction and MemoryPressure conditions. These all depend on cgroups in Linux, and the kubelet will need new work specific to Windows to raise and respond to memory pressure conditions. See Memory Overprovisioning later in this doc.
• Fix run_as_username for Windows (https://github.com/kubernetes/kubernetes/issues/73387)
• Support for Local Traffic Policy and DSR mode on Windows (https://github.com/kubernetes/kubernetes/issues/62046)

Custom DNS updates for CNI plugins

For v1.14, custom pod DNS configuration tests are not running. Some CNI implementations updates are needed to Azure-CNI, win-bridge, OVN, and flannel which are out of the kubernetes/kubernetes tree. Once those are updated, the tests are tracked in issue 73414/pr 74925 will be merged.

As part of Azure-CNI PR#305, manual tests were run with Pod.Spec.DNSPolicy = DNSNone. Hostname, Subdomain, and DNSConfig.Nameservers, and DNSConfig.Searches were set correctly based on the Pod spec.

Tracking Issues:

• win-bridge #271 - this is also used in the test passes for GCE, see gce-k8s-windows-testing#7
• Azure-CNI PR#305

What will never work

Note that some features are plain unsupported while some will not work without underlying OS changes

• Certain Pod functionality
  • Privileged containers
  • Pod security context privilege and access control settings. Any Linux Capabilities, SELinux, AppArmor, Seccomp, Capabilities (POSIX Capabilities), and others are not supported
  • Reservations are not enforced by the OS, but overprovisioning could be blocked with --enforce-node-allocatable=pods (pending: tests needed)
  • Certain volume mappings
    • Subpath volume mounting
    • Subpath volume mounting for Secrets
    • Host mount projection
    • DefaultMode (due to UID/GID dependency)
    • readOnly root filesystem. Mapped volumes still support readOnly
    • Block device mapping
  • Expanding the mounted volume (resizefs)
  • HugePages
  • Memory as the storage medium
• CSI plugins, which require privileged containers
• File system features like uui/guid, per-user Linux filesystem permissions, and read-only root filesystems (see note above and also later in the doc about read-only volumes)
• NFS based storage/volume support (https://github.com/kubernetes/kubernetes/issues/56188)
• Host networking is not available in Windows
• ClusterFirstWithHostNet is not supported for DNS. Windows treats all names with a . as a FQDN and skips PQDN resolution
• Not all features of shared namespaces are supported. This is clarified in the API section below
• The existing node problem detector is Linux-only and requires privileged containers. In general, we don't expect these to be used on Windows because there's no privileged support
• Overlay networking support in Windows Server 1803 is not fully functional using the win-overlay CNI plugin. Specifically service IPs do not work on Windows nodes. This is currently specific to win-overlay; other CNI plugins (OVS, AzureCNI) work. Since Windows Server 1803 is not supported for GA, this is mostly not applicable. We left it here since it impacts beta
• Outbound communication using the ICMP protocol via the win-overlay, win-bridge, and Azure-CNI plugin. Specifically, the Windows data plane (VFP) doesn't support ICMP packet transpositions. This means:
  • ICMP packets directed to destinations within the same network (e.g. pod to pod communication via ping) will work as expected and without any limitations
  • TCP/UDP packets will work as expected and without any limitations
  • ICMP packets directed to pass through a remote network (e.g. pod to external internet communication via ping) cannot be transposed and thus will not be routed back to their source
    • Since TCP/UDP packets can still be transposed, one can substitute ping <destination> with curl <destination> to be able to debug connectivity to the outside world.

Windows Container Compatibility

As noted above, there are compatibility issues enforced by Microsoft where the host OS version must match the container base image OS. Changes to this compatibility policy must come from Microsoft. For GA, since we will only support Windows Server 2019 (aka 1809), both container host OS and container OS must be running the same version of Windows, 1809.

Having said that, a customer can deploy Kubernetes v1.14 with Windows 1809.

• We will support Windows 1809 with at least 2 additional Kuberneres minor releases (v1.15 and v.1.16)
• It is possible additional Windows releases (for example Windows 1903) will be added to the support matrix of future Kubernetes releases and they will also be supported for the next 2 versions of Kubernetes after their initial support is announced
• SIG-Windows will announce support for new Windows operating systems at most twice per year, based on Microsoft's published release cycle

Kubernetes minor releases are only supported for 9 months (https://kubernetes.io/docs/setup/version-skew-policy/), which is a smaller support interval than the support interval for Windows bi-annual releases (https://docs.microsoft.com/en-us/windows-server/get-started/windows-server-release-info)

We don't expect all Windows customers to update the operating system for their apps twice a year. Upgrading your applications is what will dictate and necessitate upgrading or introducing new nodes to the cluster. For the customers that chose to upgrade their operating system for containers running on Kubernetes, we will offer guidance and step-by-step instructions when we add support for a new operating system version. This guidance will include recommended upgrade procedures for upgrading user applications together with cluster nodes.

Windows nodes will adhere to Kubernetes version-skew policy (node to control plane versioning) the same way as Linux nodes do today (https://kubernetes.io/docs/setup/version-skew-policy/)

Relevant resources/conversations

• sig-architecture thread
• cncf-k8s-conformance thread
• kubernetes/enhancements proposal

Risks and Mitigations

Second class support: Kubernetes contributors are likely to be thinking of Linux-based solutions to problems, as Linux remains the primary OS supported. Keeping Windows support working will be an ongoing burden potentially limiting the pace of development.

User experience: Users today will need to use some combination of taints and node selectors in order to keep Linux and Windows workloads separated. In the best case this imposes a burden only on Windows users, but this is still less than ideal. The recommended approach is outlined below, with one of its main goals being that we should not break compatibility for existing Linux workloads

Ensuring OS-specific workloads land on appropriate container host

As you can see below, we plan to document how Windows containers can be scheduled on the appropriate host using Taints and Tolerations. All nodes today have the following default labels (These labels will be graduating to stable soon)

• beta.kubernetes.io/os = [windows|linux]
• beta.kubernetes.io/arch = [amd64|arm64|...]

If a deployment does not specify a nodeSelector like "beta.kubernetes.io/os": windows, it is possible the Pods can be scheduled on any host, Windows or Linux. This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux. The best practice we will recommend is to use a nodeSelector.

However, we understand that in many cases customers have a pre-existing large number of deployments for Linux containers, as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic pod generation cases, such as with Operators. Customers will be hesitant to make the configuration change to add nodeSelectors. Our proposal as an alternative is to use Taints. Because the kubelet can set Taints during registration, it could easily be modified to automatically add a taint when running on Windows only (--register-with-taints='os=Win1809:NoSchedule'). By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods). In order for a Windows Pod to be scheduled on a Windows node, it would need both the nodeSelector to choose Windows, and a toleration.

nodeSelector:
    "beta.kubernetes.io/os": windows
tolerations:
    - key: "os"
      operator: "Equal"
      value: "Win1809"
      effect: "NoSchedule"

Memory Overprovisioning

Windows always treats all user-mode memory allocations as virtual, and pagefiles are mandatory. The net effect is that Windows won't reach out of memory conditions the same way Linux does, and processes will page to disk instead of being subject to out of memory (OOM) termination. There is no way to guarantee a physical memory allocation or reserve for a process - only limits. See #73417 for more details on the investigation for 1.14.

Keeping memory usage within reasonable bounds is possible with a two-step process. First, use the kubelet parameters --kubelet-reserve and/or --system-reserve to account for memory usage on the node (outside of containers). This will reduce NodeAllocatable. As you deploy workloads, use resource limits and reserves on containers. This will also subtract from NodeAllocatable and prevent the scheduler from adding more pods once a node is full. These will be documented as best practices for v1.14. The related kubelet parameters --eviction-hard, --eviction-soft, and --enforce-node-allocatable are invalid for v1.14.

For later releases, we can work on a configurable heuristic to detect memory pressure, report it through the kubelet MemoryPressure condition, and implement pod eviction.

§ Graduation Criteria

• All features and functionality under What works today is fully tested and vetted to be working by SIG-Windows
• SIG-Windows has high confidence to the stability and reliability of Windows Server containers on Kubernetes
• 100% green/passing conformance tests that are applicable to Windows (see the Testing Plan section for details on these tests). These tests are adequate, non flaky, and continuously run. The test results are publicly accessible, enabled as part of the release-blocking suite
• Compatibility will not be broken, either for existing users/clusters/features or for the new features going forward, and we will adhere to the deprecation policy (https://kubernetes.io/docs/reference/using-api/deprecation-policy/).
• Comprehensive documentation that includes but is not limited to the following sections. Documentation will reside at https://kubernetes.io/docs and will adequately cover end user and admin documentation that describes what the user does and how to use it. Not all of the documentation will be under the Getting Started Guide for Windows. Part of it will reside in its own sections (like the Group Managed Service Accounts) and part will be in the Contributor development guide (like the instructions on how to build your own source code)

1. Outline of Windows Server containers on Kubernetes
2. Getting Started Guide, including Prerequisites
3. How to deploy Windows nodes in Kubernetes and where to find the proper binaries (Listed under the changelog for every release. For example https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.13.md#server-binaries)
4. Overview of Networking on Windows
5. Links to documentation on how to deploy and use CNI plugins for Windows (example for OVN - https://github.com/openvs