Windows Group Managed Service Accounts for Container Identity

§ Table of Contents

<!-- toc -->

• Summary
• Motivation
  • Goals
  • Non-Goals
• Proposal
  • Background
    • What is Active Directory?
    • What is a Windows service account?
    • How is it applied to containers?
  • User Stories
    • Web Applications with MS SQL Server
  • Implementation Details/Notes/Constraints [optional]
    • GMSA specification for pods and containers
    • GMSAExpander webhook
    • GMSAExpander and GMSAAuthorizer Webhooks
    • Changes in Kubelet/kuberuntime for Windows:
    • Changes in CRI API:
    • Changes in Dockershim
    • Changes in CRIContainerD
    • Changes in Windows OCI runtime
  • Risks and Mitigations
    • Threat vectors and countermeasures
    • Transitioning from Alpha annotations to Beta/Stable fields
• Graduation Criteria
• Implementation History
• Drawbacks [optional]
• Alternatives
  • Other authentication methods
  • Injecting credentials from a volume
  • Specifying only the name of GMSACredentialSpec objects in pod spec fields/annotations
  • Enforce presence of GMSAAuthorizer and RBAC mode to enable GMSA functionality in Kubelet

<!-- /toc -->

§ Summary

Active Directory is a service that is built-in and commonly used on Windows Server deployments for user and computer identity. Apps are run using Active Directory identities to enable common user to service, and service to service authentication and authorization. This proposal aims to support a specific type of identity, Group Managed Service Accounts (GMSA), for use with Windows Server containers. This will allow an operator to choose a GMSA at deployment time, and run containers using it to connect to existing applications such as a database or API server without changing how the authentication and authorization are performed.

§ Motivation

There has been a lot of interest in supporting GMSA for Windows containers since it's the only way for a Windows application to use an Active Directory identity. This is shown in asks and questions from both public & private conversations:

• https://github.com/kubernetes/kubernetes/issues/51691 "For windows based containers, there is a need to access shared objects using domain user contexts."
• Multiple large customers are asking the Microsoft Windows team to enable this feature through container orchestrators
• Multiple developers have blogged how to use this feature, but all are on standalone machines instead of orchestrated through Kubernetes
  • https://artisticcheese.wordpress.com/2017/09/09/enabling-integrated-windows-authentication-in-windows-docker-container/
  • https://community.cireson.com/discussion/3853/example-cireson-scsm-portal-on-docker-windows-containers
  • https://cloudiqtech.com/windows-2016-docker-containers-using-gmsa-connect-to-sql-server/
  • https://success.docker.com/api/asset/.%2Fmodernizing-traditional-dot-net-applications%2F%23integratedwindowsauthentication

Goals

• Windows users can run containers with an existing GMSA identity on Kubernetes
• No extra files or Windows registry keys are needed on each Windows node. All needed data should flow through Kubernetes+Kubelet APIs
• Prevent pods from being inadvertently scheduled with service accounts that do not have access to a GMSA

Non-Goals

• Running Linux applications using GMSA or a general Kerberos based authentication system
• Replacing any existing Kubernetes authorization or authentication controls. Specifically, a subset of users cannot be restricted from creating pods with Service Accounts authorized to use certain GMSAs within a namespace if the users are already authorized to create pods within that namespace. Namespaces serve as the ACL boundary today in Kubernetes and we do not try to modify or enhance this in the context of GMSAs to prevent escalation of privilege through a service account authorized to use certain GMSAs.
• Providing unique container identities. By design, Windows GMSA are used where multiple nodes are running apps as the same Active Directory principal.
• Isolation between container users and processes running as the GMSA. Windows already allows users and system services with sufficient privilege to create processes using a GMSA.
• Preventing the node from having access to the GMSA. Since the node already has authorization to access the GMSA, it can start processes or services using as the GMSA. Containers do not change this behavior.
• Restricting specification of GMSA credential specs in pods or containers if RBAC is not enabled or the admission webhook described below is not installed/enabled.

§ Proposal

Background

What is Active Directory?

Windows applications and services typically use Active Directory identities to facilitate authentication and authorization between resources. In a traditional virtual machine scenario, the computer is joined to an Active Directory domain which enables it to use Kerberos, NTLM, and LDAP to identify and query for information about other resources in the domain. When a computer is joined to Active Directory, it is given an unique identity represented as a computer object in LDAP.

What is a Windows service account?

A Group Managed Service Account (GMSA) is a shared Active Directory identity that enables common scenarios such as authenticating and authorizing incoming requests and accessing downstream resources such as a database server, file share, or other workload. It can be used by multiple authorized users or computers at the same time.

How is it applied to containers?

To achieve the scale and speed required for containers, Windows uses a group managed service account in lieu of individual computer accounts to enable Windows containers to communicate with Active Directory. As of right now, the Host Computer Service (which exposes the interface to manage containers) in Windows cannot use individual Active Directory computer & user accounts - it only supports GMSA.

Different containers on the same machine can use different GMSAs to represent the specific workload they are hosting, allowing operators to granularly control which resources a container has access to. However, to run a container with a GMSA identity, an additional parameter must be supplied to the Windows Host Compute Service to indicate the intended identity. This proposal seeks to add support in Kubernetes for this parameter to enable Windows containers to communicate with other enterprise resources.

It's also worth noting that Docker implements this in a different way that's not managed centrally. It requires dropping a file on the node and referencing it by name, eg: docker run --credential-spec='file://gmsa-cred-spec1.json' . For more details see the Microsoft doc.

User Stories

Web Applications with MS SQL Server

A developer has a Windows web app that they would like to deploy in a container with Kubernetes. For example, it may have a web tier that they want to scale out running ASP.Net hosted in the IIS web server. Backend data is stored in a Microsoft SQL database, which all of the web servers need to access behind a load balancer. An Active Directory Group Managed Service Account is used to avoid hardcoded passwords, and the web servers run with that credential today. The SQL Database has a user with read/write access to that GMSA so the web servers can access it. When they move the web tier into a container, they want to preserve the existing authentication and authorization models.

When this app moves to production on containers, the team will need to coordinate to make sure the right GMSA is carried over to the container deployment.

1. An Active Directory domain administrator will:

• Join all Windows Kubernetes nodes to the Active Directory domain.
• Provision the GMSA and gives details to Kubernetes cluster admin.
• Assign access to a AD security group to control what machines can use this GMSA. The AD security group should include all authorized Kubernetes nodes.

2. A Kubernetes cluster admin (or cluster setup tools or an operator) will install a cluster scoped CRD (of kind GMSACredentialSpec) whose instances will be used to store GMSA credential spec configuration:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: gmsacredentialspecs.windows.k8s.io
spec:
  group: windows.k8s.io
  version: v1alpha1
  names:
    kind: GMSACredentialSpec
    plural: gmsacredentialspecs
  scope: Cluster
  validation:
    openAPIV3Schema:
      properties:
        credspec:
          description: GMSA Credential Spec
          type: object

A GMSACredentialSpec may be used by sidecar containers across different namespaces. Therefore the CRD needs to be cluster scoped.

3. A Kubernetes cluster admin will create a GMSACredentialSpec object populated with the credential spec associated with a desired GMSA:

• The cluster admin may run a utility Windows PowerShell script to generate the YAML definition of a GMSACredentialSpec object populated with the GMSA credential spec details. Note that the credential spec YAML follows the structure of the credential spec (in JSON) as referred to in the OCI spec. The utility Powershell script for generating the YAML will be largely identical to the already published Powershell script with the following differences: [a] it will output the credential spec in YAML format and [b] it will encapsulate the credential spec data within a Kubernetes object YAML (of kind GMSACredentialSpec). The GMSACredentialSpec YAML will not contain any passwords or crypto secrets. Example credential spec YAML for a GMSA webapplication1:

apiVersion: windows.k8s.io/v1alpha1
kind: GMSACredentialSpec
metadata:
  name: "webapp1-credspec"
credspec:
  ActiveDirectoryConfig:
    GroupManagedServiceAccounts:
    - Name: WebApplication1
      Scope: CONTOSO
    - Name: WebApplication1
      Scope: contoso.com
  CmsPlugins:
  - ActiveDirectory
  DomainJoinConfig:
    DnsName: contoso.com
    DnsTreeName: contoso.com
    Guid: 244818ae-87ca-4fcd-92ec-e79e5252348a
    MachineAccountName: WebApplication1
    NetBiosName: CONTOSO
    Sid: S-1-5-21-2126729477-2524075714-3094792973

• With the YAML from above (or generated manually), the cluster admin will create a GMSACredentialSpec object in the cluster.

4. A Kubernetes cluster admin will configure RBAC for the GMSACredentialSpec so that only desired service accounts can use the GMSACredentialSpec:

• Create a cluster wide role to get and "use" the GMSACredentialSpec:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: webapp1-gmsa-user
rules:
- apiGroups: ["windows.k8s.io"]
  resources: ["gmsacredentialspecs"]
  resourceNames: ["webapp1-credspec"]
  verbs: ["get", "use"]

• Create a rolebinding and assign desired service accounts to the above role:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: use-webapp1-gmsa
  namespace: default
subjects:
- kind: ServiceAccount
  name: webapp-sa
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: webapp1-gmsa-user
  apiGroup: rbac.authorization.k8s.io

5. Application admins will deploy app pods that require a GMSA identity along with a Service Account authorized to use the GMSAs. There will be two ways to specify the GMSA credential spec details for pods and containers. It is expected that users will typically use the first option as it is more user friendly. The second option is available mainly due to an artifact of the design choices made and described here for completeness.

• Specify the name of the desired GMSACredentialSpec object (e.g. webapp1-credspec): If an application administrator wants containers to be initialized with a GMSA identity, specifying the names of the desired GMSACredentialSpec objects is mandatory. In the alpha stage of this feature, the name of the desired GMSACredentialSpec can be set through an annotation on the pod (applicable to all containers): pod.alpha.windows.kubernetes.io/gmsa-credential-spec-name as well as for each container through annotations of the form: <containerName>.container.alpha.windows.kubernetes.io/gmsa-credential-spec-name. In the beta stage, the annotations will be superseded by fields in the securityContext of the pod: podspec.securityContext.windows.gmsaCredentialSpecName and in the securityContext of each container: podspec.container[i].securityContext.windows.gmsaCredentialSpecName. The GMSACredentialSpec name for a container will override the GMSACredentialSpec name specified for the whole pod. Sample pod spec showing specification of GMSACredentialSpec name at the pod level and overriding it for one of the containers:

apiVersion: v1
kind: Pod
metadata:
  name: iis
  labels:
    name: iis
  annotations: {
    pod.alpha.windows.kubernetes.io/gmsa-credential-spec-name : webapp1-credspec
    iis.container.alpha.windows.kubernetes.io/gmsa-credential-spec-name : webapp2-credspec
  }
spec:
  containers:
    - name: iis
      image: mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019
      ports:
        - containerPort: 80
    - name: logger
      image: eventlogger:2019
      ports:
        - containerPort: 80
  nodeSelector:
    beta.kubernetes.io/os : windows

• Specify the contents of the credspec field of GMSACredentialSpec that gets passed down to the container runtime: Specifying the credential spec contents in JSON form is optional and unnecessary. GMSAExpander will automatically populate this field (as described in the next section) based on the name of the GMSACredentialSpec object. In the alpha stage of this feature, a JSON representation of the contents of the desired GMSACredentialSpec may be set through an annotation on the pod (applicable to all containers): pod.alpha.windows.kubernetes.io/gmsa-credential-spec as well as for each container through annotations of the form: <containerName>.container.alpha.windows.kubernetes.io/gmsa-credential-spec. In the beta stage and beyond, the annotations will be superseded by a field the securityContext of the pod podspec.securityContext.windows.gmsaCredentialSpec and in the securityContext of each container: podspec.container[i].securityContext.windows.gmsaCredentialSpec. The credential spec JSON for a container will override the credential spec JSON specified for the whole pod.

The ability to specify credential specs for each container within a pod aligns with how security attributes like runAsGroup, runAsUser, etc. can be specified at the pod level and overridden at the container level if desired.

Note that as this feature graduates to Beta, support for the annotations will be removed in favor of securityContext fields in podspec. The implication of the removal of support for the Alpha annotations is covered in the Risks and Mitigations section later.

6. A mutating webhook admission controller, GMSAExpander, will act on pod creations. GMSAExpander will look up the GMSACredentialSpec object referred to by name and use the contents in the credspec field to populate the GMSA credential spec JSON if absent or empty in the necessary annotations [in Alpha] or securityContext fields [Beta onwards]. Specifics of the checks performed, fields affected and error scenarios for GMSAExpander is covered in details in the Implementation section below.

7. A validating webhook admission controller, GMSAAuthorizer, will act on pod creations (as well as updates as discussed later in Step 11) and execute a series of checks and authorization around the GMSA annotations [in Alpha] or securityContext fields [in Beta]. Specifics of the annotations or securityContext fields examined and authorizations checks performed along with error scenarios for GMSAAuthorizer is covered in details in the Implementation section below.

8. Kubelet.exe in Windows nodes will examine the credential spec related annotations [in Alpha] or securityContext fields [Beta onwards] for a given pod as well as for each container in the pod. For each container, Kubelet will compute an effective credential spec - either the credential spec specified for the whole pod or a credential spec specified specifically for the container. During Alpha, Kubelet will set the effective credential spec for each container as annotation: <containerName>.container.alpha.windows.kubernetes.io/gmsa-credential-spec. Beta onwards, Kubelet will set the effective credential spec for each container in a new security context field WindowsContainerSecurityContext.CredentialSpec which will require an update to the CRI API. Please see the Implementation section below for details on the enhancements necessary in Kubelet and CRI API.

9. The Windows CRI implementation will access the credential spec JSON through annotations [<containerName>.container.alpha.windows.kubernetes.io/gmsa-credential-spec for Alpha] or securityContext field [WindowsContainerSecurityContext.CredentialSpec Beta onwards] in CreateContainerRequest for each container in a pod. The CRI implementation will transmit the credential spec JSON through a runtime implementation dependent mechanism to a specific container runtime. For example:

• Docker (to be supported in Alpha): will receive the path to a file created on the node's file system under C:\ProgramData\docker\CredentialSpecs\ and populated by dockershim with the credential spec JSON. Docker will read the contents of the credential spec file and pass it to Windows Host Compute Service (HCS) when creating and starting a container in Windows.
• ContainerD (to be supported in Beta): will receive a OCI Spec with windows.CredentialSpec populated by CRIContainerD with the credential spec JSON. The OCI spec will be passed to a OCI runtime like RunHCS.exe when starting a container. Please see the Implementation section below for details on the enhancements necessary for select CRI implementations and their corresponding runtimes.

10. The Windows container runtime (Docker or ContainerD + RunHCS) will depend on Windows Host Compute Service (HCS) APIs/vmcompute.exe to start the containers and assign them user identities corresponding to the GMSA configured for each container. Using the GMSA identity, the processes within the container can authenticate to a service protected by GMSA like database. The containers that fail to start due to invalid GMSA configuration (as determined by HCS or container runtime like Docker) will end up in a failed state with the following when described:

State:              Waiting
      Reas