Securing the Kubernetes API server. Part 1

By Willy Vidable October 14, 2021

About the API server
The Kubernetes API server is the central component used by all other components and by clients, like kubectl. It provides a CRUD (Create, Read, Update, Delete) interface for querying and modifying the cluster state over a RESTful API. It stores that state in the etcd database.
In addition to providing a consistent way of storing objects in etcd, it also performs validation of those objects, so clients can’t store improperly configured objects (which they could if they were writing to the store directly). Along with validation, it also handles optimistic locking, so changes to an object are never overridden by other clients in the event of concurrent updates.
One of the API server’s clients is the command-line tool kubectl. When creating a resource from a JSON file, for example, kubectl posts the file’s contents to the API server through an HTTP POST request. The figure below shows what happens inside the API server when it receives the request.

Authenticating the client with Authentication Plugins
First, the API server needs to authenticate the client sending the request. This is performed by one or more authentication plugins configured in the API server. The API server calls these plugins in turn, until one of them determines who is sending the request. It does this by inspecting the HTTP request.
Depending on the authentication method, the user can be extracted from the client’s certificate or an HTTP header, such as Authorization. The plugin extracts the client’s username, user ID, and groups the user belongs to. This data is then used in the next stage, which is authorization.

Authorizing the Client with Authorization Plugins
Besides authentication plugins, the API server is also configured to use one or more authorization plugins. Their job is to determine whether the authenticated user can perform the requested action on the requested resource. For example, when creating pods, the API server consults all authorization plugins in turn, to determine whether the user can create pods in the requested namespace. As soon as a plugin says the user can perform the action, the API server progresses to the next stage.

Validating And/Or Modifying the Resource in the Request with Admission Control Plugins
If the request is trying to create, modify, or delete a resource, the request is sent through Admission Control. Again, the server is configured with multiple Admission Control plugins. These plugins can modify the resource for different reasons. They may initialize fields missing from the resource specification to the configured default values or even override them. They may even modify other related resources, which aren’t in the request, and can also reject a request for whatever reason. The resource passes through all Admission Control plugins.
Notice that when the request is only trying to read data, the request doesn’t go through the Admission Control.
Examples of Admission Control plugins include

AlwaysPullImages—Overrides the pod’s imagePullPolicy to Always, forcing the image to be pulled every time the pod is deployed.
ServiceAccount—Applies the default service account to pods that don’t specify it explicitly.
NamespaceLifecycle—Prevents creation of pods in namespaces that are in the process of being deleted, as well as in non-existing namespaces.
ResourceQuota—Ensures pods in a certain namespace only use as much CPU and memory as has been allotted to the namespace.

You’ll find a list of additional Admission Control plugins in the Kubernetes documentation at https://kubernetes.io/docs/admin/admission-controllers/.

Validating the Resource and Storing it Persistently
After letting the request pass through all the Admission Control plugins, the API server then validates the object, stores it in etcd, and returns a response to the client.

Understanding Authentication
We said the API server can be configured with one or more authentication plugins (and the same is true for authorization plugins). When a request is received by the API server, it goes through the list of authentication plugins, so they can each examine the request and try to determine who’s sending the request. The first plugin that can extract that information from the request returns the username, user ID, and the groups the client belongs to back to the API server core. The API server stops invoking the remaining authentication plugins and continues onto the authorization phase.
Several authentication plugins are available. They obtain the identity of the client using the following methods:

Validating the Resource and Storing it Persistently
After letting the request pass through all the Admission Control plugins, the API server then validates the object, stores it in etcd, and returns a response to the client.

From the client certificate
From an authentication token passed in an HTTP header
Others

The authentication plugins are enabled through command-line options when starting the API server.

Users and groups
An authentication plugin returns the username and group(s) of the authenticated user. Kubernetes doesn’t store that information anywhere; it uses it to verify whether the user is authorized to perform an action or not.

Understanding Users
Kubernetes distinguishes between two kinds of clients connecting to the API server:

Actual humans (users)
Pods (more specifically, applications running inside them)

Both these types of clients are authenticated using the already mentioned authentication plugins. Users are meant to be managed by an external system, such as a Single Sign On (SSO) system, but the pods use a mechanism called service accounts, which are created and stored in the cluster as ServiceAccount resources. In contrast, no resource represents user accounts, which means you can’t create, update, or delete users through the API server.
It is assumed that a system that is independent from the cluster, manages normal Users in the following ways:

an administrator distributing private keys
a user store like Keystone or Google Accounts
a file with a list of usernames and passwords

In this regard, Kubernetes does not have objects which represent normal user accounts. Normal users cannot be added to a cluster through an API call.

Even though a normal user cannot be added via an API call, any user that presents a valid certificate signed by the cluster's certificate authority (CA) is considered authenticated.

In this configuration, Kubernetes determines the username from the common name field in the 'subject' of the cert (e.g., "/CN=bob"). From there, the role-based access control (RBAC) sub-system would determine whether the user is authorized to perform a specific operation on a resource. For more details, refer to the normal users topic in certificate request for more details about this.

In contrast, service accounts are users managed by the Kubernetes API. They are bound to specific namespaces, and created automatically by the API server or manually through API calls. Service accounts are tied to a set of credentials stored as Secrets, which are mounted into pods allowing in-cluster processes to talk to the Kubernetes API.

API requests are tied to either a normal user or a service account, or are treated as anonymous requests. This means every process inside or outside the cluster, from a human user typing kubectl on a workstation, to kubelets on nodes, to members of the control plane, must authenticate when making requests to the API server, or be treated as an anonymous user.

Understanding Groups
Both human users and ServiceAccounts can belong to one or more groups. We’ve said that the authentication plugin returns groups along with the username and user ID.
Groups are used to grant permissions to several users at once, instead of having to grant them to individual users.
Groups returned by the plugin are nothing but strings, representing arbitrary group names, but built-in groups have special meaning:

The system:unauthenticated group is used for requests where none of the authentication plugins could authenticate the client.
The system:authenticated group is automatically assigned to a user who was authenticated successfully.
The system:serviceaccounts group encompasses all ServiceAccounts in the system.
The system:serviceaccounts:includes all ServiceAccounts in a specific namespace.

Introducing ServiceAccounts
Let’s explore ServiceAccounts in detail. You’ve already learned that the API server requires clients to authenticate themselves before they’re allowed to perform operations on the server. And you’ve already seen how pods can authenticate by sending the contents of the file /var/run/secrets/kubernetes.io/serviceaccount/token, which is mounted into each container’s filesystem through a secret volume.

But what exactly does that file represent? Every pod is associated with a Service-Account, which represents the identity of the app running in the pod. The token file holds the ServiceAccount’s authentication token. When an app uses this token to connect to the API server, the authentication plugin authenticates the ServiceAccount and passes the ServiceAccount’s username back to the API server core. Service-Account usernames are formatted like this:

system:serviceaccount:<namespace>:<service account name>

The API server passes this username to the configured authorization plugins, which determine whether the action the app is trying to perform is allowed to be performed by the ServiceAccount.
ServiceAccounts are nothing more than a way for an application running inside a pod to authenticate itself with the API server. As already mentioned, applications do that by passing the ServiceAccount’s token in the request.

Understanding the ServiceAccount Resource
ServiceAccounts are resources just like Pods, Secrets, ConfigMaps, and so on, and are scoped to individual namespaces. A default ServiceAccount is automatically created for each namespace (that’s the one your pods have used all along).
You can list ServiceAccounts like you do other resources:

As you can see, the current namespace only contains the default ServiceAccount. Additional ServiceAccounts can be added when required. Each pod is associated with exactly one ServiceAccount, but multiple pods can use the same ServiceAccount. As you can see in figure below a pod can only use a ServiceAccount from the same namespace.

Understanding how ServiceAccounts tie into Authorization
You can assign a ServiceAccount to a pod by specifying the account’s name in the pod manifest. If you don’t assign it explicitly, the pod will use the default ServiceAccount in the namespace.
By assigning different ServiceAccounts to pods, you can control which resources each pod has access to. When a request bearing the authentication token is received by the API server, the server uses the token to authenticate the client sending the request and then determines whether or not the related ServiceAccount is allowed to perform the requested operation. The API server obtains this information from the system-wide authorization plugin configured by the cluster administrator. One of the available authorization plugins is the role-based access control (RBAC) plugin, which we will discuss later. From Kubernetes version 1.6 on, the RBAC plugin is the plugin most clusters should use.

Creating ServiceAccounts
We’ve said every namespace contains its own default ServiceAccount, but additional ones can be created if necessary. But why should you bother with creating Service-Accounts instead of using the default one for all your pods?
The obvious reason is cluster security. Pods that don’t need to read any cluster metadata should run under a constrained account that doesn’t allow them to retrieve or modify any resources deployed in the cluster. Pods that need to retrieve resource metadata should run under a ServiceAccount that only allows reading those objects’ metadata, whereas pods that need to modify those objects should run under their own ServiceAccount allowing modifications of API objects.
Let’s see how you can create additional ServiceAccounts, how they relate to Secrets, and how you can assign them to your pods.

Creating a ServiceAccount
Creating a ServiceAccount is incredibly easy, thanks to the dedicated kubectl create serviceaccount command. Let’s create a new ServiceAccount called foo:

Now, you can inspect the ServiceAccount with the describe command, as shown in the following listing.

You can see that a custom token Secret has been created and associated with the ServiceAccount. If you look at the Secret’s data with kubectl describe secret foo-token-f5p4x, you’ll see it contains the same items (the CA certificate, namespace, and token) as the default ServiceAccount’s token does, as shown in the following listing.

Note that the authentication tokens used in ServiceAccounts are JSON Web Tokens (JWT). JWT is an open standard used to share security information between two parties, a client, and a server. Each JWT contains encoded JSON objects, including a set of claims. JWTs are signed using a cryptographic algorithm to ensure that the claims cannot be altered after the token is issued.

Assigning a ServiceAccount to a pod
After you create additional ServiceAccounts, you need to assign them to pods. This is done by setting the name of the ServiceAccount in the spec.serviceAccountName field in the pod definition. Notice that a pod’s ServiceAccount must be set when creating the pod. It can’t be changed later.

Creating a Pod which uses a Custom ServiceAccount
Let’s deploy a pod that ran a container based on the tutum/curl image and an ambassador container alongside it. We can use this pod to explore the API server’s REST interface. The ambassador container run the kubectl proxy process, which use the pod’s ServiceAccount’s token to authenticate with the API server.
We will set the pod to use the foo ServiceAccount we created above. The listing below shows the pod definition.

To confirm that the custom ServiceAccount’s token is mounted into the two containers, you can print the contents of the token as shown in the following listing.

You can see the token is the one from the foo ServiceAccount by comparing the token string with the previous listing.

Using the Custom Serviceaccount’s Token to talk to the Api Server
Let’s see if you can talk to the API server using this token. As mentioned previously, the ambassador container uses the token when talking to the server, so you can test the token by going through the ambassador, which listens on localhost:8001, as shown in the following listing.

As shown above, you got back a response from the server. Notice that I did not show the whole response that could be either success which means the custom ServiceAccount is allowed to list pods or an error. In this case would be because your cluster doesn’t use the RBAC authorization plugin that we will talk about later, or you gave all ServiceAccounts full permissions by disabling RBAC authorization plugin by using something called permissive-binding.
When your cluster isn’t using proper authorization, creating and using additional ServiceAccounts doesn’t make much sense, since even the default ServiceAccount is allowed to do anything. The only reason to use ServiceAccounts in that case is to enforce mountable Secrets or to provide image pull Secrets through the Service-Account.
But creating additional ServiceAccounts is practically a must when you use the RBAC authorization plugin.

REFERENCES
Kubernetes Documentation. Kubernetes.io/docs
Marko Luksa. Kubernetes in Action. 2018 Edition