kubectl inside your container

2 July, 2023 - Tags: kubernetes, kubectl

A big thanks for Marcus for this idea, I was not aware of the kubectl implementation. kubectl is also capable of authentication to apiserver from inside the pod(rest.InClusterConfig). This is great because you'll no longer need to invoke curl to the apiserver endpoints using multiple certs. I'll tell you about the curl method on which I used to rely in the past.

In this blog, I'll cover how to add kubectl inside your running container. I'll cover alpine based scenarios but you can add it to others as well.

Why?

You'll need this mostly to test our RBAC permissions corresponding to your serviceAccount. I was working on an issue to remove cluster-admin dependencies from kubearmor daemonsets. This was because if you're using cluster-admin permissions for your application then any admission controllers out there will flag it out. It's not a good security practice to use cluster-admin. After working and finalizing the required permissions, it came the time to test whether everything is working correct or not. For testing, I used to use the following method which was very tedious. I use to hit different endpoints and check whether it's giving me a success or a forbidden response. For example, to check whether we have permissions to list the nodes or not, I used to invoke the following command from inside the pod.

curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
--header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
https://kubernetes.default.svc/api/v1/nodes

Demo

Let's see this in action. I'll create a test namespace and then we will create a pod and add kubectl to it. For this purpose, I'll use nginx:alpine image. Assuming that you have a running cluster, let's create a test namespace.

kubectl create namespace test

Now, let's create a service account first.

kubectl -n test create sa test-sa

After this we will create a role which will have get and list permissions for pods and configmaps in the test namespace.

kubectl -n test create role test-role --resource=pods,configmaps --verb=get,list

We will create a rolebinding after this to bind our role.

kubectl -n test create rolebinding test-rb --role=test-role --serviceaccount=test:test-sa

Finally, we will create a pod to test everything out. I'll use the imperative method to create a pod manifest first.

kubectl -n test run test-pod --image=nginx:alpine --dry-run=client -oyaml

As of now, the pod is using the default service account which is not good. Add serviceAccountName: test-sa in pod.spec to override the default service account.

You final pod manifest will look like this.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: test-pod
  name: test-pod
  namespace: test 
spec:
  containers:
  - image: nginx:alpine
    name: test-pod
    resources: {}
  dnsPolicy: ClusterFirst
  serviceAccountName: test-sa
  restartPolicy: Always
status: {}

Save it and apply it using kubectl apply.

Create the pod and exec into it.

kubectl -n test exec -it test-pod -- sh

Now if you will try to add kubectl then you'll get an error like this.

/ # apk add kubectl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
ERROR: unable to select packages:
  kubectl (no such package):
    required by: world[kubectl]

Let's resolve the error by adding the edge repository to /etc/apk/repositories

/ # echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" | tee -a /etc/apk/repositories
http://dl-cdn.alpinelinux.org/alpine/edge/community

/ # cat /etc/apk/repositories
https://dl-cdn.alpinelinux.org/alpine/v3.17/main
https://dl-cdn.alpinelinux.org/alpine/v3.17/community
http://dl-cdn.alpinelinux.org/alpine/edge/community

You can see that it's added now.

Let's make another attempt to install kubectl

/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.17.4-29-ga78582e6ba9 [https://dl-cdn.alpinelinux.org/alpine/v3.17/main]
v3.17.4-28-g19c69fee4bb [https://dl-cdn.alpinelinux.org/alpine/v3.17/community]
v3.18.0-4970-g485ba79de1f [http://dl-cdn.alpinelinux.org/alpine/edge/community]
OK: 32443 distinct packages available

/ # apk add kubectl
(1/1) Installing kubectl (1.27.2-r0)
Executing busybox-1.35.0-r29.trigger
OK: 95 MiB in 63 packages

Now if you will test then you'll see that you're able to get and list pods and configmaps respectively but cannot perform any other operation using kubectl.

/ # kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
test-pod   1/1     Running   0          85s

/ # kubectl get ns
Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:test-sa:" cannot list resource "namespaces" in API group "" at the cluster scope

This way, we can easily test out RBAC permissions from inside the cluster.