kubectl inside your container
Tags: #kubernetes #kubectl
2 July 2023
A big thanks for Marcus for this idea, I was not aware of the kubectl implementation. kubectl is also capable of authentication to apiserver from inside the pod(rest.InClusterConfig). This is great because you'll no longer need to invoke curl to the apiserver endpoints using multiple certs. I'll tell you about the curl method on which I used to rely in the past.
In this blog, I'll cover how to add kubectl inside your running container. I'll cover alpine based scenarios but you can add it to others as well.
Why?
You'll need this mostly to test our RBAC permissions corresponding to your serviceAccount. I was working on an issue to remove cluster-admin dependencies from kubearmor daemonsets. This was because if you're using cluster-admin permissions for your application then any admission controllers out there will flag it out. It's not a good security practice to use cluster-admin. After working and finalizing the required permissions, it came the time to test whether everything is working correct or not. For testing, I used to use the following method which was very tedious. I use to hit different endpoints and check whether it's giving me a success or a forbidden response. For example, to check whether we have permissions to list the nodes or not, I used to invoke the following command from inside the pod.
curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
--header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
https://kubernetes.default.svc/api/v1/nodes
Demo
Let's see this in action. I'll create a test namespace and then we will create a pod and add kubectl to it. For this purpose, I'll use nginx:alpine image.
Assuming that you have a running cluster, let's create a test namespace.
kubectl create namespace test
Now, let's create a service account first.
kubectl -n test create sa test-sa
After this we will create a role which will have get and list permissions for pods and configmaps in the test namespace.
kubectl -n test create role test-role --resource=pods,configmaps --verb=get,list
We will create a rolebinding after this to bind our role.
kubectl -n test create rolebinding test-rb --role=test-role --serviceaccount=test:test-sa
Finally, we will create a pod to test everything out. I'll use the imperative method to create a pod manifest first.
kubectl -n test run test-pod --image=nginx:alpine --dry-run=client -oyaml
As of now, the pod is using the default service account which is not good. Add serviceAccountName: test-sa in pod.spec to override the default service account.
You final pod manifest will look like this.
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
labels:
run: test-pod
name: test-pod
namespace: test
spec:
containers:
- image: nginx:alpine
name: test-pod
resources: {}
dnsPolicy: ClusterFirst
serviceAccountName: test-sa
restartPolicy: Always
status: {}
Save it and apply it using kubectl apply.
Create the pod and exec into it.
kubectl -n test exec -it test-pod -- sh
Now if you will try to add kubectl then you'll get an error like this.
/ # apk add kubectl
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
ERROR: unable to select packages:
kubectl (no such package):
required by: world[kubectl]
Let's resolve the error by adding the edge repository to /etc/apk/repositories
/ # echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" | tee -a /etc/apk/repositories
http://dl-cdn.alpinelinux.org/alpine/edge/community
/ # cat /etc/apk/repositories
https://dl-cdn.alpinelinux.org/alpine/v3.17/main
https://dl-cdn.alpinelinux.org/alpine/v3.17/community
http://dl-cdn.alpinelinux.org/alpine/edge/community
You can see that it's added now.
Let's make another attempt to install kubectl
/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
v3.17.4-29-ga78582e6ba9 [https://dl-cdn.alpinelinux.org/alpine/v3.17/main]
v3.17.4-28-g19c69fee4bb [https://dl-cdn.alpinelinux.org/alpine/v3.17/community]
v3.18.0-4970-g485ba79de1f [http://dl-cdn.alpinelinux.org/alpine/edge/community]
OK: 32443 distinct packages available
/ # apk add kubectl
(1/1) Installing kubectl (1.27.2-r0)
Executing busybox-1.35.0-r29.trigger
OK: 95 MiB in 63 packages
Now if you will test then you'll see that you're able to get and list pods and configmaps respectively but cannot perform any other operation using kubectl.
/ # kubectl get pods
NAME READY STATUS RESTARTS AGE
test-pod 1/1 Running 0 85s
/ # kubectl get ns
Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:test-sa:" cannot list resource "namespaces" in API group "" at the cluster scope
This way, we can easily test out RBAC permissions from inside the cluster.