Editing Running Pods in k8s

30 December 2022

There are a number of times when you want to edit pods that are crashing on the go, but you can't. You cannot edit everything from a running/crashing pod in Kubernetes. You can only edit certain fields while you use kubectl edit

Let me explain this to you using an example: Let's run an imperative command to run a pod based on busybox image which will sleep for 1 hour.

kubectl run pod --image=busybox -- sleep 3600

If you are interested in pod manifest, then you can use --dry-run=client and -oyaml to get the output of your pod manifest.

$ kubectl run pods --image=busybox --dry-run=client -oyaml -- sleep 3600
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: pods
  name: pods
spec:
  containers:
  - args:
    - sleep
    - "3600"
    image: busybox
    name: pods
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

Now comes the interesting part, let's try to run a pod and this time we will intentionally pass the wrong command.

kubectl run wrongpod --image=busybox -- sleepy 3600

Notice the error while passing the command, it should not be sleepy Now if you see the status of the pod after few seconds then you will see this.

$ kubectl get pods 
wrongpod   0/1     CrashLoopBackOff   1 (12s ago)    13s

At this point of time, you might want to jump and edit the command. Let's try that.

$ kubectl edit pod wrongpod

When you'll attempt to do this, you will notice that you can't. You see another vim session popping up with an information message that says the following.

spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)

To summarize the above message, it says that: There are only certain fields in the pod specification, where you're allowed to make changes which includes

Container images
Init container images
Tolerations
spec.activeDeadlineSeconds
spec.terminationGracePeriodSeconds

From whatever little I've learnt till now, it's the tolerations that you're going to use the most.

What to do if you're not able to edit the pods?

There are a number of approaches to this, but here's what I do. I get the YAML manifest of a running pod using the following command.

kubectl get pod running-pod -oyaml >> pod.yaml

After this I use vim and then I delete everything under the status section, make some changes and apply the manifest again.

I hope you'll keep this things in your mind while editing pods.