Kubernetes Resource Operator
Table of Contents
This was published shortly after the v0.2.1 version of the Kubernetes Resource Orchestrator was released. This was a very early build and as such the project is expected to make a lot of changes.
#Introduction
Anyone who has talked to me about Kubernetes in the last year knows I will not stop talking about operators and how powerful they can be for taking your applications and building a platform. The problem is that they have always been extremely complicated to get started and even with the Operator Framework it is nontrivial to get off the ground but the new Kubernetes Resource Operator (KRO) aims to change that and it might just be the best way for teams to build a great developer experience and create and manage complex custom Kubernetes resources.
#Before KRO
Over the last few years I have been focused on improving developer experience deploying to Kubernetes clusters. There have been many iterations of the way deployments were handled, each trying to make it easier based on previous learnings.
We first started deploying using Kubernetes yaml manifests in the project repository. They would be automatically deployed using Github Actions after building and pushing a new container. The problem we faced was that it was difficult to make changes per-environment. Next we moved to Kustomize to allow templating and changes between environments. We would instead deploy from the overlay for the environment and have a base folder. Here is an example of how a project’s .deploy folder might look:
1└── .deploy
2 ├── base
3 | ├── deployment.yaml
4 | ├── hpa.yaml
5 | └── kustomization.yaml
6 └── overlays
7 ├── dev
8 | ├── hpa.yaml
9 | └── kustomization.yaml
10 ├── stage
11 | ├── hpa.yaml
12 | └── kustomization.yaml
13 └── prod
14 ├── hpa.yaml
15 └── kustomization.yaml
When we started to use more microservices there ended up being a lot of duplication between projects. It made it more difficult to create new services and keep them up to date with our evolving best practices. Projects often created a Deployment, HPA, Service, and Virtual Service.
What we did was to create a Helm chart that each service could use to create the template. We did not use Helm for deployment and management of the service, but instead used it for templating only. For each environment we would have a values file that would be used to create the install.yaml file that would then be deployed.
In GitHub Actions, we ran commands like:
1helm template $GITHUB_REPO internal/backend-service --values "values-$GITHUB_REF_NAME.yaml" > install.yaml
2kubectl diff -f install.yaml
3kubectl apply -f install.yaml
This improved consistency but made it harder for non-infra engineers to understand deployments. Additionally some microservices do not get deployed often and since our chart gets updated frequently the applications do not get updated until they are released. Sometimes it could take months for all of our microservices to get updated to use the latest chart version.
This is where the idea of an internal operator came in: a way to enforce best practices while letting developers focus on their apps. The idea was to decouple managing our best practices with what the developers really care about, getting their application deployed and running. The hope for building our internal operator was that we could release a new version and all of the services deployed using it would also get updated. Instead of having many different values files in their repository they would use a custom resource like:
1apiVersion: duffy.xyz/v1alpha1
2kind: BackendService
3metadata:
4 name: my-app
5spec:
6 replicas: 3
7 ingress:
8 enabled: true
9 host: test.duffy.xyz
To do this we planned on using the Operator Framework powered by Helm. This would let us build on top of what we already had. The main problem is that as powerful as Helm is, our chart had become extremely complex and hard to manage so we were already planning on rebuilding it. What we really wanted was a solution that could be easy enough that non-infrastructure engineers could make changes without needing to involve our small team. This did not seem like it was going to help that.
#Enter the Kubernetes Resource Operator (KRO)
Thinking about all of this for the last year “A New Bombshell Enters The Villa” and their name is Kubernetes Resource Operator (KRO). KRO is a Kubernetes controller that lets you define and manage custom resources using simple YAML-based Resource Graphs, reducing the complexity of building an operator from scratch. It takes a lot of what I wanted to build using the Operator Framework and makes it easier. The setup time to build an operator that does the same thing as KRO would probably be at least a few days if not more, but you could have something with KRO in just a couple hours.
#Example
Let’s build a simple example and you can see just how easy this is:
Install KRO into your cluster:
1helm install kro oci://ghcr.io/kro-run/kro/kro \ 2--namespace kro \ 3--create-namespace \ 4--version=0.2.1Once installed, you can check that KRO is running with:
1kubectl get pods -n kroDeploy your first ResourceGraphDefinition by applying the below yaml
1apiVersion: kro.run/v1alpha1 2kind: ResourceGraphDefinition 3metadata: 4 name: httpbin 5spec: 6 schema: 7 group: duffy.xyz 8 apiVersion: v1alpha1 9 kind: HttpbinApp 10 spec: 11 name: string | default="httpbin" 12 replicas: integer | default=1 13 image: string | default="docker.io/mccutchen/go-httpbin:v2.15.0" 14 status: 15 deploymentConditions: ${deployment.status.conditions} 16 availableReplicas: ${deployment.status.availableReplicas} 17 resources: 18 # Service Account 19 - id: serviceAccount 20 template: 21 apiVersion: v1 22 kind: ServiceAccount 23 metadata: 24 name: ${schema.spec.name} 25 26 # Service 27 - id: service 28 template: 29 apiVersion: v1 30 kind: Service 31 metadata: 32 name: ${schema.spec.name} 33 labels: 34 app: ${schema.spec.name} 35 service: ${schema.spec.name} 36 spec: 37 selector: ${deployment.spec.template.metadata.labels} 38 ports: 39 - name: http 40 port: 8000 41 targetPort: 8080 42 43 # Deployment 44 - id: deployment 45 template: 46 apiVersion: apps/v1 47 kind: Deployment 48 metadata: 49 name: ${schema.spec.name} 50 spec: 51 replicas: ${schema.spec.replicas} 52 selector: 53 matchLabels: 54 app: ${schema.spec.name} 55 version: v1 56 template: 57 metadata: 58 labels: 59 app: ${schema.spec.name} 60 version: v1 61 spec: 62 serviceAccountName: ${serviceAccount.metadata.name} 63 containers: 64 - name: ${schema.spec.name} 65 image: ${schema.spec.image} 66 imagePullPolicy: IfNotPresent 67 ports: 68 - containerPort: 8080Confirm the new API resources were created otherwise jump to Troubleshooting
1kubectl api-resources 2NAME SHORTNAMES APIVERSION NAMESPACED KIND 3... 4httpbinapps duffy.xyz/v1alpha1 true HttpbinApp 5...Deploy a custom app:
1apiVersion: duffy.xyz/v1alpha1 2kind: HttpbinApp 3metadata: 4 name: demo-httpbin 5spec: 6 replicas: 2Check your custom app started
1kubectl describe httpbinapps 2Name: demo-httpbin 3Namespace: kro 4Labels: kro.run/controller-pod-id=kro-pod 5 kro.run/kro-version=0.2.1 6 kro.run/owned=true 7 kro.run/resource-graph-definition-id=d5b5f689-3f49-444e-9180-67567bb5097c 8 kro.run/resource-graph-definition-name=httpbin 9 kro.run/resource-graph-definition-namespace= 10Annotations: <none> 11API Version: duffy.xyz/v1alpha1 12Kind: HttpbinApp 13Metadata: 14 Creation Timestamp: 2025-02-27T04:15:10Z 15 Finalizers: 16 kro.run/finalizer 17 Generation: 1 18 Resource Version: 27216 19 UID: 32baec95-d136-487a-889d-9fe0e3984679 20Spec: 21 Image: docker.io/mccutchen/go-httpbin:v2.15.0 22 Name: httpbin 23 Replicas: 2 24Status: 25 Available Replicas: 2 26 Conditions: 27 Last Transition Time: 2025-02-27T04:15:20Z 28 Message: Instance reconciled successfully 29 Observed Generation: 1 30 Reason: ReconciliationSucceeded 31 Status: True 32 Type: InstanceSynced 33 Deployment Conditions: 34 Last Transition Time: 2025-02-27T04:15:18Z 35 Last Update Time: 2025-02-27T04:15:18Z 36 Message: Deployment has minimum availability. 37 Reason: MinimumReplicasAvailable 38 Status: True 39 Type: Available 40 Last Transition Time: 2025-02-27T04:15:13Z 41 Last Update Time: 2025-02-27T04:15:18Z 42 Message: ReplicaSet "httpbin-7b549f7859" has successfully progressed. 43 Reason: NewReplicaSetAvailable 44 Status: True 45 Type: Progressing 46 State: ACTIVE 47Events: <none>
#Troubleshooting
Whenever you are having issues with KRO the easiest thing to do is to review the logs of the controller:
1kubectl logs -l app.kubernetes.io/name=kro -n kro --all-containers=true -f