Kubernetes Resource Operator

// By James Duffy • February 26, 2025

Table of Contents

This was published shortly after the v0.2.1 version of the Kubernetes Resource Orchestrator was released. This was a very early build and as such the project is expected to make a lot of changes.

# Introduction

Anyone who has talked to me about Kubernetes in the last year knows I will not stop talking about operators and how powerful they can be for taking your applications and building a platform. The problem is that they have always been extremely complicated to get started and even with the Operator Framework it is nontrivial to get off the ground but the new Kubernetes Resource Operator (KRO) aims to change that and it might just be the best way for teams to build a great developer experience and create and manage complex custom Kubernetes resources.

# Before KRO

Over the last few years I have been focused on improving developer experience deploying to Kubernetes clusters. There have been many iterations of the way deployments were handled, each trying to make it easier based on previous learnings.

We first started deploying using Kubernetes yaml manifests in the project repository. They would be automatically deployed using Github Actions after building and pushing a new container. The problem we faced was that it was difficult to make changes per-environment. Next we moved to Kustomize to allow templating and changes between environments. We would instead deploy from the overlay for the environment and have a base folder. Here is an example of how a project’s .deploy folder might look:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


└── .deploy
    ├── base
    |   ├── deployment.yaml
    |   ├── hpa.yaml
    |   └── kustomization.yaml
    └── overlays
        ├── dev
        |   ├── hpa.yaml
        |   └── kustomization.yaml
        ├── stage
        |   ├── hpa.yaml
        |   └── kustomization.yaml
        └── prod
            ├── hpa.yaml
            └── kustomization.yaml

When we started to use more microservices there ended up being a lot of duplication between projects. It made it more difficult to create new services and keep them up to date with our evolving best practices. Projects often created a Deployment, HPA, Service, and Virtual Service.

What we did was to create a Helm chart that each service could use to create the template. We did not use Helm for deployment and management of the service, but instead used it for templating only. For each environment we would have a values file that would be used to create the install.yaml file that would then be deployed.

In GitHub Actions, we ran commands like:

1
2
3


helm template $GITHUB_REPO internal/backend-service --values "values-$GITHUB_REF_NAME.yaml" > install.yaml
kubectl diff -f install.yaml
kubectl apply -f install.yaml

This improved consistency but made it harder for non-infra engineers to understand deployments. Additionally some microservices do not get deployed often and since our chart gets updated frequently the applications do not get updated until they are released. Sometimes it could take months for all of our microservices to get updated to use the latest chart version.

This is where the idea of an internal operator came in: a way to enforce best practices while letting developers focus on their apps. The idea was to decouple managing our best practices with what the developers really care about, getting their application deployed and running. The hope for building our internal operator was that we could release a new version and all of the services deployed using it would also get updated. Instead of having many different values files in their repository they would use a custom resource like:

1
2
3
4
5
6
7
8
9


apiVersion: duffy.xyz/v1alpha1
kind: BackendService
metadata:
  name: my-app
spec:
  replicas: 3
  ingress:
    enabled: true
    host: test.duffy.xyz

To do this we planned on using the Operator Framework powered by Helm. This would let us build on top of what we already had. The main problem is that as powerful as Helm is, our chart had become extremely complex and hard to manage so we were already planning on rebuilding it. What we really wanted was a solution that could be easy enough that non-infrastructure engineers could make changes without needing to involve our small team. This did not seem like it was going to help that.

# Enter the Kubernetes Resource Operator (KRO)

Thinking about all of this for the last year “A New Bombshell Enters The Villa” and their name is Kubernetes Resource Operator (KRO). KRO is a Kubernetes controller that lets you define and manage custom resources using simple YAML-based Resource Graphs, reducing the complexity of building an operator from scratch. It takes a lot of what I wanted to build using the Operator Framework and makes it easier. The setup time to build an operator that does the same thing as KRO would probably be at least a few days if not more, but you could have something with KRO in just a couple hours.

# Example

Let’s build a simple example and you can see just how easy this is:

Install KRO into your cluster:

1
2
3
4


$ helm install kro oci://ghcr.io/kro-run/kro/kro \
--namespace kro \
--create-namespace \
--version=0.2.1

Once installed, you can check that KRO is running with:

1

$ kubectl get pods -n kro

Deploy your first ResourceGraphDefinition by applying the below yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68


apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: httpbin
spec:
  schema:
    group: duffy.xyz
    apiVersion: v1alpha1
    kind: HttpbinApp
    spec:
      name: string | default="httpbin"
      replicas: integer | default=1
      image: string | default="docker.io/mccutchen/go-httpbin:v2.15.0"
    status:
      deploymentConditions: ${deployment.status.conditions}
      availableReplicas: ${deployment.status.availableReplicas}
  resources:
    # Service Account
    - id: serviceAccount
      template:
        apiVersion: v1
        kind: ServiceAccount
        metadata:
          name: ${schema.spec.name}

    # Service
    - id: service
      template:
        apiVersion: v1
        kind: Service
        metadata:
          name: ${schema.spec.name}
          labels:
            app: ${schema.spec.name}
            service: ${schema.spec.name}
        spec:
          selector: ${deployment.spec.template.metadata.labels}
          ports:
            - name: http
              port: 8000
              targetPort: 8080

    # Deployment
    - id: deployment
      template:
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: ${schema.spec.name}
        spec:
          replicas: ${schema.spec.replicas}
          selector:
            matchLabels:
              app: ${schema.spec.name}
              version: v1
          template:
            metadata:
              labels:
                app: ${schema.spec.name}
                version: v1
            spec:
              serviceAccountName: ${serviceAccount.metadata.name}
              containers:
                - name: ${schema.spec.name}
                  image: ${schema.spec.image}
                  imagePullPolicy: IfNotPresent
                  ports:
                    - containerPort: 8080

Confirm the new API resources were created otherwise jump to Troubleshooting

1
2
3
4
5


$ kubectl api-resources
NAME           SHORTNAMES   APIVERSION           NAMESPACED   KIND
...
httpbinapps                 duffy.xyz/v1alpha1   true         HttpbinApp
...

Deploy a custom app:

1
2
3
4
5
6


apiVersion: duffy.xyz/v1alpha1
kind: HttpbinApp
metadata:
  name: demo-httpbin
spec:
  replicas: 2

Check your custom app started

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


$ kubectl describe httpbinapps
Name:         demo-httpbin
Namespace:    kro
Labels:       kro.run/controller-pod-id=kro-pod
              kro.run/kro-version=0.2.1
              kro.run/owned=true
              kro.run/resource-graph-definition-id=d5b5f689-3f49-444e-9180-67567bb5097c
              kro.run/resource-graph-definition-name=httpbin
              kro.run/resource-graph-definition-namespace=
Annotations:  <none>
API Version:  duffy.xyz/v1alpha1
Kind:         HttpbinApp
Metadata:
  Creation Timestamp:  2025-02-27T04:15:10Z
  Finalizers:
    kro.run/finalizer
  Generation:        1
  Resource Version:  27216
  UID:               32baec95-d136-487a-889d-9fe0e3984679
Spec:
  Image:     docker.io/mccutchen/go-httpbin:v2.15.0
  Name:      httpbin
  Replicas:  2
Status:
  Available Replicas:  2
  Conditions:
    Last Transition Time:  2025-02-27T04:15:20Z
    Message:               Instance reconciled successfully
    Observed Generation:   1
    Reason:                ReconciliationSucceeded
    Status:                True
    Type:                  InstanceSynced
  Deployment Conditions:
    Last Transition Time:  2025-02-27T04:15:18Z
    Last Update Time:      2025-02-27T04:15:18Z
    Message:               Deployment has minimum availability.
    Reason:                MinimumReplicasAvailable
    Status:                True
    Type:                  Available
    Last Transition Time:  2025-02-27T04:15:13Z
    Last Update Time:      2025-02-27T04:15:18Z
    Message:               ReplicaSet "httpbin-7b549f7859" has successfully progressed.
    Reason:                NewReplicaSetAvailable
    Status:                True
    Type:                  Progressing
  State:                   ACTIVE
Events:                    <none>

# Troubleshooting

Whenever you are having issues with KRO the easiest thing to do is to review the logs of the controller:

1

kubectl logs -l app.kubernetes.io/name=kro -n kro --all-containers=true -f