Kubernetes Custom Resources: Extending the Kubernetes API

Kubernetes is renowned for its flexibility and extensibility. While it provides a robust set of built-in resources (like Pods, Services, and Deployments), many organizations need to manage domain-specific objects that Kubernetes doesn’t natively understand. This is where Custom Resources and Custom Resource Definitions (CRDs) come into play.

What Are Kubernetes Custom Resources?

A custom resource is an extension of the Kubernetes API that allows you to define and manage your own resource types, just like built-in resources. These resources are not available by default in Kubernetes, but you can add them dynamically to your cluster to represent any object your application needs to manage.

For example, you might define a custom resource called Database to represent database instances, or Widget for a business-specific object.

Custom Resource Definitions (CRDs): The Foundation

A Custom Resource Definition (CRD) is the Kubernetes object that defines a new custom resource type. When you create a CRD, Kubernetes automatically creates a new RESTful API endpoint for your custom resource, making it possible to use kubectl and other Kubernetes tools to create, read, update, and delete instances of your custom resource.

Key Features of CRDs:

API Extension: CRDs extend the Kubernetes API without requiring you to write your own API server.

Declarative Management: Manage custom resources using YAML manifests, just like built-in resources.

Schema Validation: Enforce structure and data types using OpenAPI schemas, reducing misconfigurations.

Namespaced or Cluster Scope: Custom resources can be namespaced (isolated to a namespace) or cluster-scoped (available cluster-wide).

CRUD Operations: You can create, read, update, and delete custom resources using standard Kubernetes commands.

How CRDs Work: From Definition to Usage

Define the CRD:

Create a YAML manifest that specifies the name, group, version, scope, and schema for your custom resource.

apiVersion: apiextensions.k8s.io/v1

kind: CustomResourceDefinition

metadata:

name: widgets.example.com

spec:

group: example.com

versions:

- name: v1

served: true

storage: true

schema:

openAPIV3Schema:

type: object

properties:

spec:

type: object

properties:

name:

type: string

replicas:

type: integer

scope: Namespaced

names:

plural: widgets

singular: widget

kind: Widget

shortNames:

- wg

Apply the CRD:

Use kubectl apply -f widget-crd.yaml to register the new resource type with the Kubernetes API.

Create Custom Resources:

Define instances of your new resource type (e.g., my-widget.yaml) and apply them with kubectl apply -f my-widget.yaml.

Access and Manage:

List and manage your custom resources using standard commands, such as kubectl get widgets or kubectl describe widget my-widget.

Custom Controllers: Automating Behavior

While CRDs define the schema and API for your custom resource, they don’t specify how Kubernetes should act on changes to these resources. For automation, you need a custom controller—a program (often deployed as a Pod) that watches for changes to your custom resources and takes action to reconcile the desired state. This is the foundation of the Operator pattern, which automates complex application lifecycle management.

Best Practices for CRDs

Versioning: Use API versioning (v1alpha1, v1beta1, v1) to introduce changes safely and maintain backward compatibility.

Schema Validation: Define detailed OpenAPI schemas to validate resource fields and prevent invalid data.

RBAC: Implement Role-Based Access Control to restrict who can access or modify your custom resources.

Testing: Perform unit and integration testing to ensure your CRDs and controllers behave as expected, especially as your cluster scales.

Status Subresource: Use the status subresource to separate status information from the spec, improving observability and avoiding conflicts.

Real-World Use Cases

Service Meshes: Istio uses CRDs like VirtualService to define traffic routing rules.

Operators: Database operators (e.g., for MySQL or MongoDB) use CRDs to represent database instances and automate their lifecycle.

Custom Workflows: Any domain-specific automation, such as provisioning cloud resources or managing internal business objects, can leverage CRDs.

Conclusion

Kubernetes Custom Resources and CRDs unlock the true power of Kubernetes as a platform—not just for containers, but for any kind of resource you need to manage. By extending the Kubernetes API, you can build scalable, robust, and declarative automation tailored to your organization’s needs