As a developer, launching a full ecosystem is crucial to test changes properly. However, as dependencies grow, these systems become hard to launch locally. Either because they consume a lot of resources or because they're complex to launch. This pushes developers to either not test properly or rely on shared environments (like dev), which can impact their team members.

At Moniepoint, we built a tool that automates resource deployment on a remote cluster, helping developers test features efficiently with minimal configuration.

It’s as straightforward as running an eshu workload with a clear, YAML-based workload definition, making the process accessible and manageable.

Launching a workload

After users install our custom CLI, the first time they run eshu workload run, it generates a random workload name/ID for the run. This ensures users don't clash with other users' workloads on the same cluster. This ID is stored in a local config file, so that we can refer back to it when needed. Next, the system loads local YAML files to create the workload. Users write these files based on the Score specification. Each YAML file contains 5 parts: apiVersion, metadata, containers, service and resources. As an example, let’s examine this workload file below:

The apiVersion identifies the version of the workload specification. The metadata portion lets you add annotations and similar metadata to the created resources, and it also specifies the workload name so other workloads can reference it. The containers section includes the images and environment configuration for our app, while the services section specifies which ports we want to expose. The resources part includes the standard resources we want to launch with the workload. Examples include Postgres or MySQL databases, Redis, Kafka, and many others.

Let’s take a look at another example. The goal is to create a chat app that includes a front-end component, a backend API, and a backend identity service for user identities. The chat app includes a dedicated database and Redis for caching, while the identity service has its own database.

To do this, we will launch the workload file above for the identity service, plus another that includes 2 containers for the front-end and backend components, plus a Redis and Postgres resource. The chat file also includes a resource that references the identity workload.

When the eshu workload runs, it merges the files into a single file. Then file references are resolved. Although the example inlines the database script, you can also import it from a separate file with a file reference (${file:initscript.sql}). After this injection, the CLI validates the workload contents against a predefined schema. This helps the user quickly correct mistakes without making any request to the cluster.

After we have a valid merged YAML definition, the CLI sends the workload through an HTTP request to our server in the remote Kubernetes cluster. This lets us create it with a single workload run command.

When the server receives the request, it first creates a namespace named after the workload ID using the Kubernetes API. It then copies any relevant secrets, like the secret used to pull images from the internal Moniepoint Artifactory, into this namespace. 

In our example, fe65c057-45760157-careful-weasel is the name of a dedicated namespace where our resources exist.

After creating the namespace, Crossplane creates the XWorkload custom resource from our YAML definition in the Score Workloads attribute in the metadata section.

Creating this resource kicks off the Crossplane composition pipeline we configured.

Composition Pipeline

The XWorkload includes the YAML file contents in the Score Workloads section. With it, Crossplane calls our custom function for each workload created, and our function parses that score file and outputs the relevant Kubernetes resources.

Using the crossplane function-sdk-go, the function needs to implement the RunFunction signature, receive the current request with context, and return the desired resources response.

Our custom function has 4 main tasks:

• It reads and parses the YAML file contents from the request (present in the score workloads attribute). 
• Afterwards, it adds the required Kubernetes resources (deployments, volumes, …) to the response. It generates a map of environment variables for all resources so it can inject them into the containers. This map includes the database username and passwords, which the function generates automatically.
• Then the containers are added.
• Finally, the required services are created to expose the resources and the proxy so it is reachable from their machine.

The main effort with this function was supporting more and more resources over time. Beyond the resource itself, we also added auxiliary components to ensure users had the same functionality across different resources. For example, while PostgreSQL has an easy way to initialise data via setting up a script file when mounting, the Google Spanner emulator does not. So a job is launched after the emulator starts to connect to the pod and run the provided initialisation script. For the user, there is no difference in how to initialise one or the other. To manage this complexity, each resource implements the same interface:

For our Postgres example above, the implementation will create a secret with randomly generated credentials and store that secret in Vault, create a config map for the SQL initialisation script, create a service to expose the Postgres pod, and create a stateful set that includes a PVC for persisting data. Since this is Postgres, the config map is mounted into the container at/docker-entrypoint-initdb.d/init.sql, and the generated credentials are passed as env vars to initialise the user. For other resources, like Spanner, a job initialises the data so the user has a unified interface. 

Then the actual function flow is a simple chain of responsibility to generate all the resources:

With these 2 simple abstractions, we can extend the function to support more and more resources with minimal changes.

Regarding the environment, each resource adds to the env map. Then, these values are injected into the containers via string interpolation, replacing the string resources.[name].[key]. For example, in our demo above, we had a Postgres database resource called pg. The container's environment used resources.pg.host, resources.pg.username and resources.pg.password.

In our example, the chat component depends on the identity component, even though they are in different YAML files. To launch them with a single command, we use a special resource called a service that references a workload by name. This service adds the target workload's host and port to the environment, so one container can easily reach another.

For the namespace, we can see that pods (deployments) were created for each app (chat with 2 containers and identity with 1) and for each resource. We can also see a proxy that handles traffic redirection from the port-forward to each pod.

This proxy is created as an Nginx pod with randomised ports for each resource and service from the user YAML file. It sets up the namespace resources so the user can port-forward to them.

Port forwarding

After creating the XWorload resource, the HTTP request terminates with a success status code. At this point, the resources aren't created yet because the composition hasn't run.

So the CLI starts a loop that checks the workload status, with a sensible timeout.

To get this status, the CLI uses the workload name mentioned above. It sends a request to the server that identifies the namespace based on that workload name. With the Kubernetes API, it queries all resources in that namespace with the Crossplane label spec.crossplane.resourceRefs and returns their status to the CLI.

When all resources are Running, then the CLI can try to port forward. To port forward, we rely on the nginx proxy that we saw before. The proxy is identified by name in that namespace, and each available port to port forward to is a port on that proxy pod. For each port, the CLI opens a connection and forwards traffic to it. 

Let’s recap:
⁠
⁠We specified what we want to create in the YAML files. As the example files show, the configuration is minimal (no Postgres version specified, no configuration environment variables, no resource or probe specifications). All of these resources were created in a remote cluster (not my local computer) and accessible via port forwarding.

Generating dynamic credentials

Credentials for resources, like database passwords, are randomly generated during provisioning. To make them accessible to the user, we store them in Vault. The CLI includes a command to retrieve a dynamically generated credential from Vault in a single command. 

This is done by configuring the Vault database secrets engine, so when the eshu workload endpoint credential command runs, the CLI requests the control plane server, which uses the Vault API to generate new temporary credentials. 

For Postgres, for example, we use the postgresql-database-plugin, and for each dynamic credential we create the user with:

As with other functionalities, we offer a unified interface. Resources without a Vault secrets engine still work the same way, but the credentials retrieved are static. In that case, it retrieves the credentials generated during resource provisioning.

Clean up of namespaces

Since these resources are hosted in a remote cluster, we need to ensure they get cleaned up if the user never does it. To ensure that, we have a dedicated server that implements the Kubernetes type-reconciler interface and is associated with the created namespace. 

Each workload resource has an annotation with a target TTL. This server checks whether the namespace has already exceeded the target TTL. If so, it deletes the namespace and vault secrets. 

Challenges & benefits

One of the first challenges we ran into was password generation. Crossplane has a render cycle: every 60 seconds, the XWorkload is reprocessed, and any changes are reflected in the cluster. Since we are generating random information, for passwords, for example, this caused the environment to be outdated on the subsequent render cycle. In fact, anything that changed the desired resources on every cycle would cause the pods in the cluster to keep restarting. So it was important to ensure the function would eventually reach a steady state. For password generation, we handled that by checking the existing state of the resources and reusing the password if it was already present.

Another challenge was avoiding clashes when so many different resources are being created. We could force users to use different names for every port and container, but that would create a worse experience, especially when workloads are shared across teams. So instead, we decided to compose the names so they need to be unique only within the workload context. This quickly hit the 63-character limit for Kubernetes names. We fixed that by capping each part of the composed name at a specific character length.

So what benefits did we actually get from all of this, when compared with something like docker compose or test containers?

First, these workloads are simpler to set up. The resources themselves require minimal configuration, and many defaults are quite sensible for most use cases. Especially when experimenting with a new technology that you might not be quite familiar with, this minimal configuration was helpful.

Another benefit is the unified resource interface. As mentioned earlier, you initialise database data via the same input attribute, even if the underlying database doesn't support it out of the box. This simplifies that process so that users can focus on what they are actually working on.

All the resources are also being launched in a remote cluster. In heavier apps, this makes quite a difference.

Lastly, all of these definitions are shareable, and a workload can import another. This lets teams more easily spin off full environments for their services. For now, this is a YAML file; in the future, we can build a centralised catalogue.