Cloud Engineering Intern

Overview

CERN runs a large private cloud built on OpenStack that thousands of people depend on for day-to-day computing. Keeping that platform healthy means constantly performing interventions — planned maintenance on the hardware and services underneath the cloud, such as taking a hypervisor out of production, applying firmware or kernel updates, or moving workloads off a host.

Every one of those interventions has consequences for whatever is running on top. I joined the Cloud Infrastructure Team to work on tooling that makes those consequences explicit, and specifically to develop an Intervention Management Framework: a system for modelling, coordinating and scheduling infrastructure interventions in an OpenStack-based cloud environment.

My role

I work as a cloud engineering intern inside the Cloud Infrastructure Team, contributing to the framework as a backend and tooling engineer. My work spans the whole vertical slice: the data model, the service layer, the HTTP API, the command-line interface on top of it, the tests, and the container and pipeline configuration used to build and ship it.

Because the framework sits between infrastructure operations and the people using the cloud, a lot of the role is about modelling the domain correctly — getting the concepts and their relationships right so that the rest of the system stays simple.

The problem

Infrastructure maintenance is straightforward to describe and hard to coordinate. An operator knows which hosts they need to work on and roughly when, but the interesting questions are the downstream ones: which virtual machines are actually sitting on those hosts, who owns them, what can be done about each one, and does any of this collide with maintenance somebody else already scheduled?

Without a shared model of an intervention, that information has to be reconstructed by hand every time. That makes maintenance slow to plan, easy to double-book, and hard to communicate to the people it will affect.

The Intervention Management Framework

The framework treats an intervention as a first-class object with a lifecycle rather than as an entry in a calendar. Once an intervention exists in the system, the affected infrastructure, the affected workloads, the users behind those workloads and the actions available for each of them can all be derived from and attached to it.

• Create and manage scheduled infrastructure interventions
• Track intervention start and end times, and their status through the lifecycle
• Associate the affected infrastructure and resources with an intervention
• Resolve the affected virtual machines and the physical / hypervisor hosts involved
• Identify the users affected by an intervention
• Manage the actions available for each affected resource
• Support actions such as cold migration, reboot, and opening a support ticket workflow
• Record which actions have been selected for each resource
• Help coordinate scheduling and avoid conflicting or overlapping interventions
• Improve visibility of infrastructure maintenance and its impact on users

How the system works

At a high level the flow is a narrowing one. An intervention is declared against some piece of infrastructure; from that the system resolves what is affected, then who is affected, then what can be done, and finally what will be done and when. Each stage is persisted, so an intervention can be inspected and revised before anything is executed.

Architecture and technical approach

The backend is a Python service with a deliberately layered structure: a relational data model describing interventions and the resources they touch, a service layer holding the domain logic, and thin interfaces on top — an HTTP API and a CLI — that both talk to the same services.

Keeping the domain logic out of the interface layer is what makes the CLI and the API behave identically, and it is what makes the logic testable without standing up a cloud. External cloud state is reached through dedicated client code, so the domain model never depends directly on the shape of an upstream API response.

Backend development

The service is written in Python, using FastAPI for the web layer, SQLAlchemy for persistence and Pydantic for validation and serialisation. Pydantic models describe what the system accepts and returns, which keeps validation declarative and gives the API accurate, generated documentation for free.

Dependencies and virtual environments are managed with uv, which keeps environment setup fast and reproducible for everyone working on the project.

API development

The framework exposes a REST API over the intervention lifecycle: creating interventions, updating and querying them, listing the resources and users they affect, and recording the actions selected for each resource.

The API is resource-oriented and typed end to end — request and response schemas are Pydantic models, so the generated OpenAPI description is always in step with the implementation rather than maintained alongside it.

Database and data modelling

Persistence uses MySQL through SQLAlchemy. The interesting part of the data model is relational rather than large: interventions relate to affected resources, resources relate to the hosts and virtual machines behind them, and each affected resource carries the set of actions available to it plus whichever action has been selected.

Modelling time correctly matters here too. Because interventions have a start and an end, overlap detection is a query concern, which pushed me to think carefully about how intervals and statuses are represented.

OpenStack integration

The framework does not own the cloud state it reasons about, so it integrates with services and concepts from the OpenStack ecosystem to obtain it — Compute (Nova) for servers and hypervisors, identity (Keystone) for authentication and for resolving ownership, and Placement for how resources map onto the underlying hardware. Underneath the hypervisors sits KVM virtualisation.

Practically, that meant learning how a real cloud describes itself: the relationship between a server, the project that owns it, and the hypervisor host it lives on, and how an operation such as a cold migration actually plays out for a running instance.

CLI tooling

Alongside the API I built command-line tooling for the intervention-management system, so interventions can be handled directly from a terminal and from scripts. The CLI covers the expected operations — create, update, delete, list and show — and calls the same service layer as the API rather than reimplementing any logic.

For operators this matters more than it sounds: a command that composes with the rest of their shell workflow gets used, whereas a web form in a separate tab often does not.

Testing

Tests are written with pytest and target the domain logic and the API surface. The layered architecture pays off here: because the service layer does not reach out to a live cloud on its own, cloud interactions can be substituted in tests, and the intervention logic — lifecycle transitions, resolving what is affected, detecting overlapping interventions — can be exercised deterministically.

Containerisation

The service is packaged as a container image. Docker is used for local development so the service and its database come up the same way on any machine, while Buildah performs daemonless image builds in CI and images are published to a Harbor registry.

CI/CD and deployment tooling

Changes go through GitLab CI/CD pipelines that lint, test and build the image on every push, so problems surface at merge-request time rather than after deployment.

On the deployment side the work involves Kubernetes as the runtime, Helm for packaging the deployment, Argo CD for GitOps-style continuous delivery, and Vault for secret management. Getting hands on that whole chain — commit, pipeline, image, chart, reconciliation — has been one of the most valuable parts of the role.