containerd Design

Architecture

To ensure clean separation of concerns, we have organized the units of containerd's behavior into components. Components are roughly organized into subsystems. Components that bridge subsystems may be referred to as modules. Modules typically provide cross-cutting functionality, such as persistent storage or event distribution. Understanding these components and their relationships is key to modifying and extending the system.

This document will cover very high-level interaction. For details on each module, please see the relevant design document.

The main goal of this architecture is to coordinate the creation and execution of bundles. Bundles contain configuration, metadata and root filesystem data and are consumed by the runtime. A bundle is the on-disk representation of a runtime container. Bundles are mutable and can be passed to other systems for modification or packed up and distributed. In practice, it is simply a directory on the filesystem.

Note that while these architectural ideas are important to understand the system, code layout may not reflect the exact architecture. These ideas should be used as a guide for placing functionality and behavior and understanding the thought behind the design.

Subsystems

External users interact with services, made available via a GRPC API.

• Bundle: The bundle service allows the user to extract and pack bundles from disk images.
• Runtime: The runtime service supports the execution of bundles, including the creation of runtime containers.

Typically, each subsystem will have one or more related controller components that implement the behavior of the subsystem. The behavior of the subsystem may be exported for access via corresponding services.

Modules

In addition to the subsystems, we have several components that may cross subsystem boundaries, referenced to as components. We have the following components:

• Executor: The executor implements the actual container runtime.
• Supervisor: The supervisor monitors and reports container state.
• Metadata: Stores metadata in a graph database. Use to store any persistent references to images and bundles. Data entered into the database will have schemas coordinated between components to provide access to arbitrary data. Other functionality includes hooks for garbage collection of on-disk resources.
• Content: Provides access to content addressable storage. All immutable content will be stored here, keyed by content hash.
• Snapshot: Manages filesystem snapshots for container images. This is analogous to the graphdriver in Docker today. Layers are unpacked into snapshots.
• Events: Supports the collection and consumption of events for providing consistent, event driven behavior and auditing. Events may be replayed to various modules
• Metrics: Each components will export several metrics, accessible via the metrics API. (We may want to promote this to a subsystem.

Client-side components

Some components are implemented on the client side for flexibility:

• Distribution: Functions for pulling and pushing images

Data Flow

As discussed above, the concept of a bundle is central to containerd. Below is a diagram illustrating the data flow for bundle creation.

Let's take pulling an image as a demonstrated example:

1. Instruct the Distribution layer to pull a particular image. The distribution layer places the image content into the content store. The image name and root manifest pointers are registered with the metadata store.
2. Once the image is pulled, the user can instruct the bundle controller to unpack the image into a bundle. Consuming from the content store, layers from the image are unpacked into the snapshot component.
3. When the snapshot for the rootfs of a container is ready, the bundle controller can use the image manifest and config to prepare the execution configuration. Part of this is entering mounts into the execution config from the snapshot module.
4. The prepared bundle is then passed off to the runtime subsystem for execution. It reads the bundle configuration to create a running container.

Data Flow

In the past, container systems have hidden the complexity of pulling container images, hiding many details and complexity. This document intends to shed light on that complexity and detail how a "pull" operation will look from the perspective of a containerd user. We use the bundle as the target object in this workflow, and walk back from there to describe the full process. In this context, we describe both pulling an image and creating a bundle from that image.

With containerd, we redefine the "pull" to comprise the same set of steps encompassed in prior container engines. In this model, an image defines a collection of resources that can be used to create a bundle. There is no specific format or object called an image. The goal of the pull is to produce a set of steps is to resolve the resources that comprise an image, with the separation providing lifecycle points in the process.

A reference implementation of the complete "pull", performed client-side, will be provided as part of containerd, but there may not be a single "pull" API call.

A rough diagram of the dataflow, along with the relevant components, is below.

While the process proceeds left to right in the diagram, this document is written right to left. By working through this process backwards, we can best understand the approach employed by containerd.

Running a Container

For containerd, we'd generally like to retrieve a bundle. This is the runtime, on-disk container layout, which includes the filesystem and configuration required to run the container.

Generically, speaking, we can say we have the following directory:

config.json
rootfs/

The contents of config.json isn't interesting in this context, but for clarity, it may be the runc config or a containerd specific configuration file for setting up a running container. The rootfs is a directory where containerd will setup the runtime container's filesystem.

While containerd doesn't have the concept of an image, we can effectively build this structure from an image, as projected into containerd. Given this, we can say that requirements for running a container are to do the following:

1. Convert the configuration from the container image into the target format for the containerd runtime.
2. Reproduce the root filesystem from the container image. While we could unpack this into rootfs in the bundle, we can also just pass this as a set of mounts to the container configuration.

The above defines the framework in which we will operate. Put differently, we can say that we want to create a bundle by creating these two components of a bundle.

Creating a Bundle

Now that we've defined what is required to run a container, a bundle, we need to create one.

Let's say we have the following:

ctr run ubuntu

This does no pulling of images. It only takes the name and creates a bundle. Broken down into steps, the process looks as follows:

1. Lookup the digest of the image in metadata store.
2. Resolve the manifest in the content store.
3. Resolve the layer snapshots in the snapshot subsystem.
4. Transform the config into the target bundle format.
5. Create a runtime snapshot for the rootfs of the container, including resolution of mounts.
6. Run the container.

From this, we can understand the required resources to pull an image:

1. An entry in the metadata store a name pointing at a particular digest.
2. The manifest must be available in the content store.
3. The result of successively applied layers must be available as a snapshot.

Unpacking Layers

While this process may be pull or run driven, the idea is quite simple. For each layer, apply the result to a snapshot of the previous layer. The result should be stored under the chain id (as defined by OCI) of the resulting application.

Pulling an Image

With all the above defined, pulling an image simply becomes the following:

1. Fetch the manifest for the image, verify and store it.
2. Fetch each layer of the image manifest, verify and store them.
3. Store the manifest digest under the provided name.

Note that we leave off using the name to resolve a particular location. We'll leave that for another doc!

Container Lifecycle

While containerd is a daemon that provides API to manage multiple containers, the containers themselves are not tied to the lifecycle of containerd. Each container has a shim that acts as the direct parent for the container's processes as well as reporting the exit status and holding onto the STDIO of the container. This also allows containerd to crash and restore all functionality to containers.

containerd

The daemon provides an API to manage multiple containers. It can handle locking in process where needed to coordinate tasks between subsystems. While the daemon does fork off the needed processes to run containers, the shim and runc, these are re-parented to the system's init.

shim

Each container has its own shim that acts as the direct parent of the container's processes. The shim is responsible for keeping the IO and/or pty master of the container open, writing the container's exit status for containerd, and reaping the container's processes when they exit. Since the shim owns the container's pty master, it provides an API for resizing.

Overall, a container's lifecycle is not tied to the containerd daemon. The daemon is a management API for multiple container whose lifecycle is tied to one shim per container.

Snapshots

Docker containers, from the beginning, have long been built on a snapshotting methodology known as layers. Layers provide the ability to fork a filesystem, make changes then save the changeset back to a new layer.

Historically, these have been tightly integrated into the Docker daemon as a component called the graphdriver. The graphdriver allows one to run the docker daemon on several different operating systems while still maintaining roughly similar snapshot semantics for committing and distributing changes to images.

The graphdriver is deeply integrated with the import and export of images, including managing layer relationships and container runtime filesystems. The behavior of the graphdriver informs the transport of image formats.

In this document, we propose a more flexible model for managing layers. It focuses on providing an API for the base snapshotting functionality without coupling so tightly to the structure of images and their identification. The minimal API simplifies behavior without sacrificing power. This makes the surface area for driver implementations smaller, ensuring that behavior is more consistent between implementations.

These differ from the concept of the graphdriver in that the Snapshotter has no knowledge of images or containers. Users simply prepare and commit directories. We also avoid the integration between graph drivers and the tar format used to represent the changesets.

The best aspect is that we can get to this model by refactoring the existing graphdrivers, minimizing the need for new code and sprawling tests.

Scope

In the past, the graphdriver component has provided quite a lot of functionality in Docker. This includes serialization, hashing, unpacking, packing, mounting.

The Snapshotter will only provide mount-oriented snapshot access with minimal metadata. Serialization, hashing, unpacking, packing and mounting are not included in this design, opting for common implementations between graphdrivers, rather than specialized ones. This is less of a problem for performance since direct access to changesets is provided in the interface.

Architecture

The Snapshotter provides an API for allocating, snapshotting and mounting abstract, layer-based filesystems. The model works by building up sets of directories with parent-child relationships, known as Snapshots.

A Snapshot represents a filesystem state. Every snapshot has a parent, where the empty parent is represented by the empty string. A diff can be taken between a parent and its snapshot to create a classic layer.

Snapshots are best understood by their lifecycle. Active snapshots are always created with Prepare or View from a Committed snapshot (including the empty snapshot). Committed snapshots are always created with Commit from an Active snapshot. Active snapshots never become committed snapshots and vice versa. All snapshots may be removed.

After mounting an Active snapshot, changes can be made to the snapshot. The act of committing creates a Committed snapshot. The committed snapshot will inherit the parent of the active snapshot. The committed snapshot can then be used as a parent. Active snapshots can never be used as a parent.

The following diagram demonstrates the relationships of snapshots:

In this diagram, you can see that the active snapshot a is created by calling Prepare with the committed snapshot P₀. After modification, a becomes a' and a committed snapshot P₁ is created by calling Commit. a' can be further modified as a'' and a second committed snapshot can be created as P₂ by calling Commit again. Note here that P₂'s parent is P₀ and not P₁.

Operations

The manifestation of snapshots is facilitated by the Mount object and user-defined directories used for opaque data storage. When creating a new active snapshot, the caller provides an identifier called the key. This operation returns a list of mounts that, if mounted, will have the fully prepared snapshot at the mounted path. We call this the prepare operation.

Once a snapshot is prepared and mounted, the caller may write new data to the snapshot. Depending on the application, a user may want to capture these changes or not.

For a read-only view of a snapshot, the view operation can be used. Like prepare, view will return a list of mounts that, if mounted, will have the fully prepared snapshot at the mounted path.

If the user wants to keep the changes, the commit operation is employed. The commit operation takes the key identifier, which represents an active snapshot, and a name identifier. A successful result will create a committed snapshot that can be used as the parent of new active snapshots when referenced by the name.

If the user wants to discard the changes in an active snapshot, the remove operation will release any resources associated with the snapshot. The mounts provided by prepare or view should be unmounted before calling this method.

If the user wants to discard committed snapshots, the remove operation can also be used, but any children must be removed before proceeding.

For detailed usage information, see the GoDoc.

Graph metadata

As snapshots are imported into the container system, a "graph" of snapshots and their parents will form. Queries over this graph must be a supported operation.

How snapshots work

To flesh out the Snapshots terminology, we are going to demonstrate the use of the Snapshotter from the perspective of importing layers. We'll use a Go API to represent the process.

Importing a Layer

To import a layer, we simply have the Snapshotter provide a list of mounts to be applied such that our destination will capture a changeset. We start out by getting a path to the layer tar file and creating a temp location to unpack it to:

layerPath, tmpDir := getLayerPath(), mkTmpDir() // just a path to layer tar file.

We start by using a Snapshotter to Prepare a new snapshot transaction, using a key and descending from the empty parent "":

mounts, err := snapshotter.Prepare(key, "")
if err != nil { ... }

We get back a list of mounts from Snapshotter.Prepare, with the key identifying the active snapshot. Mount this to the temporary location with the following:

if err := mount.All(mounts, tmpDir); err != nil { ... }

Once the mounts are performed, our temporary location is ready to capture a diff. In practice, this works similar to a filesystem transaction. The next step is to unpack the layer. We have a special function unpackLayer that applies the contents of the layer to target location and calculates the DiffID of the unpacked layer (this is a requirement for docker implementation):

layer, err := os.Open(layerPath)
if err != nil { ... }
digest, err := unpackLayer(tmpLocation, layer) // unpack into layer location
if err != nil { ... }

When the above completes, we should have a filesystem the represents the contents of the layer. Careful implementations should verify that digest matches the expected DiffID. When completed, we unmount the mounts:

unmount(mounts) // optional, for now

Now that we've verified and unpacked our layer, we commit the active snapshot to a name. For this example, we are just going to use the layer digest, but in practice, this will probably be the ChainID:

if err := snapshotter.Commit(digest.String(), key); err != nil { ... }

Now, we have a layer in the Snapshotter that can be accessed with the digest provided during commit. Once you have committed the snapshot, the active snapshot can be removed with the following:

snapshotter.Remove(key)

Importing the Next Layer

Making a layer depend on the above is identical to the process described above except that the parent is provided as parent when calling Snapshotter.Prepare, assuming a clean tmpLocation:

mounts, err := snapshotter.Prepare(tmpLocation, parentDigest)

We then mount, apply and commit, as we did above. The new snapshot will be based on the content of the previous one.

Running a Container

To run a container, we simply provide Snapshotter.Prepare the committed image snapshot as the parent. After mounting, the prepared path can be used directly as the container's filesystem:

mounts, err := snapshotter.Prepare(containerKey, imageRootFSChainID)

The returned mounts can then be passed directly to the container runtime. If one would like to create a new image from the filesystem, Snapshotter.Commit is called:

if err := snapshotter.Commit(newImageSnapshot, containerKey); err != nil { ... }

Alternatively, for most container runs, Snapshotter.Remove will be called to signal the Snapshotter to abandon the changes.

Mounts

Mounts are the main interaction mechanism in containerd. Container systems of the past typically end up having several disparate components independently perform mounts, resulting in complex lifecycle management and buggy behavior when coordinating large mount stacks.

In containerd, we intend to keep mount syscalls isolated to the container runtime component, opting to have various components produce a serialized representation of the mount. This ensures that the mounts are performed as a unit and unmounted as a unit.

From an architecture perspective, components produce mounts and runtime executors consume them.

More imaginative use cases include the ability to virtualize a series of mounts from various components without ever having to create a runtime. This will aid in testing and implementation of satellite components.

Structure

The Mount type follows the structure of the historic mount syscall:

Field	Type	Description
Type	string	Specific type of the mount, typically operating system specific
Target	string	Intended filesystem path for the mount destination.
Source	string	The object which originates the mount, typically a device or another filesystem path.
Options	[]string	Zero or more options to apply with the mount, possibly =-separated key value pairs.

We may want to further parameterize this to support mounts with various helpers, such as mount.fuse, but this is out of scope, for now.