Public Cloud Toolchains in SUSE Linux Enterprise 16: Evolution and Transparent Containers

Public Cloud Toolchains in SUSE Linux Enterprise 16: Evolution and Transparent Containers The release of the SUSE Linux Enterprise (SLE) 16 distributions has long come and gone and the development cycle for SLE 16.1 is well on the way and will culminate in the SLE 16.1 release later this year. As such, and as part of the “better late than never” story of time, I am finally getting around to sharing information around the changes that happened to the Public Cloud Toolchains for SLE 16 and that will carry forward in the SLE 16 minor releases. Background: Why Change? Public Cloud Services move fast. While each individual service evolves at a measured pace, the sheer volume of services and their steady expansion force associated toolchains to move quickly—at least for an enterprise distribution. Whether it is new instance types, regions, services, or network construct features, change at Cloud Service Providers (CSPs) is constant. These changes are directly reflected in the toolchains, Software Development Kits (SDKs), and Command Line Interfaces (CLIs) that ship with the Python versions in SLE. Fast-moving codebases and enterprise distributions built around a “slow and steady” philosophy have historically had a contentious relationship. In SLE 12 and SLE 15, we attempted to address this tension using Modules. In principle, the module concept allowed fast-moving codestreams (receiving regular version updates) to coexist alongside a slow-moving core system. However, as with most things in software development, the module concept introduced significant tradeoffs: - Repository Confusion: Modules introduced numerous repositories. It was difficult for users to track which package belonged to which module, and the criteria for what went where was often ambiguous or arbitrary. - Registration Overhead: Upon system registration, every module repository had to be configured to provide access to all available RPM packages. For example, in a SUSE Linux Enterprise Server (SLES) 15 SP7 Pay-As-You-Go (PAYG) instance, registration took roughly 26 seconds and configured 55 repositories. By contrast, a SLES 16 PAYG instance takes only 6 seconds to register and configures just 5 repositories. Dropping the module concept yielded two major benefits: - It eliminated confusion over “what comes from where.” - It significantly improved system registration performance. The Fast-Moving Codestream Dilemma Moving away from modules implies an “everything together” approach, which naturally raises a critical question: What about fast-moving codestreams? Some form of isolation is still required. The theory of isolating fast-moving codestreams inside modules didn’t quite work as intended in SLE 12 and SLE 15. While CSP toolchains were “isolated” in the Public Cloud Module, they still depended on packages in the Basesystem, which were not supposed to change rapidly or receive frequent version updates. Furthermore, because many other system components shared those base dependencies, updating package versions was precarious. Updating the Azure toolchain, for example, could take a full year—from initiating package updates in openSUSE Factory, to propagating them into SLE 15, to eventual release. No matter how diligently we tested, dependency package updates occasionally broke unrelated components. Additionally, the Module concept didn’t allow us to ship newer package versions without affecting the broader system. Updating any CSP toolchain under SLE 12 and SLE 15 remained a high-risk operational process. Our primary goal for SLE 16 was to enable frequent, regular releases of CSP toolchains on a ~3-month cycle to keep pace with CSP feature development. To achieve this isolation without relying on modules, we revisited the toolchain design and introduced Transparent Containers. What Are Transparent Containers? Containers provide isolated environments from the surrounding operating system, allowing toolchains to move faster than core system components. However, standard container isolation comes at the cost of duplication. A “fat” container carries its own libraries (such as libssl or glibc), even if those libraries already exist on the host system. If a vulnerability arises in one of those libraries, administrators must update the host, rebuild the container, and redeploy it. This turns a single-step patch into a three-step process: - Update the host system running the container(s). - Rebuild the container. - Redeploy the container. The Transparent Container Concept For CSP toolchains, we didn’t want users to adopt entirely new workflows just to receive basic updates. While a standard container wasn’t the right fit, containerization still provided the ideal conceptual boundary for isolation. What we needed was a container capable of using updated libraries directly from the host system—hence the “Transparent Container” architecture. Core Design Goals Before exploring how this works under the hood, here are the core baseline goals we established: - Unchanged Update Workflows: Standard tools like zypper up or zypper patch must continue delivering the latest toolchain updates. - Automatic System Library Inheriting: When libssl is updated on the host, the next run of a CSP toolchain uses that updated system library automatically. - Simplified Dependency Overlaying: Dependency testing must not block or delay new toolchain releases. - Faster Release Cycles: Target a ~3-month release cycle for CSP toolchains. How It Works Under the Hood Because containers are structured in layers, the path forward conceptually was straightfoward: - Build the SDK into one layer. - Build the CLI on top of the SDK in a second layer. - Create a layer for runtime host dependencies. This structure delivers isolation without duplicating core libraries. If a system-level dependency requires a version bump that the host doesn’t yet provide, we can temporarily layer that updated package directly into the SDK or CLI layer until the base OS catches up. Layer Building with KIWI At SUSE, KIWI is our tool of choice for building CSP-related images. To support transparent containers, KIWI was enhanced to build new layers on top of multiple existing layers and pass metadata about required runtime host files. Building the CLI toolchain requires three distinct layers: - Runtime Provider Container: Represents host-provided system dependencies. - SDK Container (Layer): Builds on the runtime provider. - CLI Container (Layer): Builds on the SDK layer. A Closer Look at the Image Setup First, we build a container representative of the host system. Using AWS as an example, this is defined via the KIWI description for aws-toolchain-runtime-provider (similar configurations exist for az and google). This description lists the host packages expected at runtime when the CLI or SDK containers start. These runtime provider containers are published on the SUSE Registry: - [registry.suse.com/suse/public-cloud-toolchain/az-toolchain-runtime-provider](https://registry.suse.com/suse/public-cloud-toolchain/az-toolchain-runtime-provider) - [registry.suse.com/suse/public-cloud-toolchain/aws-toolchain-runtime-provider](https://registry.suse.com/suse/public-cloud-toolchain/aws-toolchain-runtime-provider) - [registry.suse.com/suse/public-cloud-toolchain/google-toolchain-runtime-provider](https://registry.suse.com/suse/public-cloud-toolchain/google-toolchain-runtime-provider) Technical Detail: Inside the KIWI description, the provide_system_files=”true” attribute tells KIWI to embed metadata into the container. When the next layer is built with require_system_files=”true”, this metadata informs the system which files to map dynamically at runtime. To avoid tight coupling (for example, breaking compatibility if running on openSUSE Leap instead of SLES), KIWI uses an ignore element to exclude build-only packages from the required runtime file metadata. The SDK Layer The next layer up is the SDK container (e.g., aws-sdk-container). These images are delivered directly as RPM packages (aws-sdk-image, az-sdk-image, google-sdk-image). When installed, the containers end up in your local system registry. Because they lack a base command shell or Python interpreter, running them directly via standard podman run commands will not produce an interactive shell. The CLI Layer The top layer is the CLI container (e.g., aws-cli-image, az-cli-image), derived from both the runtime and SDK layers using KIWI’s delta_root=”true” and derived_from=”…” attributes. Runtime Execution with flake-pilot To tie these disconnected layers together at execution time, we use flake-pilot (which supports Podman and Firecracker backends). Installing the command RPM packages (aws-cli-cmd or az-cli-cmd) registers a “flake” with flake-pilot. This gives transparent containers two operational dimensions: - Backend Transparency: The container utilizes dependencies directly from the host system. - Frontend Transparency: The aws or az commands behave like standard binaries installed locally on the OS, sparing the user from constructing lengthy podman commands. Managing Startup Overhead and Container Persistence When aws or az is executed, flake-pilot dynamically copies the necessary system bits into a runtime layer that Podman uses to launch the container. Because copying files takes time, there is a small startup delay on initial execution. To eliminate this delay for subsequent calls, you can enable container persistence: - Open the generated configuration file in /usr/share/flakes/ (e.g., aws.yml or az.yml). - Under the runtime: section, set resume: true: YAML runtime: resume: true Why isn’t resume: true the default? Defaulting to resume: false ensures that every invocation reflects the latest patches applied to the host system. If resume is set to true, a running container will continue using the older in-memory versions of system libraries until the container is restarted or the system is rebooted. - Maintenance Windows: If your environment updates on a fixed schedule (e.g., quarterly maintenance with reboots), setting resume: true is recommended. - Frequent Patching: If you patch continuously without regular reboots, keep resume: false or manually stop the container to apply updates using: Bash aws help %remove (Note: The % prefix distinguishes execution engine directives from CLI flags. Re-executing the command afterward rebuilds the runtime layer using updated system libraries). Building Custom Tools The frontend and backend transparency layers operate independently. Building Traditional Containers on SDK Images If you prefer a self-contained (“fat”) image incorporating glibc, openssl, and Python, combine the runtime and SDK layers at build time without the delta_root attribute: KIWI: XML derived_from=”registry.suse.com/suse/public-cloud-toolchain/$CSP-toolchain-runtime-provider#latest,registry.suse.com/suse/public-cloud-toolchain/$CSP-sdk#latest” Dockerfile: Dockerfile FROM registry.suse.com/suse/public-cloud-toolchain/$CSP-toolchain-runtime-provider:latest FROM registry.suse.com/suse/public-cloud-toolchain/$CSP-sdk:latest Building Fully Transparent Custom Tools To create your own host-transparent containers, use KIWI with the delta_root, derived_from, and require_system_files attributes. Frontend transparency can then be applied to any container using flake-ctl: Bash flake-ctl podman register --app /usr/bin/my-tool ... Running an Interactive SDK Environment If you need an interactive Python environment to test or experiment with SDKs directly, configure a custom flake: Bash flake-ctl podman register \ --app /usr/bin/aws-sdk-interactive \ --target /usr/bin/python3 \ --container localhost/aws-sdk \ --check-host-dependencies \ --opt "--interactive" \ --opt "--security-opt label=disable" \ --opt "--volume %HOME:/root" \ --opt "--volume /var/lib/ca-certificates:/var/lib/ca-certificates" \ --opt "-e HOME=/root" \ --resume Executing aws-sdk-interactive launches a Python shell where commands like import botocore work directly against the transparent SDK environment. Conclusion The Public Cloud Toolchain architecture in SUSE Linux Enterprise 16 addresses long-standing maintenance bottlenecks through two manageable tradeoffs: - Initial Startup Latency: Can be eliminated via the resume: true configuration depending on your system maintenance cadence. - SDK Workflow Shifts: Developer workflows for custom tools shift toward container-native patterns—a model already widely adopted across the industry. By moving to Transparent Containers, SLE 16 provides significantly faster registration times, drastically fewer repository requirements, and rapid 3-month CSP toolchain updates without risking host stability. Related Articles Apr 18th, 2025