Advancing AI model interoperability with Docker and ModelPack

The rise in the number of tools available for creating and running AI content has lowered the barrier of entry and offered the flexibility of choice when choosing the appropriate solution for a given use case. However, one such challenge that has emerged, especially in the AI model management space, is that there is often a tight coupling between the tool and how it manages models – potentially limiting the flexibility to move between frameworks or distribute the content more broadly. These concerns are one of the primary areas of focus for the ModelPack project. The Challenge of AI Model Management For a single user working solely on their local machine, the importance of having to think about how to manage models may not be top of mind. However, as we have seen previously with containers, individuals are typically not isolated to just their content. They will either want or need to leverage assets that have been produced by others or be able to distribute their work so that it does not solely reside on their local machine. How this is facilitated depends primarily on the tool or framework that is being used. Some of the common options include: Packaging | Name | Description | | Compressed Archive | Single artifact containing all model related assets | | Container Image | Assembling all model related assets within a standard container image | | Wrapper | Content assembled using a set of proprietary metadata/structures | Storage | Name | Description | | Object Storage | On-premise or cloud object based solutions | | Git LFS | Leveraging the extended support for large files within the Git Version Control System | | Custom Model Registries | Implementation-specific storage of model related assets | Ultimately, the decision to support one tool or framework over another is typically constrained based on the formats supported by the Models themselves and the backends for which they are stored. Standardizing an AI Model Format Addressing the limitations that have become inherent to the majority of model management and runtime tools has been the primary goal of the ModelPack project. By developing an open standard that defines how AI models should be constructed, and by leveraging OCI artifacts as the framework for which models are assembled and distributed, the overall burden for developing and running AI models is reduced. OCI artifacts have been gaining popularity in the AI space and an increasing number of components are leveraging the burgeoning ecosystem of solutions. ModelPack is just another project that is leveraging the fundamental benefits of OCI. Learn more about the ModelPack technical specification here. Partnering with Docker to expand Model Interoperability The ModelPack project includes maintainers and contributors from several organizations including Nutanix, Ant, Jozu, and Red Hat. Several members from Docker approached the ModelPack community regarding one of the tools they had developed, the Docker Model Runner (DMR), as it was also leveraging OCI artifacts as a packaging and distribution method. In the case of DMR, models are packaged using the OCI media type application/vnd.docker.ai.model.config.v0.1+json along with a series of additional Media Types representing the content packaged (as OCI layers) within the artifacts. ModelPack alternatively uses application/vnd.cncf.model.manifest.v1+json. The following table provides a comparison of the most common Media Types between the two formats: | Resource | ModelPack | Docker Model Format | | Config descriptor | application/vnd.cncf.model.config.v1+json | application/vnd.docker.ai.model.config.v0.1+json | | Artifact Type | application/vnd.cncf.model.manifest.v1+json | N/A | | License | application/vnd.cncf.model.doc.v1.tar (Can also include other documentation, such as README.md files. Support is available for compressed archives through one of the supported suffix types) | application/vnd.docker.ai.license | You can read more about the Docker Model format as well as why Docker chose OCI artifacts as its packaging and distribution method in this blog post. While the use of OCI artifacts works great for tooling that understands Docker Model Runner formatted content, it potentially restricts where and how the packaged models can be used. Fortunately, since Docker Models still comply with the guidelines set forth for OCI artifacts, OCI registries, like Docker Hub and Quay, can still be used regardless of artifact type. To overcome these limitations and expand how DMR content could be used, Docker associates began to work together with members of the ModelPack community to see how both groups could benefit by enhancing the capabilities of both projects. Aside from the community collaboration, updates were made within both projects that would make each easier to produce and consume AI models from an end user point of view. For ModelPack users, the specification was hardened, ensuring that it is a viable solution for anyone looking to standardize how they package AI models. On the Docker Model Runner side, users now have the ability to format the models they produce in ModelPack format, realizing the goal of true model interoperability. By specifying the –format=cncf option as part of the docker model package command, the produced model will be constructed as a ModelPack formatted OCI artifact. docker model package –format=cncf Once the model has been produced, the resulting artifact can be published to an OCI registry for consumption by individuals and compatible serving frameworks. Realizing the Value of Collaboration The partnership between Docker and ModelPack is yet another example of how community collaboration using Open Source principles yields benefits. By sharing the types of challenges that Docker is seeing and hearing in the wild from their customers and users, the ModelPack project has refined and hardened the ModelPack specification and associated tooling. And, thanks to Docker’s leadership within the tech industry, the inclusion of ModelPack as a supported format raises visibility of the project and offers an onramp for future contributors to grow the project even further. - Learn more about the ModelPack project - Join in on the conversation in the #modelpack channel in CNCF Slack - Learn more about Docker Model Runner and how to get started