The rise in usage of cloud computing resources and container management platforms for executing AI (Artificial Intelligence) and ML (Machine Learning) workloads has led many engineers and companies to question the suitability and effectiveness of Kubernetes' resource management and scheduling to meet the growing requirements of these workloads.

So why’s that? What patterns, architectures, and procedures has led these companies and engineers to this problem of scaling ML platforms on Kubernetes? And what kind of solution could we apply to help solve those problems?

I’ve often wondered what it would be like if all software applications (databases, APIs, UI clients, pubsub, secret managers, etc) ran on the same infrastructure, so I’d never have to worry about working with different infrastructure patterns e.g. CDNs, Kubernetes, VMs, Functions, the one off PaaS or BaaS everyone forgot about, load balanced, multi-region, auto-scaling groups, etc!

I wanted; a familiar frontend, a lightweight and unopinionated API layer that plays nice with top data science software and can also be made available as a standalone service, a data layer with a natively distributed design, high fidelity pubsub (ideally with some delivery guarantee), and a secrets manager – all to run together on a kubernetes cluster.