Metaflow

Netflix built Metaflow, then released it as open source. Metaflow is a Python framework for building, running and deploying machine learning, AI and data science workflows on cloud infrastructure.

You write workflows in plain Python, so data scientists and ML engineers can develop, debug and deploy without learning a separate pipeline language.

Metaflow uses cloud resources for scalable compute and data access, which means a workflow built on a laptop can later run on larger infrastructure.

Deployment is supported on AWS, Azure, Google Cloud and Kubernetes.

Metaflow also works with the infrastructure, security and data governance policies a company already has instead of asking you to replace them, which matters most where those rules are fixed.

This is an ML workflow framework for the engineering side of the job, not a point-and-click automation product.

It makes sense for teams that already write Python and want one consistent way to structure projects from experiment to production.

Because the code is open source, you can read it and adapt it to your own setup.