Causes of fragmentation for Machine learning processes and why many practices are still bespoke.

Every time I design a new microservice, there is a TODO checklist that I check against. An early item in this list is about setting up CI/CD for the new service. I think about what the artifacts are and how they get formed at various stages. I consider how it eventually gets deployed to production and how it gets monitored post-deployment.

This situation is not unique to me. Modern software engineering practices implement well-thought CI/CD principles as a norm. …

Key AWS services shaping Machine Learning landscape in 2021

Photo by Christian Wiediger on Unsplash

It’s not news that AWS has the broadest set of Machine Learning capabilities among the big three cloud providers. With the public availability of Sagemaker Studio in early 2020, they created a fully integrated ML Dev environment – a first-of-it’s-kind.

Anchoring an IDE at the center of all ML offerings is a smart move — provided your surrounding services fill the key ops gaps correctly. If done well, Amazon has the opportunity to once-and-for-all reshape the way Machine Learning is done in the industry.

Even before Sagemaker Studio, AWS had a handful of services geared towards MLOps. However, Re:invent 2020

Bad practices and other symptoms in your Machine Learning process that indicates a deeper problem

Code smells are a common way of identifying hidden problems in your code using a surface level inspection. Here are some practices that show similar smells in your Machine learning process. They increase the probability of hidden errors in your ML practice over the long term.

You are directly manipulating model artifacts.

After training is complete, data scientists sometimes send models into standard locations such as S3 bucket. This is then picked up by the production inference framework. Or even picked up manually by another team for further processing.

This is a single example of how models artifacts get manipulated by-hand in a data science company…

Vishnu Prathish

Engineering Management @ Innovyze AI. We use AI to solve water problems.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store