How to apply DevOps to Machine learning

Thanks for continuing to read my series on applying DevOps to Machine Learning. If you’re new here you might want to start at the begining or check out some technical content. Both should halp make this all easier to digest.

So let’s remind ourselves of what we are trying to achieve. We are looking to apply DevOps to Machine Learning, to remove the need to have a data engineer. We want to use automation where we can, CI/CD and a load of tools all working in harmony to achieve this. We have focused on a lot of the theory for the last few blogs, and this is no exception. The next blog is the detailed how to. So keep reading.

 

Based on the my research and a review of existing books and blogs on this topic, I will implementing the following DevOps techniques to reduce the difficulty of promoting a model in to production.

  • Source control
  • Infrastructure as code / Configuration as code
  • Continuous integration
  • Continuous deployment
  • Microservices & Containers
  • Monitoring and lineage

I have limited the scope to services which are in Azure, however as Azure supports IAAS servers, I could install any Apache open-source tool. The design principles restrict this as one indicates the need for PAAS over IAAS. As Azure is vast, even when working under restrictions, there are still multiple options which could work. The following sections indicates the tools I will use and the decisions I made during the selection process.

Continue reading