[Discussion] End2end MLOps platform with notebook vs. Testable python modules

[penghanyuan]

Hi thanks for these impressive courses, They really help me a lot in my career.
I have some thoughts that I want to discuss. As there are and more more end2end MLOps platforms that use notebooks to deliver models to production, what is your opinion about converting notebooks to fully testable python modules (in 2023)? Is that still bring some benefits if the platform could ensure the reproducibility for training/data processing...?

Thanks in advance for your reply.

Hanyuan

[GokuMohandas]

Hi @penghanyuan, I'm glad you're finding the content useful (a LOT more to come this July)! This is a great question and it's true that many platform are allowing for production ML directly from notebooks. This is mainly because data scientist / MLEs enjoy developing in notebooks and these platforms are trying to reduce the friction between development and deployment. However, my personal views are that notebooks are great for exploration and proof-of-concepts. And while you could do things like testing within notebooks, the clarity and modularity (ex. packaging) you get from Python scripts / projects is unparalleled. And when putting things into production, you definitely want these capabilities. But there are very successful teams that have some very high stakes models in production so 🤷‍♂️. I would say the most important thing is standardization.