Read on Twitter
New ML Engineered episode with Luigi Patruno from @mlinproduction!
Why and how to monitor your ML applications Where ML actually drives biz value The future of software and skills needed to take advantagehttps://www.mlengineered.com/episode/luigi-patruno
Thread with best quotes
Monitoring pt 1
"A lot of teams think about monitoring too late, [it] only becomes a concern after there's been an issue, either a customer has complained, or there's been some unplanned downtime"
"A lot of teams think about monitoring too late, [it] only becomes a concern after there's been an issue, either a customer has complained, or there's been some unplanned downtime"
Monitoring pt 2
"What are the the metrics we want to understand on a daily basis and where in the application would those need to be emitted from?"
"There's no one size fits all. It really depends on your specific application and models"
"What are the the metrics we want to understand on a daily basis and where in the application would those need to be emitted from?"
"There's no one size fits all. It really depends on your specific application and models"
Monitoring pt 3
"Observability: How can we understand the context of the application... so we can debug issues even when you don't have specific metrics planned beforehand?"
"Observability: How can we understand the context of the application... so we can debug issues even when you don't have specific metrics planned beforehand?"
Monitoring pt 4
"Logging is something most data scientists don't have much experience with, but is very important."
"Think through issues that might occur, what sort of context you would want to see in the log messages [while debugging]."
"Logging is something most data scientists don't have much experience with, but is very important."
"Think through issues that might occur, what sort of context you would want to see in the log messages [while debugging]."
Monitoring pt 5
"There are a lot of tools emerging in the [model] monitoring space that... encapsulate best practices for exactly what metrics to measure.
I definitely recommend using those instead of trying to develop these things yourself."
"There are a lot of tools emerging in the [model] monitoring space that... encapsulate best practices for exactly what metrics to measure.
I definitely recommend using those instead of trying to develop these things yourself."
ML biz value pt 1
"When I was an IC, I really focused on the technical research, the latest and greatest."
"But when I went into management, I started thinking in terms of the time and money spent on this and the payoff I'm getting [from it]."
"When I was an IC, I really focused on the technical research, the latest and greatest."
"But when I went into management, I started thinking in terms of the time and money spent on this and the payoff I'm getting [from it]."
ML biz value pt 2
"If you take that perspective in your ML work, you realize that there are a small set of algorithms that work pretty well and are very well understood"
"Other algos might be more exciting, but at the end of the day, considered fluff from a biz perspective"
"If you take that perspective in your ML work, you realize that there are a small set of algorithms that work pretty well and are very well understood"
"Other algos might be more exciting, but at the end of the day, considered fluff from a biz perspective"
ML biz value pt 3
"At the end of the day, the ML [org] is just another part of the company.
And what the company is trying to do is hit its business KPIs and metrics.
If the ML enterprise isn't contributing to those, it's a net negative"
"At the end of the day, the ML [org] is just another part of the company.
And what the company is trying to do is hit its business KPIs and metrics.
If the ML enterprise isn't contributing to those, it's a net negative"
Adopting ML in a biz pt 1
"There's this famous diagram, the AI hierarchy of needs
It's tempting to go straight to the top, but if you don't have one of the layers below, you're not going to get the outcomes you want...
You'll eventually need to go back down"
"There's this famous diagram, the AI hierarchy of needs
It's tempting to go straight to the top, but if you don't have one of the layers below, you're not going to get the outcomes you want...
You'll eventually need to go back down"
Adopting ML in a biz pt 2
"It's hard to do that convincing when you have an older co that's gotten to where it is without the use of complicated analytics
Whereas if a co is brand new and has a data scientist on the founding team, they'll be thinking about this from the get-go"
"It's hard to do that convincing when you have an older co that's gotten to where it is without the use of complicated analytics
Whereas if a co is brand new and has a data scientist on the founding team, they'll be thinking about this from the get-go"
Adopting ML in a biz pt 3
"Focus on small problems where you can get quick wins and then evangelize them to different teams"
"Tie projects to the profit and loss. If you can demo that ML adds revenue or reduces costs, that's how you get buy-in from senior people"
"Focus on small problems where you can get quick wins and then evangelize them to different teams"
"Tie projects to the profit and loss. If you can demo that ML adds revenue or reduces costs, that's how you get buy-in from senior people"
Code 2.0 pt 1
"Software engineers will "bootstrap" the system by creating a rules-based system and then instrument it with the right sensors so that we are able to store and retrieve the data generated"
"Software engineers will "bootstrap" the system by creating a rules-based system and then instrument it with the right sensors so that we are able to store and retrieve the data generated"
Code 2.0 pt 2
"Then ML people will go in and replace the rules (software 1.0) with learning-based solutions (2.0).
We have metrics and KPIs for the business and then we'll just have the machine optimize for those directly."
"Then ML people will go in and replace the rules (software 1.0) with learning-based solutions (2.0).
We have metrics and KPIs for the business and then we'll just have the machine optimize for those directly."
Code 2.0 pt 3
"These things are just going to do optimizations, and we want to spend more of our time thinking about whether we're optimizing for the right things, rather than trying to understand exactly what the algorithm is doing for a particular output"
"These things are just going to do optimizations, and we want to spend more of our time thinking about whether we're optimizing for the right things, rather than trying to understand exactly what the algorithm is doing for a particular output"
Everyone will benefit from knowing ML
"UI/UX has a huge impact on the types of ML systems you can build. So designers have plenty of ability to get in to the field"
"Clearly DevOps has a lot to contribute"
"And of course traditional software engineers, PMs, business folks"
"UI/UX has a huge impact on the types of ML systems you can build. So designers have plenty of ability to get in to the field"
"Clearly DevOps has a lot to contribute"
"And of course traditional software engineers, PMs, business folks"
What skills to focus on pt 1
"Do you really want to be competing with the top PhD researchers at Google in terms of understanding the algorithms?
I'd rather develop skills that are not traditionally combined with ML, inter-disciplinary skills"
"Do you really want to be competing with the top PhD researchers at Google in terms of understanding the algorithms?
I'd rather develop skills that are not traditionally combined with ML, inter-disciplinary skills"
What skills to focus on pt 2
"Self-reflection is the most important thing: do you enjoy what you're doing?"
"I have this concept of excellence. Whatever I do, I want to be excellent at it. Work on the things you enjoy doing and do them very well"
"Self-reflection is the most important thing: do you enjoy what you're doing?"
"I have this concept of excellence. Whatever I do, I want to be excellent at it. Work on the things you enjoy doing and do them very well"
Rapid fire Qs pt 1
- For fun: snowboarding
and MMA 
- Books: The War of Art, Sapiens, How to Change Your Mind
- Under-rated ML use-case: "Small- to medium-sized tabular data sets is where all the money's at"
- For fun: snowboarding
and MMA - Books: The War of Art, Sapiens, How to Change Your Mind
- Under-rated ML use-case: "Small- to medium-sized tabular data sets is where all the money's at"
Rapid fire Qs pt 2
- Advice: Don't take yourself too seriously, focus on fundamentals and on learning what the biz cares about
- Contrarian take: View your life as an exploration vs exploitation problem
- Advice: Don't take yourself too seriously, focus on fundamentals and on learning what the biz cares about
- Contrarian take: View your life as an exploration vs exploitation problem
If you liked this thread, check out the full episode for a ton more insights!
Link at the top
On your way up, spread the word with
s and a 
Link at the top
On your way up, spread the word with
s and a
