Read on Twitter
1/ Our latest Production ML meetup features Erik Reppel ( @programmer -- that handle tho 
), a Sr ML Platform Engineer at Coinbase. We covered building reproducible / performant ML pipelines, data management at scale, and advanced web serving of ML models, and more 
), a Sr ML Platform Engineer at Coinbase. We covered building reproducible / performant ML pipelines, data management at scale, and advanced web serving of ML models, and more
2/ Security is top priority for Coinbase's ML Platform team. Most existing ML platforms fail security tests, so Coinbase's platform is built internally, leveraging AWS SageMaker where possible. Challenges include imperfect data, w strict & real-time SLAs
3/ 3-phase production lifecycle: 1. Prototype: build datasets, construct models 2. Productionize: minimize time-to-production, satisfy Coinbase's scaling requirements 3. Experiment: observe treatment and control performance, correlate model metrics with business metrics
4/ Strict SLA req's shape model choices: 1. While prototyping, engineers are cognizant of model inference performance & use techniques like pruning & distillation when necessary 2. Must monitor key metrics (eg AUC), which can disrupt business metrics
5/ Engineering infrastructure is tailored for model/framework flexibility, ie moving from an XGBoost model to a CatBoost model for lift in core metrics); quickly incorporating research into production systems
6/ Their stack includes 1. feature store w hybrid historical/aggregated stream feature vectors (Flink + DynamoDB) 2. containerized platform based on SageMaker 3. inference engine with proprietary & SageMaker endpoints. Must understand tradeoffs and downstream limitations of tools
7/ Model monitoring is currently more art than science - fraud is an evolving problem!
8/ Model code is separate from infra code. ML Engineers review model code; platform engineers review backend code. Recommended code review practices: 1. Sub-sample datasets to unittest model code (saves debugging time) 2. Set random state consistency to ensure reproducibility
9/ Coinbase' team experiments heavily with different interpretability methods like SHAP, feature importance. SHAP values are used by analysts to understand patterns in what the models are discovering
10/ The best MLEs have T-shaped skillsets: a variety of SW/DS skills + an ML specialization. It's about building better solutions, reasoning w computers, writing clean code. The smaller the company/more general the problem, the more eng skills you should develop
11/ Thoughts on the hierarchy of ML: good data â good monitoring and metrics â good models â good model inference and integration. Based on that, people in his team are assigned projects according to their skillset. Hire data platform engineers first.
