Read on Twitter
"Good thread here. You need data + eyes" is the boring answer. The interesting discussion is HOW you integrate them. IMO, every team should have a rigorous answer to doing that in a unified process, and there are 3 key places to do it. Quick thread: https://twitter.com/EvolvingWild/status/1352531036387815429
1. Include the eye test in model design. Make WAR model v1, then check which results look off. The key is to look for patterns, not just laugh at outliers: are the players that look low super physical? That informs how to improve v2
Maybe physical play is overrated by the eye test, or maybe the model isn't recognizing the genuine value those players provide. That's a question that should be studied to inform whether changes are needed in v2.
2. Turn the eye test into data, then put it in the model. As always, my favorite hockey paper: http://statsportsconsulting.com/main/wp-content/uploads/TextMiningScoutingNHLDraftAnalyticsFeb2017.pdf. It finds that taking scouting reports and building model features out of the text improves a draft model vs. one with stats alone. Every team should do this.
3. Post-hoc adjustments. People will always take a "final" ranking and move players around. That's okay. The key is to develop a formal process for making those adjustments & reviewing afterwards whether they help, rather than an ad hoc, personality-driven debate for every player
Note that this system *still produces a single, rigorous, ranked output*. That's a "model". It's a broader definition of a model than "the immediate output of a stats-driven ML algorithm", but it's the kind of model you should use to make decisions. Otherwise you're just guessing
