Read on Twitter
How to evaluate a cybersecurity vendor's ML claims even if you don't know much about ML (thread).
1) Ask them why they didn't solely rely on rules/signatures in their system -- why is ML necessary? If they don't have a clear explanation, deduct a point.
1) Ask them why they didn't solely rely on rules/signatures in their system -- why is ML necessary? If they don't have a clear explanation, deduct a point.
2) Ask them how they know their ML system is good. Where does their test data come from? How do they know their test data is anything like real life data? How do they monitor system performance in the field? If their story isn't convincing, deduct a point.
3) Ask them where on Wikipedia you can read more about the approach they took. If you can't read about it on Wikipedia, ask them where their paper is in the peer-review and on arXiv. If the paper doesn't exist / is a "trade secret", deduct 3 points
4) Ask them why they didn't take a simpler approach than the approach they took. If they can't explain, deduct a point. If they say they tried other, simpler approaches, but can't show comparison data, deduct a point.
5) If they're now down a few points, ask them if they'd like to fess up and say they're really mostly just using rules and claiming to use ML to satisfy investors and industry analysts. If they fess up and then describe a solid rules-based approach, give 'em a point or two back.
6) If they claim "zero false positives!" or "zero false negatives!" don't deduct any points, just end the Zoom call.
7) Ask for examples of attacks the system missed and explanations as to why. Deduct points if they can't think of any misses or can't easily produce a story about their system's known weaknesses.
8) Now ask for a live demo where you control the inputs and test the system on both malicious and benign data. Give them some points depending on how their system performs on a test that you yourself have designed.
You don't have to know much about ML to use something like the system I'm pitching here. Just don't get intimidated by obfuscation attempts from the vendor (common tactic) and keep pressing for clear and convincing answers.
Also, what do you do when you have your final point count? Depends on lots of other factors, including the other tech that the vendor is selling with the ML system, and variables like price and the overall value of the technology in your ecosystem.
