Read on Twitter
This week I was given a lesson in Hyrum's Law:
With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.
https://www.hyrumslaw.com/
<
>
With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.
https://www.hyrumslaw.com/
<
>
I changed the behavour of the steam client to address a reflection attack vulnerability in the Steam server browser protocol. This is an *ancient* protocol (early 2000's?) with an ecosystem that's evolved to workaround its flaws. A perfect situation where Hyrum's Law applies.
[Quick primer on reflection attack: Attacker sends a UDP packet to a host, forging source IP to be IP of the intended victim. The host replies -- to the *victim*. ("reflection"). If reply size > request size (as in this protocol), attacker can amplify a DoS attack.]
My first attempt to fix this was to simply pad the request packet to 1200 bytes. This is a standard approach. It removes the amplification and makes the request expensive for the client. Also, it makes it easy to drop any bad packets at the edge.
I have access to the "official" code that processes these packets, so I knew that that code discards any extra data appended to the end, so padding was compatible. I couldn't know what any *other* code or middleboxes would do, but I did some quick testing and things seemed OK.
Unfortunately, when I shipped the beta client with this change, I broke @playrust. (This is why we have a beta.) A bunch of servers disappeared from the list.
Some gameserver companies apply pretty aggressive filtering to these packets to protect against DDoS.
Some gameserver companies apply pretty aggressive filtering to these packets to protect against DDoS.
So....padding is not going to work. Attempt #2: the other standard approach: authenticate using an anti-spoofing challenge:
1.) Initial client request.
2.) Respond with a random number (the "challenge").
3.) Client includes challenge # w/ request, proving it's their IP
1.) Initial client request.
2.) Respond with a random number (the "challenge").
3.) Client includes challenge # w/ request, proving it's their IP
That didn't work either! Why? I had changed the request to be similar to two other related queries that already used challenges:
1.) Initial request uses challenge = 0.
2.) Server responds w/ challenge.
3.) Real request w/ challenge.
Request pkts #1 & #3 are same format.
1.) Initial request uses challenge = 0.
2.) Server responds w/ challenge.
3.) Real request w/ challenge.
Request pkts #1 & #3 are same format.
I discovered that if the initial packet differed from *exact* original sequence of bytes, it would be dropped. And my change caused the initial packet to have 4 extra bytes.
Now it's hard to fix a protocol when you can't anything other than an exact sequence of bytes.
Now it's hard to fix a protocol when you can't anything other than an exact sequence of bytes.
The solution: the client must act EXACTLY like an old client until the server opts into the new behaviour. If the server wants to reply (possibly to a spoofed address!), it can. Or it can ask for a challenge.
But the server needs to be in charge of migrating to new protocol.
But the server needs to be in charge of migrating to new protocol.
OK, so there's no official documentation warning, "Extra stuff might be added to the end of this packet in the future."
But even if it existed:
1.) Nobody would read it.
2.) Spec lawyering would not change the practical reality of this being my "fault".
</>
h/t @kabdib
But even if it existed:
1.) Nobody would read it.
2.) Spec lawyering would not change the practical reality of this being my "fault".
</
>h/t @kabdib
The fix has been released: https://steamcommunity.com/groups/SteamClientBeta/announcements/detail/2896341257765264787
