Read on Twitter
Oh wow, this was quite a long time ago, and I have a couple of things to add: a thread
https://twitter.com/golab_conf/status/1350036028858527744
https://twitter.com/golab_conf/status/1350036028858527744
When I made this talk I found out that many Go services out there could be taken down with a ridiculously little amount of requests by just finding pathological corner cases of the runtime and triggering them.
The summary is: the Go runtime used to be cooperative and it used to have yield-points auto-insertions.
This means that when you compiled your code the compiler would look at it and decide when to ask the runtime if it was time to switch tasks.
This means that when you compiled your code the compiler would look at it and decide when to ask the runtime if it was time to switch tasks.
Memory allocations, syscalls, function calls, "go" statements were common points to add these checks.
This would allow for a very efficient green threading: instead of having a runtime that peeked into function execution the functions would themselves yield.
This would allow for a very efficient green threading: instead of having a runtime that peeked into function execution the functions would themselves yield.
Moreover this is *transparent* to the user: when you write Go you don't have to care where your preemption points are. The compiler will take care of them for you.
But what happens if an attacker could trigger a behavior that caused the program to spend a very long time in a block that didn't have preemption points?
This would cause a big mess: GC could not happen, other goroutines could not run, everything would hang.
This would cause a big mess: GC could not happen, other goroutines could not run, everything would hang.
This has (1.15) been fixed: the runtime is capable of preempting most functions if they take too long to yield back execution.
It took years of work to make this happen and I am grateful to the Go team.
This is so old the issue has 3-digits https://github.com/golang/go/issues/543
It took years of work to make this happen and I am grateful to the Go team.
This is so old the issue has 3-digits https://github.com/golang/go/issues/543
So, if this is now fixed, what's there to talk about?
Well, this is fixed in Go, but not in other languages/runtimes.
JS and Rust have the same issue Go used to have, but worse.
Well, this is fixed in Go, but not in other languages/runtimes.
JS and Rust have the same issue Go used to have, but worse.
They both use the "poor person" implementation of async calls: colored functions and "awaits".
This means that runtimes (e.g. the browser or tokio or what is trendy in Rust at the moment) can only schedule stuff on preemption points.
This means that runtimes (e.g. the browser or tokio or what is trendy in Rust at the moment) can only schedule stuff on preemption points.
And those preemption points are *not* automatically inserted: they only happen on async/await calls.
With this mechanism this issue *cannot* be fixed in Rust or JS. It's going to stay there forever.
With this mechanism this issue *cannot* be fixed in Rust or JS. It's going to stay there forever.
So to recap: Rust and JS have a cooperative scheduling mechanism that cannot be made preemptive and preemption points are *manually* added by programmers (so it's even worse than the state Go was in when I gave the talk).
Moreover the Go runtime more or less knows what it is scheduling, the Rust one does not (the JS one is... weird on this regard).
So this caused some very bad behaviors already to the point tokio (one of the most used Rust runtimes) had to implement some quite odd mechanisms
So this caused some very bad behaviors already to the point tokio (one of the most used Rust runtimes) had to implement some quite odd mechanisms
(Note that tokio implements a runtime that is very similar to the Go one: https://tokio.rs/blog/2019-10-scheduler)
They added *counters* https://tokio.rs/blog/2020-04-preemption
Basically to avoid some of the pathological behaviors they give functions budgets.
They added *counters* https://tokio.rs/blog/2020-04-preemption
Basically to avoid some of the pathological behaviors they give functions budgets.
If a function is keeping the runtime for themselves for too long they are forcefully scheduled away... but this can only happen if two circumstances are met:
1) Every single library you depend on uses the tokio (and only the tokio) socket, timer, channel, etc.
1) Every single library you depend on uses the tokio (and only the tokio) socket, timer, channel, etc.
2) A yielding point is reached.
Yes, because the issue here is not just that yielding points have to be manually added, they might also still not give a chance to the runtime to schedule smth else.
Yes, because the issue here is not just that yielding points have to be manually added, they might also still not give a chance to the runtime to schedule smth else.
"How bad can this possibly be?"
I guess time will tell, but let's just say that a good chunk of the Go runtime efforts went into fixing this issue *for two years*, so it's probably not smth people can just ignore.
I guess time will tell, but let's just say that a good chunk of the Go runtime efforts went into fixing this issue *for two years*, so it's probably not smth people can just ignore.
And this is without even considering the memory leak issues Rust has, but that's a story for another time.
