Read on Twitter
I am releasing details today. All information has been gathered, timelines analyzed, old code vetted, major flaw fixed, and results scrutinized. Strap-in boys and girls, this rollercoaster goes to 11! Sun Microsystems #GridEngine has been hiding an ace 🂡 up its sleeve for 16 yrs https://twitter.com/freebsdfrau/status/1274627170573225985
Show you the code? OK ... https://github.com/FrauBSD/sge/
Still interested in how a decades-old job scheduler might have contained the necessary bits to dethrone even future contenders?
Stick around and we'll see how Sun #GridEngine may just surprise you
Still interested in how a decades-old job scheduler might have contained the necessary bits to dethrone even future contenders?
Stick around and we'll see how Sun #GridEngine may just surprise you
Schedulers: Airflow, Mesos, Torque, Univa, Open Grid, Sun Grid, Cook, Slurm, there's so many. For the sake of argument, let's say you need to schedule 10M jobs but you only have 100K cores. Does your scheduler break down like most of those listed at 10-30K pending?
Some schedulers thought they would out-perform decades-old Sun #GridEngine by using an optimization, but little did they know that SGE contained the same exact optimizations (off by default and unstable) more than 10 years earlier https://slurm.schedmd.com/SUG14/sched_tutorial.pdf
Univa -- the ultimate successor-to, and based-on, Sun's code -- even spent time optimizing the scheduler and boasts a whole 6x improvement over SGE's scheduler. But, young grasshoppers, what Univa and friends do not know is that SGE has been hiding this feature I will show you
Hop in the time machine with me, and let's travel to the year 2004, a hair over 16 years ago. A Sun Microsystems employee by the name of Stephan Grell had been working on an experimental feature to improve the performance of #GridEngine dramatically and on July 30, he commits it!
Stephan Grell did this in 2 landmark commits (would have been one, but he forgot to add the documentation to his first commit, so soon thereafter came another)
https://github.com/gridengine/gridengine/commit/710eef6c342109abaafe09c21a4e33252b79f0f1
https://github.com/gridengine/gridengine/commit/2d013816ea42b2922d515d8eef134436f73fda46
We have the history thanks to Oracle and Univa for keeping it around
https://github.com/gridengine/gridengine/commit/710eef6c342109abaafe09c21a4e33252b79f0f1
https://github.com/gridengine/gridengine/commit/2d013816ea42b2922d515d8eef134436f73fda46
We have the history thanks to Oracle and Univa for keeping it around
About 20 days later, Stephan Grell then updated the SGE tuning guide to advertise his new feature. My estimates put his optimization anywhere between 10x and 600x improved efficiency depending on real-world testing.
https://github.com/gridengine/gridengine/commit/935d04f03bd96d7619a0762898905635a1ef7dbf
http://gridscheduler.sourceforge.net/howto/tuning.html
https://github.com/gridengine/gridengine/commit/935d04f03bd96d7619a0762898905635a1ef7dbf
http://gridscheduler.sourceforge.net/howto/tuning.html
What was to happen next is an absolute tragedy (with a deferred happy ending albeit 16 years later). The feature sat idle (disabled by default) and likely became forgotten. 3 years 9 months later, Sun employee Roland Dittel motions to remove it https://github.com/gridengine/gridengine/commit/2c847db69b9ada5d326f485239829126d9170f50
2 months after Roland decided that this experimental feature (now ~4yrs old) needed to go, he sent out one single e-mail to the #GridEngine mailing lists, asking if anyone uses it before he wields his axe http://arc.liv.ac.uk/pipermail/gridengine-users/2008-June/019407.html
If it were not for Jeff Beadles [same-day] response to Dittel, I myself might never have learned of the feature. Spoiler alert, despite Jeff's plea to save the feature, Dittel marks it for death anyway, just 14 days after Beadles' plea https://github.com/gridengine/gridengine/commit/29dfb0c4442fd64b5fa7dd18ca9d7aec38a248ff#diff-28d6c34e92ac1f3a1ab9a1a0f2ecf9dc
Through a VERY fortunate turn of events, we still have the feature to work with today. Less than 4 months after Roland Dittel marked the optimization for removal, IBM and Sun began to discuss a potential merger, on November 6th, 2008 https://web.archive.org/web/20130615185114/http://www.sdtimes.com/link/33610
Another 4 months later, in March of 2009, talk of merger between IBM and Sun had stalled https://en.wikipedia.org/wiki/Sun_acquisition_by_Oracle#History
The following month, on April 20, 2009, Oracle and Sun announced a definitive agreement. 9 months later (because that's just how long these things take ^_^), on January 27th, 2010, Oracle announces completion of its acquisition of Sun Microsystems https://en.wikipedia.org/wiki/Sun_acquisition_by_Oracle#History
12 months after that, on January 18, 2011, Univa announced it had acquired from Oracle the core Sun #GridEngine developers including the CTO https://en.wikipedia.org/wiki/Univa_Grid_Engine#History
2 months later, in March, 2011, Univa rebrands SGE as Univa Grid Engine primarily through these 3 commits
https://github.com/gridengine/gridengine/commit/de2ac8be587464424d354d88de41a1ade5cff5dd#diff-c49182dc0c7a70b9cd2e10853d9ec6c7
https://github.com/gridengine/gridengine/commit/b094d2d025293cb1cce8727828821e3a0e621262#diff-c49182dc0c7a70b9cd2e10853d9ec6c7
https://github.com/gridengine/gridengine/commit/af5932a0ad03037bc71176dfac67e881bfc1ef31#diff-c49182dc0c7a70b9cd2e10853d9ec6c7
https://github.com/gridengine/gridengine/commit/de2ac8be587464424d354d88de41a1ade5cff5dd#diff-c49182dc0c7a70b9cd2e10853d9ec6c7
https://github.com/gridengine/gridengine/commit/b094d2d025293cb1cce8727828821e3a0e621262#diff-c49182dc0c7a70b9cd2e10853d9ec6c7
https://github.com/gridengine/gridengine/commit/af5932a0ad03037bc71176dfac67e881bfc1ef31#diff-c49182dc0c7a70b9cd2e10853d9ec6c7
2 weeks after they do that, they (Univa) release as the company's first initial offering, Univa Grid Engine 8.0, on April 12, 2011 https://en.wikipedia.org/wiki/Univa_Grid_Engine#Releases
Univa continues to develop their "Open Core" publicly on GitHub for another 13.5 months https://github.com/gridengine/gridengine/commits/master
The feature created by Stephan Grell at Sun 16 years ago which got lost in the turmoil still exists in both the last version of the Univa Grid Engine Open Core, last updated June 1, 2012, as well as latest son-of-gridengine, updated June 15, 2018 (just 2 years ago)
You can see Stephan Grell's work, never actually removed, despite being on death-row for 12 years, still in the code here:
https://github.com/gridengine/gridengine/blob/6a5407d56c85b39290ac2488fb6dec1a4404a974/doc/man/man5/sched_conf.5#L361
https://github.com/son-of-gridengine/sge/blob/5432d9d6447a926209f130c05a26fcc640e2d630/doc/man/man5/sched_conf.5#L374
https://github.com/gridengine/gridengine/blob/6a5407d56c85b39290ac2488fb6dec1a4404a974/doc/man/man5/sched_conf.5#L361
https://github.com/son-of-gridengine/sge/blob/5432d9d6447a926209f130c05a26fcc640e2d630/doc/man/man5/sched_conf.5#L374
So, what do we do with features that were put on death-row by companies that don't control the code anymore and don't even exist anymore?
We give them a pardon! We buy them a nice suit, give a hot meal, and a let them lose in the World and see what they can do.
We give them a pardon! We buy them a nice suit, give a hot meal, and a let them lose in the World and see what they can do.
Technically speaking, this feature is old enough to legally work, so put it to work we did.
Immediately it had solved the problem (expertly described to Dittel by Beadles in 2008).
But we turned out backs, and by the 3rd day it had vomited all over the place!
Immediately it had solved the problem (expertly described to Dittel by Beadles in 2008).
But we turned out backs, and by the 3rd day it had vomited all over the place!
And now we knew why it was marked experimental, given a death sentence, put on death-row, and awaited execution. However, ... the people making those prior decisions never had me look at it. Recognizing the raw performance improvement, I more than doubled-down. It would be fixed.
It was May 28th, 2020 @ 17:23:08 PST when we turned it on for the first time. We couldn't even get to June before it crashed. Super nasty. Would take over 15 minutes to come back up. Definitely not acceptable. It wouldn't be until July 29 @ 11:42:48 that we had the fixes in-place
Over a period of 2 months I would be wracked with highs when I thought it was fixed and lows when it proved the code had more to teach me. 2 months of doing battle in dark caves with no help and a dimly lit torch (the documentation). Weeks of abject horror followed by euphoria
The end result is that this feature, now with over 5M jobs under its belt and a full month of service, is looking like it has the mustard to dethrone every modern/maintained scheduler on the market today. I'm not joking. There's even a bakeoff happening in Amazon EC2 right now!
Here's some before/after photos of job-wait times for pending jobs on the cluster showing that in the department of getting your jobs onto the cluster, it's definitely winning. Red line represents when we turned the feature on
But let me put it into context of numbers.
With Slurm, you'll start to feel the pinch at 10K+ pending jobs. With SGE (without JC_FILTER by Stephan Grell), you'll feel the pinch at 30K+ pending jobs.
SGE + JC_FILTER ...
No pinch at 300k pending jobs and we're still going
With Slurm, you'll start to feel the pinch at 10K+ pending jobs. With SGE (without JC_FILTER by Stephan Grell), you'll feel the pinch at 30K+ pending jobs.
SGE + JC_FILTER ...
No pinch at 300k pending jobs and we're still going
So where do we get 600x optimization at the upper-end? With as little as 100K pending jobs, it can take the scheduler up to 30min to complete. Compared with just 3 seconds to complete with the same pending-job load and all you have to do is turn on JC_FILTER=1 using qconf
But, without this patch, you better not use qalter if you have JC_FILTER enabled https://github.com/FrauBSD/sge/commit/5788e0fa277aafd07e0c5288305d2fe0c1a5883a
