I have finally finished @robinhanson's Age of Em. I definitely cannot recommend the book as-is, but I think that a condensed version that was about 25% as long and contained the main ideas and some narrative would be an incredibly important book. https://twitter.com/RokoMijicUK/status/1337348367383470080

Age of Em really is an intellectual tour de force though. If you accept the assumptions of the scenario I suspect that the analysis will likely be quite accurate.

My main criticism of Age of Em is that I think Robin bends a bit too hard towards making people feel like Ems are relatable and nice.

I think that given Robin's Assumptions, Ems will be horrific.

To be concrete, imagine your boss at work giving you orgasms for hitting your KPIs. And then summarily deleting you because you made a mistake. And then another copy of you complaining and being told that it's in the contract.

Imagine a world where everyone is a type-A person but with all the hobbies and quirks deleted to save space. Trillions of person-hours with people's emotions, orgasms and romantic feelings being tied to financial results.

Friendships being deleted as part of a contract negotiation.

Robin has various reasons why that might not happen, but I think that an equally skilled analyst with a good imagination could think of plenty of inventive ways that the Em world will be a horror.

Ultimately the robust reason for why Ems will be horrific is because they reproduce via a different channel than us. We come out of vaginas, they don't.

Ems are created by direct copying, and over time their world will be shaped by that.

Do I personally think that the assumptions of Age of Em are likely? No, not at all.

Every new breakthrough in deep learning is evidence against the Uploads First scenario. Intelligence is generated by a few general algorithms applied to large datasets and with lots of compute.

Still, one of the advantages of the Em Scenario is that it's possible to analyze it. And that's really important!

And many of the aspects of Age of Em generalize to non-Em futures, such as the importance of reversible computing and cooling.

If you want the most accurate picture of the future we have today, imagine Age of Em but with the Em clans replaced by a few big deep learning systems that are gradually doing more and more self-modification.