Read on Twitter
I don't have time to write a blog post, so I'm going to do a Twitter thread instead about soft/hard object fields and perturbation confusion. 1/n
Say you want to implement forward AD. The "obvious" thing to do is to associate all your objects with tangents (expressing the differential change) and overload your operators so it works. 2/n
If you do this the naive way, a famous problem called "perturbation confusion" arises when you try to nest multiple levels of forward. This is because you get confused about which level of AD a tangent corresponds to. 3/n
The classic fix to perturbation confusion is to associate tangents with a label, so you don't get "confused" about which level the tangent is associated with. 4/n
OK. Let me talk about something completely different: weak maps and private symbols in JavaScript. http://blog.ezyang.com/2013/03/duality-of-weak-maps-and-private-symbols/ 5/n
Whenever you decide to add a new "private" field to an object to store some extra information, you have a choice; you can directly store it on the object, or you can instead store it out-of-line in a weak map 6/n
The features are inter-expressive; you can implement one with the other and vice versa. But they are distinct concepts. 7/n
Back to forward AD. Instead of defining dual numbers and storing tangents directly on the things you're differentiating, you could flip to the other side and instead maintain a weak map of tangents in your system. 8/n
Whenever you start a new level of forward AD, that's a new weak map. There is no possibility for forward confusion, because whenever you do forward AD for a level, you consult only the map at that level. 9/n
If a dual number "escapes" from its scope, well, that's just fine, you use the primal only. (Or maybe you allow for non-lexical forward AD, which is fine too.) 10/n
It is more efficient to implement dual numbers as private symbols than weak maps. But the weak map interpretation is more semantically coherent and solves design problems you might have. Thanks for coming to my TED talk. FIN.
