I've seen people asking: "What is the biggest problem that Avalanche faces?"

It's a brand new system; built from the ground up. Which means it's still relatively untested.

At 5:30 AM yesterday (12/10/2020), a transaction was issued to the network.

The transaction was quickly accepted by the network, except by one of the nodes.

https://explorer.avax.network/tx/2tr2pyFwpSVDCEUHMS3TPRNutZYypvjzGKkWNzmNGij6UGKLYU

On the impacted node this transaction was being abandoned, before being verified, because one of it's dependencies was failing to report as accepted.

Interestingly, the parent transaction was also accepted by the rest of the network.

https://explorer.avax.network/tx/DWeJ3dqpFbDCqDnbAUKDLfiuj2pDgur7k5zqhz8QbbEQjawo2

Looking a bit more into this transaction, we found that it was issued into vertex "zECXcoQQntuqKhcnX3tsLc5xbptqco3cj3JTB9UYihbTy6ZHF". Which also reports as being accepted by the network.

But the impacted node also claimed to have accepted this vertex!

This was the primary invariant break. A vertex should only be marked as being accepted if
- all the parents of the vertex are marked as accepted
- all the transactions inside of the vertex are marked as accepted

Looking even more into this transaction, we see that it was actually issued to the network on 9/26/2020, 2 days before we released the v1.0.1 upgrade. This transaction was processed by a v1.0.0 node, which had a low FD limit and didn't properly handle DB write errors.

What seems to have happened is that this transaction was accepted. However, the write of its acceptance failed due to the low FD limit.

Regardless of this failure, the transaction cached that it was accepted in memory.

The vertex was then evaluated to see if it was acceptable, the consensus engine confirmed it followed the above invariant, and then accepted the vertex. The vertex's DB write then succeeded.

When the node was later upgraded to v1.0.1, the FD bug was fixed, but this node's DB had already been silently corrupted. This didn't impact the node until the dependent transaction was attempted to be issued yesterday.