Read on Twitter
My experience trying to run a full Ethereum node for the last few days ended in a failure. Here's a log of my attempt.
First, background: I'm a programmer with 10+ years experience in development on Linux and another decade or so on Windows before that.
First, background: I'm a programmer with 10+ years experience in development on Linux and another decade or so on Windows before that.
My computer is an Intel NUC that I use for working from home. 16GB of RAM. 1TB SSD. 8 cores.
First, I'll try to download and verify Geth binaries.
Can't seem to ascertain who signed the AMD 64 binaries. The fingerprint is 9BA28146 but googling it leads nowhere.
First, I'll try to download and verify Geth binaries.
Can't seem to ascertain who signed the AMD 64 binaries. The fingerprint is 9BA28146 but googling it leads nowhere.
My aim is to use pgp to establish a chain of trust between the signed Ethereum artifacts and some well known developer/maintainer, from whom I could obtain the fingerprints out of band (like asking on discord, twitter, etc).
I'm operating under the assumption that I'm about to store 20% of my net worth in Eth and I don't to trust 3rd parties like GitHub too much, some CI server, or even my Chrome for that matter.
The how-to guide for getting started with Geth recommends getting the keys with `gpg --recv-keys` but that leads to an error: 'keyserver receive failed'. I'll just try to import them directly instead. But it seems the binaries are signed by some bot, not a human.
I'll try to build Geth from sources instead. The latest release on GitHub is v1.9.18. Downloaded the tar ball. There's no signature data to be found anywhere. How do I know I can trust this commit?
Friendly developers on Discord say the commit is verified by GitHub, but that amounts to trusting GitHub basically. He says they don't really do manual signing of source tar balls...
Overall impression thus far is the Geth people don't really stress PGP and chain of trust too much. Things are being built by bots and people just download binaries without verifying them.
So I guess I'll proceed by trusting the binary and won't really put 20% of my money there.
So I guess I'll proceed by trusting the binary and won't really put 20% of my money there.
You gotta love golang! A single binary without any dependencies. Let's figure out how to run this thing.
Seems the important bit for me is the syncmode. I want to verify the entire blockchain from genesis.
./geth --syncmode full
.... and it's running! Easy.
Seems the important bit for me is the syncmode. I want to verify the entire blockchain from genesis.
./geth --syncmode full
.... and it's running! Easy.
After running overnight it seems to have crashed two hours into the sync. It just says "Killed" at around block number 2M. 
Oh well, stuff happens I guess. Let's resume it.
Oh, crap:
>> Rewound blockchain to past state number=0
it's starting from scratch.
Oh well, stuff happens I guess. Let's resume it.
Oh, crap:
>> Rewound blockchain to past state number=0
it's starting from scratch.
About two hours later it crashed again, seemingly in the same way and roughly in the same block number:
2281628 vs. 2258560 last time. I'm betting it has to do with resources. It's probably exhausting resources at roughly the same point in the history, but logs don't say which.
2281628 vs. 2258560 last time. I'm betting it has to do with resources. It's probably exhausting resources at roughly the same point in the history, but logs don't say which.
Let's try running again but watching the RAM over time in a split tmux pane:
while true; do free -h | grep Mem | ts; sleep 60; done
Keeping it simple. This way I see what the RAM situation is when it crashes again. I got 4GB free before starting up again.
while true; do free -h | grep Mem | ts; sleep 60; done
Keeping it simple. This way I see what the RAM situation is when it crashes again. I got 4GB free before starting up again.
Around block 2M I see the syslog getting CPU overheating errors. This thing is smoking all my 8 cores. Maybe that's how it's crashing? Being throttled because of cpu temp is killing it somehow? :shrug:
Oh man, my computer is almost nonresponsive.
Oh man, my computer is almost nonresponsive.
I'll throttle the cpu for the Geth process with `cpulimit`, so at least I can use the machine while it's syncing, and maybe that would prevent the crashing as well?
Running it again now (again, not resuming, starting from scratch) but this time with cpu capped at 50%.
Running it again now (again, not resuming, starting from scratch) but this time with cpu capped at 50%.
After a few hours it's still running! Albeit more slowly now since the CPU is capped. But I have patience. Self sovereignty isn't for high time-preference people.
Let's let it run overnight.
Let's let it run overnight.
Morning after: it crashed. Same deal. No log errors, it just says "Killed".
RAM seems to not have been the issue. There was 1.2GB free around that time. And it couldn't have been CPU - with cpulimit in place in never went above 30% +/-.
Why is geth crashing?
RAM seems to not have been the issue. There was 1.2GB free around that time. And it couldn't have been CPU - with cpulimit in place in never went above 30% +/-.
Why is geth crashing?
Okay, last attempt running Geth. I'll increase the log verbosity to debug and wait for the next crash. Also limiting cpu with cpulimit so my computer stays responsive (geth team, this should really be a CLI option...).
After like half a day it crashes again, this time a bit farther (block 2.4ish). But the culprit is clearly RAM. The OOM killer done killed it.
kernel: Out of memory: Killed process 30852 (geth) total-vm:8469772kB, anon-rss:5359604kB, file-rss:0kB, shmem-rss:0kB
kernel: Out of memory: Killed process 30852 (geth) total-vm:8469772kB, anon-rss:5359604kB, file-rss:0kB, shmem-rss:0kB
I guess that's it. I'm not going to buy a bigger machine for this ($), and I'm not going to run it on the cloud (trust). This was supposed to be a home Geth setup. I guess a sturdy NUC isn't enough. By comparison, my Bitcoin full node runs on a Raspberry Pi 3 (much weaker).
Geth team, please consider the following:
* Allow throttling CPU as a command line option.
* If there's a well know minimum RAM requirement please warn during startup.
* Resuming from last sync point after hours of work is crucial and doesn't seem to work when OOM killed.
* Allow throttling CPU as a command line option.
* If there's a well know minimum RAM requirement please warn during startup.
* Resuming from last sync point after hours of work is crucial and doesn't seem to work when OOM killed.
