Nimbus Development Update: A Public Ethereum 2.0 Testnet
We did it 😱🎉 We have a Nimbus-to-Nimbus testnet running, not just in a simulation on a single machine, but with a remote bootstrap node and people connecting to it - even from outside of Status!
This is a major milestone in the development of Ethereum 2.0, and while stability issues persist and bugs will happen, the fact that we now have a Beacon chain synchronizing across nodes that aren't necessarily local to each other is a big deal.
We've named it testnet0 - this is the first in a series of testnets that will follow, each adding new features, scale and stability. During the first weeks, testnet0 will be frequently restarted to incorporate what we learn from feedback and monitoring. This iteration is based on spec version 0.5.1 and uses 400 validators to secure the network, of which 50 are reserved for brave explorers - when you join us, you'll get one randomly assigned to you.
Note that if at any point you get stuck, you can ask us for help in the #status-nimbus channel in Status, or if Chaos Unicorn Day knocks Status out, find us in Gitter.
Starting from Nimbus
We'll be starting all instructions from the Nimbus parent repository. The nim-beacon-chain repository is a dependency of Nimbus which contains all Ethereum 2.0 code, but this code is meant to be launched from the Nimbus parent repository. The nim-beacon-chain code is pulled in as a git submodule automatically and comes with a corresponding Makefile of its own.
Please note that the instructions in this post apply to Linux and Mac OS only - you will not be able to run them on Windows unless you're using something like our nim-vagrant VM.
Joining a Nimbus testnet
Want to try joining our testnet? Here's how.
Joining from scratch
Here is the full process if you're starting from scratch, without even Nim installed (you still need RocksDB though, so install that first):
# Ensure you have rocksdb installed before running this!
git clone https://github.com/status-im/nimbus
cd nimbus
make update # this might take a few minutes
cd vendor/nim-beacon-chain # All Ethereum 2.0 functionality is in here
make testnet0
./build/testnet0_node # this launches the testnet0-specific node you just built
Joining after an update
If you've run our testnet before, you may have kept some cache around that's no longer compatible with the newest version. You may also have outdated sources which now need to be brought in sync with our master branch and rebuilt. This is done similarly to starting from scratch:
cd nimbus
make update # update dependencies
cd vendor/nim-beacon-chain
make clean-testnet0 testnet0 # clean cache and rebuild binary
./build/testnet0_node # this launches the testnet0-specific node you just built
Congratulations, you should now be joining us - your node will start syncing with the current state of our beacon chain. Once you're in sync, you should also start proposing your own blocks and providing attestations - exciting! You are now among Ethereum 2.0 Nimbus pioneers!
So what does make testnet0 do?
Behind the scenes
When running the make testnet0 target, a custom beacon node pre-configured for running with this specific testnet gets built by executing a special shell script. That shell script has this for content:
#!/bin/bash
[ -z "$1" ] && { echo "Usage: `basename $0` testnetX"; exit 1; }
set -eu
cd $(dirname "$0")
NETWORK_NAME="$1"
source "$NETWORK_NAME.env"
cd ..
NIM_FLAGS="-d:release --lineTrace:on -d:chronicles_log_level=DEBUG -d:SECONDS_PER_SLOT=$SECONDS_PER_SLOT -d:SHARD_COUNT=$SHARD_COUNT -d:SLOTS_PER_EPOCH=$SLOTS_PER_EPOCH -d:DEFAULT_NETWORK=$NETWORK_NAME --hints:off --verbosity:0"
BEACON_NODE_BIN="build/${NETWORK_NAME}_node"
CMD="nim c $NIM_FLAGS -o:$BEACON_NODE_BIN beacon_chain/beacon_node"
echo "$CMD"
$CMD
if [ ! -d ~/.cache/nimbus/BeaconNode/${NETWORK_NAME}/validators ]; then
$BEACON_NODE_BIN --network=$NETWORK_NAME importValidator
fi
echo
echo "A binary for connecting to $NETWORK_NAME was placed in '$BEACON_NODE_BIN'"
echo "To sync with the network, launch it with default parameters"
echo
The first argument of this command (i.e. "testnet0") loads the contents of testnet0.env which contains environment variables needed by the beacon node we'll be launching. Then some Nim flags are set for optimized compilation and other variables are passed into the node in order to make it compatible with the testnet's rules - notably, the number of slots per epoch and shards. We need special rules because we're running a small testnet that differs from the spec significantly.
Then, if this is your first time joining the testnet, the beacon node will use the importValidator command to actually go to the server hosting our testnet's bootstrap node and download a random private key for you - we've got some pre-generated ones ready. With this key, it will be as if you had deposited 32 ether via this address to become a validator. If your node dies or you need to rejoin the network for some reason, the same validator will be reused. Note that you cannot plug your own private key in there because it's not on our list of those who "deposited" 32 ether into the network to be qualified.
Finally, the beacon_node binary is generated in the build directory, named after the network in question: build/testnet0_node. When launched, it will do the following:
- go to our infrastructure server
- find the metadata file matching the name of the provided network
- read the bootnode address(es) from this file
- connect the beacon node to bootnodes, thereby joining the network
We're running it this way to keep a close eye on things - we're scanning the situation in both Grafana and Kibana, but if you'd prefer to roll your own testnet, see below.
What happens when you join the testnet
Click the images to enlarge them.
First, your node will connect to the bootstrap node specified by the downloaded metadata file. All other nodes are also connected through it. Then, this node will start fetching all the up until now produced blocks.
Once it has all this data, it will store it to state - write the data into its local database and build the chain. So first it grabs the raw data, and then it turns it into a blockchain by verifying it.
Once it stored everything it'll do another round of fetching - in case new slots passed since the last block history was downloaded. A few rounds of this will follow until the node is in sync.
At that point, it will attach its local validator (the validator you downloaded automatically when you joined the testnet), shuffle it into the committee responsible for attesting and proposing of a given shard and check regularly if it is its turn to do work.
The node then waits for the validator's turn to do something, while at the same time constantly requesting data from the other nodes and keeping the chain up to date. When time comes for the validator to do its magic, the node will poke it and it'll either generate an attestation which the node then sends into the network, or it will propose a block, or both. The node will also justify and finalize blocks.
When quitting the node and relaunching later, the data you already have wil be reused: both the validator key and the database. To clean things up and start from scratch, run make clean-testnet0.
Creating a Nimbus testnet
To start the testnet, our beacon node now has a createTestnet command. This helps it generate an initial snapshot, metadata file for other joiners, select the proper ports and network for others to join it on, and more.
Before we begin, please note that all beacon nodes joining your custom testnet MUST be compiled with the same beacon chain constants - i.e. if a node is compiled with 8 slots per epoch, one with 16 slots per epoch will not be compatible with it. With that in mind, let's do this step of building the basic beacon node binary from within vendor/nim-beacon-chain, and let's also build the tool that can generate validator keys.
export NIMFLAGS="-d:release -d:SECONDS_PER_SLOT=30 -d:SHARD_COUNT=8 -d:SLOTS_PER_EPOCH=8" \
&& make beacon_node validator_keygen
This will place the beacon_node binary and the validator_keygen tool in build in the current folder, i.e. vendor/nim-beacon-chain.
Let's generate the folders where the node will store its data and then add the validator keys in there. I picked 500 keys.
mkdir -p $HOME/testnets/custom-network/data
./build/validator_keygen --totalValidators=500 --outputDir="$HOME/testnets/custom-network"
This will have placed 500 private key files and 500 public key files into the specified output folder.
Next, let's have the node generate a testnet for us with all the bells and whistles we might need to have others connect to us.
export NETWORK_DIR=$HOME/testnets/custom-network && ./build/beacon_node \
--dataDir=$NETWORK_DIR/data \
createTestnet \
--networkId=666 \
--validatorsDir=$NETWORK_DIR \
--totalValidators=500 \
--outputGenesis=$NETWORK_DIR/genesis.json \
--outputNetwork=$NETWORK_DIR/custom-network.json \
--bootstrapAddress=$(curl -s ifconfig.me) \
--bootstrapPort=34000 \
--genesisOffset=600
We set the home folder of the custom network, pass it to the required params and have it generate the genesis file and the metadata file for others to join us through. The bootstrap address part is dynamically generated from the ifconfig.me service - you can manually input your public IP address here if you know it or if that service fails to detect it - test with curl -s ifconfig.me on the command line. The port is optional but recommended so you don't get some cross-chain noise when accidentally connecting to other nodes on the default port.
The genesisOffset flag sets the time of genesis to some time in the future - in this case 10 minutes. We do this to give everyone who intends to join ample chance to join on genesis time because if they don't and the chain expects validators to be there and perform their duties, they'll be seen as offline and penalized and eventually kicked off the beacon chain.
Running the above command will result in two new files being created. In my case $HOME/testnets/custom-network/genesis.json and $HOME/testnets/custom-network/custom-network.json.
We are now ready to connect and to share these files with those who would connect to us. Please note that in order to allow others to connect to you, you need to open the port you chose manually on your router and firewall for as long as Nimbus doesn't yet have UPnP implemented. If you're hosting your bootnode on something like DigitalOcean or AWS, there are firewall controls there that are quite intuitive. For your own local computer, please consult your router's manual on how to do that.
The genesis.json file is the starting state, "block 0" of your testnet beacon chain. It contains the listed validators, initial shufflings, and everything the system needs in order to have the clients connecting to the network build on the same foundation. The custom-network.json file is the "metadata" file of your new network - it has identified your node, the one this file was generated with - as the bootstrap node and included its enode address under bootstrapNodes, along with the other required data and the root of the genesis. You can host this file on a server somewhere and give others your genesis file and they'll be able to join your network if they execute the command:
./build/beacon_node --network=$HOME/testnets/custom-network/custom-network.json --stateSnapshot=$HOME/testnets/custom-network/genesis.json --tcpPort=34001 --udpPort=34001
[
Visualization
To better present what's going on, we are building a rudimentary visualizer which will display the nodes, slots, blocks, epochs and attestations in a graph on screen. Hopefully, that'll help those uninitiated understand the beacon chain better, and will help us visually detect some anomalies.
This is an early preview right now, so things will look significantly different when done. Source code coming soon.
We hope you'll join us on our journey through Ethereum 2.0 - connect to our testnet, ask questions in our chat rooms, and stay tuned for more exciting updates coming in the next fortnight!