Nethermind
Understanding the Nethermind Syncing Process
Speed Advantage of Nethermind
Nethermind can sync to the network relatively quickly, by first downloading only the latest state, headers, and optionally bodies and receipts. This enables the Consensus Client to begin performing attestations in just around 3 hours.
Syncing is a multi-stage process that Nethermind undergoes to download, verify, and reconstruct the complete global state of Ethereum. Initially, Nethermind connects to other participating nodes, known as peers, on the network to exchange data. It then proceeds to download chain and state data, and rebuilds a local representation of the entire Ethereum state.
The following sections will outline the different stages of syncing and provide insights into what you can expect to observe in the logs. This knowledge will enable you to quickly assess whether the syncing process is proceeding smoothly or if any issues require your attention.
Monitoring the Syncing Process
There is a known issue with the progress bar on the Home screen not functioning properly for Nethermind. You may notice that the progress remains at 0% for a significant amount of time, and then suddenly jumps to 100% when Nethermind is fully synced. To reliably monitor the progress, it is recommended to use the Logs, as they provide a more accurate depiction of the syncing process.
To ensure a smooth syncing experience, we recommend familiarizing yourself with the sync process and actively monitoring the Logs. By doing so, you can gain confidence in determining whether the syncing process is progressing smoothly or if any issues require your attention.
To access the Logs, go to the Management Page for Nethermind. You can navigate to this page from the My DApps menu. The Management Page offers a range of controls and settings to manage Nethermind.
On the Management Page, simply scroll down to the bottom to find the logs.
The Health Check Page
Nethermind features a Health Check page that provides a quick overview of the system's status, indicating whether everything is functioning normally. You can reach this page by clicking on the Open button on the My DApps page (or simply clicking the name of the Nethermind package). Alternatively, you can also come to this page by clicking on the Nethermind icon on the Home Page.
In this example, the status of Nethermind is "Healthy". It is fully synced, and it is connected to 50 peers (which is a good number for and Execution Client).
Stage 1. Startup
When Nethermind starts up, it goes through a series of configuration details and status checks. Afterward, it swiftly proceeds with the following tasks:
Initiating the syncing of beacon headers
Commencing the snap data sync
Initiating the download of "old headers"
A healthy representation of the Logs should appear as follows. Take note that the number of "Peers" should gradually increase towards the maximum of 50. Additionally, the percentage count of "Snap" and "Old Headers" should steadily rise.
However, there are some common issues that may arise:
Nethermind unable to find peers
Nethermind cannot sync until it is connected to a sufficient number of peers.
If Nethermind has problem connecting to peers, you may see the following messages in the Logs:
Simultaneously, this will also show up on the "Health Check" screen:
Most likely there is a network problem. Resolving this may require adjusting certain settings, such as "enabling uPnP" or "opening port 40303" (port 30305 for Gnosis Nethermind) on your router. You can refer to this guide on Opening network ports.
No Consensus Client detected or Consensus Client not operational
Nethermind cannot sync until the connected Consensus Client is synced.
If you have not installed and synchronized a Consensus Client (such as Nimbus, Teku, or Prysm), Nethermind cannot sync. If you encounter the following messages:
It is likely that either you have not installed and synced a Consensus Client, or there is an issue with your existing Consensus Client. Check the logs of the Consensus Client to determine the cause of the problem.
Sometimes, there may be a communication problem between Nethermind and the Consensus Client. Restarting both can often resolve this. Begin by restarting the Consensus Client, and then restart Nethermind. You can find the Restart button on the respective client's Management Page.
Connection refused
When initially starting Nethermind, you may see this error on the Home Page:
This warning may suggest a potential issue with Nethermind, but it could also be a non-issue. To determine the cause, it is advisable to examine the Nethermind logs for any problems related to starting Nethermind, finding peers, or connecting to the beacon chain. By addressing any identified issues, the errors and warnings should no longer persist.
If the Nethermind logs appear normal and do not indicate any problems, it is possible that the anomalies are transient and will resolve themselves without further intervention.
Blocks Synced: 0
Also during the initial phase, you may find that the progress bar on the Home Page does not progress and remains at "0":
There is a known issue with the progress bar on the Home Page not functioning properly for Nethermind. You may notice that the progress remains at 0% for a significant amount of time, and then suddenly jumps to 100% when Nethermind is fully synced. To reliably monitor the progress, it is recommended to use the Logs, as they provide a more accurate depiction of the syncing process.
Stage 2. Snap / Fast Sync, and State Sync
Nethermind employs an efficient syncing method to quickly sync to the blockchain, enabling the Consensus Client to commence validation early on. It achieves this by initially downloading only the latest state, headers, bodies and receipts, while deferring the download of older block data to a later stage. This progress is indicated by the following Log message:
with the percentage value steadily increasing.
During the process, you will observe that CPU usage remains consistently high (100%). Nethermind regularly reports the following actions in the Logs:
Syncing... Inserting block ...
Received ForkChoice ...
Received new block ...
Synced chain Head to ...
Changing state ...
Sync mode changed ...
Peers | with best block: 25 | all: 25 | ...
These Log messages are normal. Regarding the "peers" numbers, it is favorable to see them gradually increasing, reaching a maximum of 50.
Once Nethermind completes the Snap / Fast Sync, it transitions into a phase known as "state sync". As the state sync nears completion, you may encounter a series of "branch sync" messages. At a certain point, the entire state is downloaded, and the node enters the "full sync" mode.
At this stage, the Consensus Client can begin its validation responsibilities!
It is typically expected for Nethermind to reach this milestone in approximately 3 hours.
Stage 3. Download Old Bodies and Old Receipts
Once the state sync process is complete, your node will be fully synced. If you have already activated your validators, you should notice that they start attesting.
However, for a few hours, Nethermind may still appear busy, and the CPU usage may remain high. This may even cause your validators to miss attestations if they are already activated.
This is because Nethermind is engaged in downloading the old blockchain data, called "Old Bodies" and "Old Receipts". These data sets are needed to ensure that your node possesses a complete view of the blockchain. Downloading the old chain data is a resource intensive task that requires significant computational power and disk space.
During this phase, you will observe similar messages in the logs:
and later:
Notice that the percentage value should steadily increase. It is normal for your validators to potentially miss some attestations during this phase due to the high system load. Operations will stabilize once the download is complete.
Remember to ensure that you continue using a fan to keep your AVADO device cool.
Normal Operations
Once your Nethermind client is fully synced and operating normally, the logs should appear similar to the following:
The shows that Nethermind continuously imports new chain segments and updates the chain head to maintain the latest state of the blockchain.
How long does it take?
The syncing process can be time-consuming, and the duration will primarily depend on the specifications of your hardware and the speed of your network connection.
Drawing from a recent experience (Mar 2024), using an AVADO i7/4TB device with a 1 Gbps fiber connection, it took only approximately 2 hrs 40 mins from the start of the process until the Consensus Client was able to initiate attestations. The complete sync took about 11.5 hours.
A more detailed breakdown is as follows. This is for reference only. Your experience may be different.
| Elapsed Time | Actions |
|---|---|
| Start of process Begin "Syncing beacon headers ..." Begin "Snap - State Ranges (Phase 1) ..." Begin "Old Headers ..." |
| Finished "Old Headers" Continued "Snap - State Ranges (Phase 1) ..." CPU usage stays high (100%) On Home Page, progress remains at 0% |
| Finished "Snap - State Ranges (Phase 1) ..." Begin "State sync" |
| Finished "State sync" Finished "Branch sync" Consensus Client started attestations On Home Page, progress jumps to 100% Begin "Old Bodies" |
| Finished "Old Bodies" Begin "Old Receipts" CPU usage stays high (100%) |
| Finished "Old Receipts" Nethermind database volume: 823.6GB |
Last updated
