Geth
Last updated
Last updated
Syncing is a multi-stage process that Geth undergoes to download, verify, and reconstruct the complete global state of Ethereum. Initially, Geth connects to other participating nodes, known as peers, on the network to exchange data. It then proceeds to download chain and state data, and rebuilds a local representation of the entire Ethereum state. However, the blockchain is also progressing at the same time, so Geth will also need to catch up to the current state.
The following sections will outline the different stages of syncing and provide insights into what you can expect to observe in the logs. This knowledge will enable you to quickly assess whether the syncing process is proceeding smoothly or if any issues require your attention.
To monitor the syncing progress, you can refer to the Home Page, specifically the Chains - Ethereum section. Keep an eye on the status bar, which will gradually advance towards 100%.
Once it reaches 100% and the progress bar turns green, your Geth client is fully synced, indicating that you are ready to perform your node duties on the Ethereum network.
To ensure a smooth syncing experience, we recommend familiarizing yourself with the sync process and actively monitoring the Logs. By doing so, you can gain confidence in determining whether the syncing process is progressing smoothly or if any issues require your attention.
To access the Logs, go to the Management Page for Geth. You can navigate to this page from the My DApps menu. The Management Page offers a range of controls and settings to manage Geth.
On the Management Page, simply scroll down to the bottom to find the logs.
When Geth starts up, it undergoes a series of configuration details and status checks. It then initiates the process of connecting to other nodes on the network until it establishes enough peers. During this phase, you will notice lines in the logs that display, for example:
If everything is functioning properly, the peercount number on the same line will gradually increase, and eventually, Geth will no longer report "Looking for peers". This is the expected behavior and indicates that the syncing process is progressing smoothly.
However, there are some common issues that may arise:
Geth cannot sync until it is connected to a sufficient number of peers.
If you consistently see the "Looking for peers" message without an increase in the peercount value, it indicates that Geth is unable to establish connections with peers on the network. The most common cause of this issue is a network problem. Resolving this may require adjusting certain settings, such as "enabling uPnP" or "opening port 30303" on your router. You can refer to this guide on Opening network ports.
Geth cannot sync until the connected Consensus Client is synced.
If you have not installed and synchronized a Consensus Client (such as Nimbus, Teku, or Prysm), Geth cannot sync. If you encounter the following messages:
It is likely that either you have not installed and synced a Consensus Client, or there is an issue with your existing Consensus Client. Check the logs of the Consensus Client to determine the cause of the problem.
Sometimes, there may be a communication problem between Geth and the Consensus Client. Restarting both can often resolve this. Begin by restarting the Consensus Client, and then restart Geth. You can find the Restart button on the respective client's Management Page.
When initially starting Geth, you may see this error on the Home Page:
Also during the initial phase, you may get an "NaN%" on the progress bar:
These warnings may suggest a potential issue with Geth, but they could also be a non-issue. To determine the cause, it is advisable to examine the Geth logs for any problems related to starting Geth, finding peers, or connecting to the beacon chain. By addressing any identified issues, the errors and warnings should no longer persist.
If the Geth logs appear normal and do not indicate any problems, it is possible that the anomalies are transient and will resolve themselves without further intervention.
Once Geth successfully establishes connections with both the peer network and the Consensus Client, the main syncing process begins. By default, Geth utilizes "snap sync". In snap sync, Geth starts from a relatively recent block and syncs from there to the head of the chain.
During this phase, you should observe messages in the logs similar to this:
To locate the eta value, scroll to the far-right end of the line. This value represents the estimated time required for this phase to complete and should gradually decrease as the process advances.
After Geth finishes syncing to the recent blocks, it proceeds to download the main bulk of the blockchain data.
Simultaneously, Geth initiates state-sync. During state-sync, Geth downloads portions of a "state trie" and reconstructs the complete information locally. A "state trie" is a data structure used in Ethereum to efficiently and securely store account balances, contract code, and other essential information related to the Ethereum network.
This is the most resource-intensive phase of the syncing process. You can observe that the CPU usage remains very high during this time. Make sure you use a fan to keep your AVADO device cool.
During this phase, you should observe messages in the logs similar to:
Notice that the percentage numbers should steadily increase. The eta, located at the far-right end of the log line, represents the estimated time required for this phase to complete and should gradually decrease as the process advances. Additionally, on the Home Page, you should observe that the progress bar indicates a percentage that is steadily progressing.
During the snap-sync process, the state download phase attempts to reconstruct the blockchain state. However, as the blockchain continues to progress, it invalidates some of the regenerated state data. To address this, a "healing" phase is necessary to reconcile the continual changes in the state.
When this healing phase occurs, Geth reports the following in the log:
Due to the dynamic nature of state changes, it is not possible to estimate the duration of the state heal. If Geth consistently reports "state healing in progress" during and after the state download process, it indicates that the state heal has not yet completed. Users are advised to patiently wait for it to finish.
Syncing no longer get "stuck" at 99%.
In earlier versions of Geth, some users experienced a situation where the syncing process appeared to be "stuck" at 99% for an extended period. This was primarily due to the state regeneration and state heal steps which which could take a long time.
However, newer versions of Geth have significantly improved this user experience. With the introduction of the "path state scheme" (--state.scheme=path) option, syncing should no longer get stuck at 99%. This option has become standard staring from Geth v1.14.0 (AVADO Geth package 10.0.64).
Once the state heal process is complete, your node will be fully synced. If you have already activated your validators, you should notice that they start attesting.
However, for a few hours, Geth may still appear busy, and the CPU usage may remain high. This may even cause your validators to miss attestations if they are already activated.
This is because Geth is engaged in generating state snapshots. State snapshots are a feature introduced in Geth to improve the efficiency and speed of syncing Ethereum nodes. Generating state snapshots is a resource intensive task that requires significant computational power and disk space. However, once the state snapshot is generated, it can be stored and shared among multiple nodes, significantly reducing the time and resources needed for syncing.
During this phase, you will observe the following messages in the logs repeatedly:
This behavior is normal. Geth attempts to perform snapshot generation between the arrival of new blocks to minimize interruption. Allow Geth to complete the snapshot generation process. This will take several hours to settle down.
Remember to ensure that you continue using a fan to keep your AVADO device cool.
Throughout the syncing process, you may encounter the following warnings. It is important to note that these warnings are harmless and do not indicate any significant issues:
Forkchoice requested sync to new head
Snapshot extension registration failed
Pivot seemingly stale, moving
Unexpected account range packet
Unexpected storage ranges packet
Unexpected bytecode packet
Unexpected trienode heal packet
Expired request does not exist
Pending request exists for joining peer
Peer stalling, dropping
Ignoring payload while snap syncing
Dropping unsynced node during sync
During normal operations, you may also come across the following warnings. These warnings are expected and do not indicate any cause for concern:
Announced transaction size mismatch
Peer delivering stale transactions
Blob transaction swapped out by signer
Once your Geth client is fully synced and operating normally, the logs should appear similar to the following:
The shows that Geth continuously imports new chain segments and updates the chain head to maintain the latest state of the blockchain.
The syncing process can be time-consuming, and the duration will primarily depend on the specifications of your hardware and the speed of your network connection.
Drawing from a recent experience (May 2024), using an AVADO i7/4TB device with a 1 Gbps fiber connection, it took approximately 11.5 hours from the start of the process until the Consensus Client was able to initiate attestations. The complete sync took about 16 hours.
A more detailed breakdown is as follows. This is for reference only. Your experience may be different.
00:00
Start of process "Looking for peers ..." "Syncing beacon headers ..."
00:30
Finished "syncing beacon headers" Syncing: chain download in progress Syncing: state download in progress CPU usage stays high throughout (100%)
07:30
Finished "state download" Syncing: chain download in progress ... Syncing: state healing in progress ... (intermittently)
11:30
Finished "chain download" Rebuilding state snapshot
Generating state snapshot (initial eta ~4h)
Aborting/Resuming state snapshot generation
Upgrading chain index
11:30
Consensus Client started attestations
16:00
Finished "generating state snapshot"
Remark: AVADO Geth package 10.0.64 (based on Geth v1.14.0), default to "path state scheme"
