Geth

Understanding the Geth Syncing Process

Syncing is a multi-stage process that Geth undergoes to download, verify, and reconstruct the complete global state of Ethereum. Initially, Geth connects to other participating nodes, known as peers, on the network to exchange data. It then proceeds to download chain and state data, and rebuilds a local representation of the entire Ethereum state. However, the blockchain is also progressing at the same time, so Geth will also need to catch up to the current state.

The following sections will outline the different stages of syncing and provide insights into what you can expect to observe in the logs. This knowledge will enable you to quickly assess whether the syncing process is proceeding smoothly or if any issues require your attention.

Monitoring the Syncing Process

To monitor the syncing progress, you can refer to the Home Page, specifically the Chains - Ethereum section. Keep an eye on the status bar, which will gradually advance towards 100%.

Once it reaches 100% and the progress bar turns green, your Geth client is fully synced, indicating that you are ready to perform your node duties on the Ethereum network.

To ensure a smooth syncing experience, we recommend familiarizing yourself with the sync process and actively monitoring the Logs. By doing so, you can gain confidence in determining whether the syncing process is progressing smoothly or if any issues require your attention.

To access the Logs, go to the Management Page for Geth. You can navigate to this page from the My DApps menu. The Management Page offers a range of controls and settings to manage Geth.

On the Management Page, simply scroll down to the bottom to find the logs.

Stage 1. Startup

When Geth starts up, it undergoes a series of configuration details and status checks. It then initiates the process of connecting to other nodes on the network until it establishes enough peers. During this phase, you will notice lines in the logs that display, for example:

Looking for peers                       peercount=2 ...

If everything is functioning properly, the peercount number on the same line will gradually increase, and eventually, Geth will no longer report "Looking for peers". This is the expected behavior and indicates that the syncing process is progressing smoothly.

However, there are some common issues that may arise:

Geth unable to find peers

Geth cannot sync until it is connected to a sufficient number of peers.

If you consistently see the "Looking for peers" message without an increase in the peercount value, it indicates that Geth is unable to establish connections with peers on the network. The most common cause of this issue is a network problem. Resolving this may require adjusting certain settings, such as "enabling uPnP" or "opening port 30303" on your router. You can refer to this guide on Opening network ports.

No beacon client detected or beacon chain not operational

Geth cannot sync until the connected Consensus Client is synced.

If you have not installed and synchronized a Consensus Client (such as Nimbus, Teku, or Prysm), Geth cannot sync. If you encounter the following messages:

Post-merge network, but no beacon client seen. Please launch one to follow the chain!
Beacon client online, but never received consensus updates. Please ensure your beacon client is operational to follow the chain!

It is likely that either you have not installed and synced a Consensus Client, or there is an issue with your existing Consensus Client. Check the logs of the Consensus Client to determine the cause of the problem.

Sometimes, there may be a communication problem between Geth and the Consensus Client. Restarting both can often resolve this. Begin by restarting the Consensus Client, and then restart Geth. You can find the Restart button on the respective client's Management Page.

Connection refused, or NaN%

When initially starting Geth, you may see this error on the Home Page:

Also during the initial phase, you may get an "NaN%" on the progress bar:

These warnings may suggest a potential issue with Geth, but they could also be a non-issue. To determine the cause, it is advisable to examine the Geth logs for any problems related to starting Geth, finding peers, or connecting to the beacon chain. By addressing any identified issues, the errors and warnings should no longer persist.

If the Geth logs appear normal and do not indicate any problems, it is possible that the anomalies are transient and will resolve themselves without further intervention.

Stage 2. Syncing Beacon Headers

Once Geth successfully establishes connections with both the peer network and the Consensus Client, the main syncing process begins. By default, Geth utilizes "snap sync". In snap sync, Geth starts from a relatively recent block and syncs from there to the head of the chain.

During this phase, you should observe messages in the logs similar to this:

Syncing beacon headers                    ... eta=1h20m

To locate the eta value, scroll to the far-right end of the line. This value represents the estimated time required for this phase to complete and should gradually decrease as the process advances.

Stage 3. Downloading Chain and State Data

After Geth finishes syncing to the recent blocks, it proceeds to download the main bulk of the blockchain data.

Simultaneously, Geth initiates state-sync. During state-sync, Geth downloads portions of a "state trie" and reconstructs the complete information locally. A "state trie" is a data structure used in Ethereum to efficiently and securely store account balances, contract code, and other essential information related to the Ethereum network.

This is the most resource-intensive phase of the syncing process. You can observe that the CPU usage remains very high during this time. Make sure you use a fan to keep your AVADO device cool.

During this phase, you should observe messages in the logs similar to:

Syncing: chain download in progress        synced=76.37% ...
Syncing: state download in progress        synced=25.05% ...

Notice that the percentage numbers should steadily increase. The eta, located at the far-right end of the log line, represents the estimated time required for this phase to complete and should gradually decrease as the process advances. Additionally, on the Home Page, you should observe that the progress bar indicates a percentage that is steadily progressing.

Stage 4. State Heal

During the snap-sync process, the state download phase attempts to reconstruct the blockchain state. However, as the blockchain continues to progress, it invalidates some of the regenerated state data. To address this, a "healing" phase is necessary to reconcile the continual changes in the state.

When this healing phase occurs, Geth reports the following in the log:

Syncing: state healing in progress

Due to the dynamic nature of state changes, it is not possible to estimate the duration of the state heal. If Geth consistently reports "state healing in progress" during and after the state download process, it indicates that the state heal has not yet completed. Users are advised to patiently wait for it to finish.

Syncing no longer get "stuck" at 99%.

In earlier versions of Geth, some users experienced a situation where the syncing process appeared to be "stuck" at 99% for an extended period. This was primarily due to the state regeneration and state heal steps which which could take a long time.

However, newer versions of Geth have significantly improved this user experience. With the introduction of the "path state scheme" (--state.scheme=path) option, syncing should no longer get stuck at 99%. This option has become standard staring from Geth v1.14.0 (AVADO Geth package 10.0.64).

Stage 5. Generating State Snapshots

Once the state heal process is complete, your node will be fully synced. If you have already activated your validators, you should notice that they start attesting.

However, for a few hours, Geth may still appear busy, and the CPU usage may remain high. This may even cause your validators to miss attestations if they are already activated.

This is because Geth is engaged in generating state snapshots. State snapshots are a feature introduced in Geth to improve the efficiency and speed of syncing Ethereum nodes. Generating state snapshots is a resource intensive task that requires significant computational power and disk space. However, once the state snapshot is generated, it can be stored and shared among multiple nodes, significantly reducing the time and resources needed for syncing.

During this phase, you will observe the following messages in the logs repeatedly:

Generating state snapshot
Aborting state snapshot generation
Resuming state snapshot generation

This behavior is normal. Geth attempts to perform snapshot generation between the arrival of new blocks to minimize interruption. Allow Geth to complete the snapshot generation process.

Remember to ensure that you continue using a fan to keep your AVADO device cool.

Common Warnings

Throughout the syncing process, you may encounter the following warnings. It is important to note that these warnings are harmless and do not indicate any significant issues:

  • Forkchoice requested sync to new head

  • Snapshot extension registration failed

  • Pivot seemingly stale, moving

  • Unexpected account range packet

  • Unexpected storage ranges packet

  • Unexpected bytecode packet

  • Unexpected trienode heal packet

  • Expired request does not exist

  • Pending request exists for joining peer

  • Peer stalling, dropping

  • Ignoring payload while snap syncing

  • Dropping unsynced node during sync

During normal operations, you may also come across the following warnings. These warnings are expected and do not indicate any cause for concern:

  • Announced transaction size mismatch

  • Peer delivering stale transactions

Normal Operations

Once your Geth client is fully synced and operating normally, the logs should appear similar to the following:

The shows that Geth continuously imports new chain segments and updates the chain head to maintain the latest state of the blockchain.

How long does it take?

The syncing process can be time-consuming, and the duration will primarily depend on the specifications of your hardware and the speed of your network connection.

Drawing from a recent experience (Dec 2023), using an AVADO i7/2TB device with a 1 Gbps fiber connection, it took approximately 13.5 hours from the start of the process until the Consensus Client was able to initiate attestations. The complete sync took about 16 hours.

A more detailed breakdown is as follows. This is for reference only. Your experience may be different.

Elapsed TimeActions

00:00

Start of process "Looking for peers ..." "Syncing beacon headers ..." (initial eta 1h20m)

00:25

Finished "syncing beacon headers" (actual 0hr25m) Syncing: chain download in progress (initial eta >3h) Syncing: state download in progress (initial eta >22h) CPU usage stays high throughout (100%)

11:20

Finished "state download" Syncing: chain download in progress ... (continued at >90%) Syncing: state healing in progress ... for many minutes at first, then intermittently for ~1m each time

13:20

Finished "chain download" Rebuilding state snapshot

Generating state snapshot (initial eta ~4h)

Aborting/Resuming state snapshot generation

Upgrading chain index

13:30

Consensus Client started attestations Geth database volume: 937.6 GB

16:00

Finished "generating state snapshot"

Remark: used the --state.scheme=path option for Geth

Last updated