Polkadot validator misses attestations with no error logs until a restart

[kukabi]

I set the session keys of Polkadot validator 123kFHVth2udmM79sn3RPQ81HukrQWCxA1vmTWkGHSvkR4k1 in session 12093 from one active node to another running node with healthy status, syncing, with the --validator flag, behind a well-configured network that had been tested before. The node instance was supposed to take over in session 12095, yet it started missing attestations at the beginning of the session, with no indication of an error in the logs. Pre-restart logs, after the beginning of the session, can be founds here: pre_restart.log

I restarted the node after noticing the missed attestations. It started submitting attestations after the restart, giving a number of error logs similar to the following:

2025-12-02 15:50:24 Cluster has too many pending statements, something wrong with our connection to our group peers
				Restart might be needed if validator gets 0 backing rewards for more than 3-4 consecutive sessions pending_statements={ValidatorIndex(529): {(ValidatorIndex(529), CompactStatement::Seconded(0x5471da2120a51e38fd7fb652d24305499396722a21bad6cb85c685f38938651c)), (ValidatorIndex(529), CompactStatement::Seconded(0xdfecdd79a1cc9fcafc464d40f805d879aa30e891e4a54f66a36dbdff5895bb7f))}, ValidatorIndex(527): {(ValidatorIndex(529), CompactStatement::Seconded(0xdfecdd79a1cc9fcafc464d40f805d879aa30e891e4a54f66a36dbdff5895bb7f)), (ValidatorIndex(529), CompactStatement::Seconded(0x5471da2120a51e38fd7fb652d24305499396722a21bad6cb85c685f38938651c))}, ValidatorIndex(528): {(ValidatorIndex(529), CompactStatement::Seconded(0xdfecdd79a1cc9fcafc464d40f805d879aa30e891e4a54f66a36dbdff5895bb7f)), (ValidatorIndex(529), CompactStatement::Seconded(0x5471da2120a51e38fd7fb652d24305499396722a21bad6cb85c685f38938651c))}, ValidatorIndex(525): {(ValidatorIndex(529), CompactStatement::Seconded(0xdfecdd79a1cc9fcafc464d40f805d879aa30e891e4a54f66a36dbdff5895bb7f)), (ValidatorIndex(529), CompactStatement::Seconded(0x5471da2120a51e38fd7fb652d24305499396722a21bad6cb85c685f38938651c))}, ValidatorIndex(526): {(ValidatorIndex(529), CompactStatement::Seconded(0x5471da2120a51e38fd7fb652d24305499396722a21bad6cb85c685f38938651c)), (ValidatorIndex(529), CompactStatement::Seconded(0xdfecdd79a1cc9fcafc464d40f805d879aa30e891e4a54f66a36dbdff5895bb7f))}} parent_hash=0x2228e9b450570432932cf327ab2914a6bca89ccf27de2b1e3aaf33ed85a44ced

No other measure was taken other than the restart, and this same node instance with the same configuration had been in the active set multiple times prior to this session key rotation. Post-restart logs can be found here: post_restart.log

There seems to be no indication of an error in the pre-restart logs, when the node was already in the active set. I cannot tell whether the node was aware of the issue pre-restart, but identifying the root cause would be possible if the logs had an indication/explanation for the condition causing the issue.

[sandreim]

CC @alexggh

[sandreim]

Looks like you have issues with connectivity: Connectivity seems low, we are only connected to 48% of available validators (see debug logs for details), if this persists more than a session action needs to be taken session_index=12095

But, improved a bit after restart: 2025-12-02 15:57:13 Connectivity seems low, we are only connected to 78% of available validators (see debug logs for details), if this persists more than a session action needs to be taken session_index=12095

[alexggh]

First, Cluster has too many pending statements, something wrong with our connection and Connectivity seems low, just tells you that your node is not properly connected after restart, it should be transient isn't it ?

Regarding on why it wasn't working properly before restart, could you describe your setup in more detail ?

I set the session keys of Polkadot validator 123kFHVth2udmM79sn3RPQ81HukrQWCxA1vmTWkGHSvkR4k1 in session 12093 from one active node to another running node with healthy status, syncing, with the --validator flag, behind a well-configured network that had been tested before. The node instance was supposed to take over in session 12095.

How is the instance handover suppose to work between 12093 and 12095 ? I would expect that once you set the keys the new node will start behaving like it is a validator, so no you would have 2 nodes with the same keys, which is problematic and should be done.

[kukabi]

Thanks @sandreim, I'm just noticing the logs you pointed at, sorry. It's not clear to me how the issue resolves under the same network configuration after a restart, but I'm going to look into the router and other network configuration. I will close the issue for now, any other pointers at a solution are much welcome.

[kukabi]

@alexggh the key rotation is done in a proper way. There are two nodes, one is active, the keys get rotated on the second node instance, and the validator's keys get set on-chain with these new keys, and 2 sessions later this second node picks up. In other words, the session keys are not shared among these two nodes. It seems to be a connectivity issue @sandreim pointed out, but I currently don't have a hint at what could be causing it at the network level, and how it can recover after only a restart. I will look further into it. Thanks.

[lexnv]

@kukabi, which version are you using for the binary?

[kukabi]

I'm reopening the issue after comments from two more validators (Gatotech and Saxemberg) stating that they are also observing non-self-healing issues with networking, especially in the fringes, away from Europe. Please view the Polkadot Validator Lounge channel on Element for the discussion.

@lexnv I'm copying my response from Element:

The nodes have been running non-stop, latest version v1.20.2. Binary upgrades were done quite a while ago, a couple of days after the release 2 weeks ago, and the node wasn't stopped ever since.

[kukabi]

Please also see #9999 for a similar non-self-healing behaviour.

[lexnv]

I set the session keys of Polkadot validator 123kFHVth2udmM79sn3RPQ81HukrQWCxA1vmTWkGHSvkR4k1 in session 12093 from one active node to another running node with healthy status

This might cause issues since it's not a common use case. When you migrate the keys, and the second PeerID "takes over", are you effectively running from a different IP address and a different port? If that's the case:

• the new addresses might need time to propagate through the network
• peers from the network might ban your newer node since they might have traces of the older states (ie, maybe on the sync protocol or similar)

syncing, with the --validator flag, behind a well-configured network that had been tested before.

You might be dealing with a local network configuration issue. Are you placing the nodes behind a load balancer?

If you are running outside of this setup, do you see any improvements?

[kukabi]

@lexnv this same key rotation method has been applied multiple times on multiple nodes and stashes without errors. Actually just today the second validator was rotated in the same fashion, and for months this is the method we've been applying without issues except for this case today. As far as I know this is the recommended method to change the session keys of a validator, and if this method is bound to cause issues, then I don't know what can be a better way to change the session keys of a validator without a restart on either machine. Preventing a restart guarantees continued attestations from the validator.

You might be dealing with a local network configuration issue. Are you placing the nodes behind a load balancer?

Nope, there are no load balancers. All validators are behind an enterprise-level router/firewall with a 3G symmetrical uplink. Like I mentioned, we've been applying the same method successfully without any issues, but these two issues seem to be pointing at a self-healing issue because there is no configuration change other than the restart itself. The nodes kept syncing successfully in both cases, so there seem to be no major connectivity problem.

Just a surface-level look at the issue gives the impression that there happens a temporary glitch in the P2P layer, and the networking component is just not able to recover from it, and the issue just keeps growing or propagating until a restart.

As a further note, there is one Kusama and one Polkadot node on each server as per Decentralized Nodes rules, and the Kusama node on the server kept running just fine in both cases.

Please also view the comments from Gatotech and Saxemberg in the Polkadot Validator Lounge channel, they are pointing at a similar experience.

Thanks.

[kukabi]

I'm attaching comments from Gatotech and Saxemberg.

[m-saxemberg]

We have experienced this exact same issue on a live validator today. Let us know what CLI flags can be used to catch any other useful logs other than the ones that have already appeared. Also hopefully to catch other non-self-healing / connectivity / network issues.

non-self-healing issues with networking, especially in the fringes, away from Europe

today's issue on our side appeared on an European colocation machine which is a first for this kind of issue which we have had mostly on Johannesburg and Bangkok nodes.

[florentina57]

I just experienced a similar situation with one of my Polkadot validators, located in Colombia. I initiated the rotation of session keys on session 12108 and on 12110 they took effect. The new location is in Europe and since the beginning of the session (12110), every 10 mins the same log message as @kukabi at this point the node was not missing attestations though.

My server has the same setup as well, 1 Polkadot and 1 Kusama validator. The Kusama validator did not experience any issues. (Binary: latest version v1.20.2)

2025-12-04 23:43:20 Connectivity seems low, we are only connected to 82% of available validators (see debug logs for details), if this persists more than a session action needs to be taken session_index=12110
2025-12-04 23:53:20 Connectivity seems low, we are only connected to 81% of available validators (see debug logs for details), if this persists more than a session action needs to be taken session_index=12110
2025-12-05 00:03:20 Connectivity seems low, we are only connected to 81% of available validators (see debug logs for details), if this persists more than a session action needs to be taken session_index=12110
2025-12-05 00:13:20 Connectivity seems low, we are only connected to 81% of available validators (see debug logs for details), if this persists more than a session action needs to be taken session_index=12110
2025-12-05 00:23:20 Connectivity seems low, we are only connected to 81% of available validators (see debug logs for details), if this persists more than a session action needs to be taken session_index=12110
2025-12-05 00:33:20 Connectivity seems low, we are only connected to 81% of available validators (see debug logs for details), if this persists more than a session action needs to be taken session_index=12110

I have restarted the node and connectivity errors persist.

2025-12-05 00:51:38 Connectivity seems low, we are only connected to 82% of available validators (see debug logs for details), if this persists more than a session action needs to be taken session_index=12110
2025-12-05 01:01:38 Connectivity seems low, we are only connected to 83% of available validators (see debug logs for details), if this persists more than a session action needs to be taken session_index=12110

Here are the logs from before and after the session key change in case they can be helpful:

validandina-lgnbog-memory.log