What happened
Over the past two rotations, three validators have been sending our nodes deliberately malformed keygen messages. We believe this to have been an attempted exploit of a live vulnerability, but we caught it and acted before the attacker was able to pull it off. No funds were lost, no keys were exposed, and swaps and every other service kept running the whole time.
We have paused rotations while we deal with it. That pause does double duty, because the three accounts can only move their funds while rotations are running, so with rotations held, the funds stay put and we get the time to sort out the rest safely. We are shipping a fix now and asking every operator to upgrade.
The upgrade now
Please upgrade to the new build (2.2.11) as soon as the binaries land. It closes off the route these validators were probing, so getting the whole set upgraded is how we make sure the suspected attack vector is fully neutralised. Rotations can only be safely re-enabled once the majority of nodes are upgraded. We will also wait for the outcome of this proposal, and if it passes, the freeze should be done before rotations come back; otherwise, the accounts could start withdrawing their funds.
The decision we want the node operators to make with us
The real question is what to do about the three accounts, and rather than settle it unilaterally, we want the operators who run the network to have a real say in how it handles a bad actor.
We think the right move is to move their FLIP into an on-chain reserve and bar them from keygen, so this operator cannot line up another attempt. Freezing a validator's balance is a serious step, though, and a decision this consequential should carry the weight of the operators behind it; that's why we are putting up this proposal.
Here is how the vote works. We will leave this proposal open for 48 hours, which gives node operators enough time to weigh in, and a simple yes in this chat is all we need from you. We will act as soon as we have enough operator support to justify it based on the responses here, or when the 48 hours are up. We are confident this is the right call, and we have set out the evidence below so you can weigh it for yourselves. Our hope is that once you have looked at it, you will see what we see and back the proposal alongside us.
Why a reserve, and not just blocking withdrawals? If the operator keeps their FLIP and only loses the ability to cash out, they may decide they have nothing left to lose and spend it trying to spam or disrupt the chain to pressure us. Moving the balance into a reserve takes that option off the table.
And none of it is permanent. The freeze is reversible and can be undone if the network decides so later. The funds come back the same way they went in, by consensus.
What they were actually doing
For the past few epochs, since before the releases of versions 2.2.8 and 2.2.9, we have been monitoring the behaviour of three suspicious nodes that silently joined the network around the 17th/18th of August.
Starting with the handover ceremony in the previous epoch, ceremony 474, each of the three validators began sending our nodes a run of malformed keygen messages, all carrying ceremony IDs that sat far outside any valid window. On their own, those IDs are inert. A message with an out-of-window ID gets dropped. For this reason, besides some log messages, there was little sign of anything suspicious.
Once we pulled the actual IDs apart, we found a pattern: what looks like a fixed campaign marker, a sender number, and a case number running from 0 to 47. This indicated to us that the messages were being deliberately tampered with.
Further to this, some community members reported their nodes falling out of the authority set, having failed to complete key handover. An analysis of their logs revealed the same pattern of invalid messages being received. In this case, the messages seem to have overloaded the receiving engine’s message processing queue, preventing it from receiving enough valid messages from the network.
At this point it became unambiguously clear that the suspicious messages were some attempt at exploiting a vulnerability in the Key Handover ceremony. We developed the following theory based on our observations of the nodes’ behaviour and analysis of the logs:
The message spam is designed to overload the receiving peer such that they cannot process enough legitimate messages. This causes the multisig protocol to proceed to a verification stage whereby participants can be required to justify their honest participation by sending a fragment of their key share as proof.
We believe that the attacker was trying to provoke a scenario whereby honest participants “over-shared” their fragments. In short, if a participant shares too many fragments, these can be used to reconstruct the full key.
The multisig protocol already guarded against this by ensuring that individual participants would never share enough information to reconstruct the full key. However, there is a scenario in which a malicious participant may be able to combine these with fragments obtained during a previous round of the ceremony to reach the reconstruction threshold.
The more validators the attacker controls, the fewer additional shares they need. With three validators in a set of 150, the attacker would need 97 participants to share their fragments during verification.
The fix
We have tightened the criterion for over-sharing. Rather than limiting sharing at the threshold number, we limit at half of this. This considerably raises the malicious validator count at which the attack becomes feasible.
In addition, we have removed a performance leak in the treatment of invalid messages, making it harder to abuse this code path to stall the message queue.
The trade-off is liveness: a malicious actor can now more easily force ceremonies to abort and retry.
What is the evidence?
The nodes are sending invalid messages to keygen participants, with invalid ceremony IDs. This in itself proves that they are running custom engine software.
The ceremony IDs follow patterns that suggest deliberate tampering rather than random data corruption.
The messages targeted a code path that allowed unbounded CPU work due to an unchecked decoding of a potentially unbounded data structure.
This was observed in logs from our own engines and those of other operators, so the symptoms are not confined to a single operator.
What was at risk?
It appears that the attacker was targeting Bitcoin handover ceremonies in an attempt to hijack the aggregate key. If they had succeeded, the contents of the Bitcoin vault would likely have been drained.
The three accounts
The behaviour came from these validators, starting around ceremony 474:
cFHwLgJSD9LnmWKveHpnmGKLowHFYwhiUVcTG15Lj2eDwpyan
cFJCSRpLfjmENqUmVLkGUNDj99QXFnqgWMurNCMZwkwhReG6K
cFLMxH46aay2vTqZwUDgYHwTGpM38ccxunFZFNkEs4D1m1gKs
If you are the operator behind these accounts and there is an honest explanation for what we have seen, we want to hear it. Speak up here or reach out to the team
Thanks to everyone across the community and the team who helped flag this and run it down early.
