netsync: Detect and log db corruption on headers. #3602
Merged
+16
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Prior to the introduction of headers-first semantics, all header processing happened in memory and therefore had a smaller set of possible errors, so the error logging for them was purely for the purposes of debug and logged accordingly at the debug level. However, now that headers-first semantics are in use, the headers are saved to the database which means database corruption is a possible error. As a result, it was observed that database corruption was being hidden behind debug logs when it really shouldn't be. This resolves that by updating the header processing path to use the same error detection logic as the block processing path to ensure any errors that are not regular rule errors are not hidden behind debug logs and also to explicitly log a critical failure in the event of database corruption.
JoeGruffins
approved these changes
Jan 24, 2026
Member
JoeGruffins
left a comment
JoeGruffins
left a comment
There was a problem hiding this comment.
I see it on the affected device.
2026-01-24 05:49:46.724 [INF] SYNC: Processed 18000 headers in the last 10.23s (progress 10.88%)
2026-01-24 05:49:57.431 [INF] SYNC: Processed 16000 headers in the last 10.71s (progress 11.74%)
2026-01-24 05:50:07.163 [ERR] SYNC: Failed to process block header 0000000000082ad5a01631416de9aeab0417c2c2877174c233e41a21bb800bd3 from peer 31.97.231.224:19108 (outbound): failed to open ldb transaction: leveldb/table: corruption on data-block (pos=288195): checksum mismatch, want=0xa9bfe451 got=0x665a6dcc [file=000383.ldb] -- disconnecting
2026-01-24 05:50:07.163 [ERR] SYNC: Criticial failure: failed to open ldb transaction: leveldb/table: corruption on data-block (pos=288195): checksum mismatch, want=0xa9bfe451 got=0x665a6dcc [file=000383.ldb]
2026-01-24 05:50:07.163 [INF] SYNC: Syncing headers to block height 1834175 from peer 195.49.75.206:19108 (outbound)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prior to the introduction of headers-first semantics, all header processing happened in memory and therefore had a smaller set of possible errors, so the error logging for them was purely for the purposes of debug and logged accordingly at the debug level.
However, now that headers-first semantics are in use, the headers are saved to the database which means database corruption is a possible error. As a result, it was observed that database corruption was being hidden behind debug logs when it really shouldn't be.
This resolves that by updating the header processing path to use the same error detection logic as the block processing path to ensure any errors that are not regular rule errors are not hidden behind debug logs and also to explicitly log a critical failure in the event of database corruption.