Repository navigation
Conversation
EventQueueSerWrapper cast the queue length to u16, so a queue with 65,536 or more events was restored truncated on restart (65,536 events came back empty). Write and read the count as CollectionLength, which keeps the two-byte encoding below 65,535 and uses the 0xffff extension above it, and bound the reader's preallocation. Add round-trip tests at 65,534, 65,535, 65,536 and 65,540 events and a test that small queues keep the legacy encoding. Written with AI assistance (Claude); reviewed and tested locally. Fixes lightningdevkit#1138. Co-authored-by: Claude <noreply@anthropic.com>
|
I've assigned @tnull as a reviewer! |
| let mut queue = VecDeque::with_capacity(len as usize); | ||
| for _ in 0..len { | ||
| let len: CollectionLength = Readable::read(reader)?; | ||
| // Bound the preallocation, as the length is read from persisted data. |
There was a problem hiding this comment.
Can you comment on why this is useful? If the data is there, we will read all the entries, and hence costly re-allocate?
There was a problem hiding this comment.
It's only there for a corrupted or truncated store. With the old u16 count the worst case was 65,535 slots, but CollectionLength can decode to any u64, so a bad length would make with_capacity abort the process before the first event read fails with a DecodeError. The Vec readers in lightning::util::ser bound it the same way (min(len, MAX_BUF_SIZE / size_of::<T>())). For a valid queue longer than the cap it only costs the usual amortized growth. Happy to switch to that MAX_BUF_SIZE form to match LDK, or to drop the cap if you'd rather keep it simple.
There was a problem hiding this comment.
Hmm, the thing is if we encounter corrupted data here there is no way to recover anyways. I guess we can bound to the minimum of actual len and u16::MAX, or just leave it out. In general, we don't expect users to ever hit an event queue of 1024 events (in fact if they do, that's very bad), but if they ever did adding additional re-allocations on top is probably not great.
3 participants
Fixes #1138.
EventQueueSerWrapperwrote the queue length asu16, while still writing every event, soEventQueueDeserWrapperrestored a truncated queue: 65,536 persisted events came back empty and 65,540 came back as 4.This switches both wrappers to
lightning::util::ser::CollectionLength, as suggested in the issue:0xffffmarker followed by the remainder;Tests (in
event::tests):event_queue_round_trips_around_u16_max: full round trip at 65,534, 65,535, 65,536 and 65,540 events. It fails onmain(65,536 events decode as 0) and passes with this change;event_queue_keeps_legacy_encoding_below_u16_max: a small queue encodes to the same bytes as the legacyu16format and decodes from them.cargo test --libpasses locally (203 tests); I didn't run the integration tests, which need bitcoind/electrs.AI disclosure: written with AI assistance (Claude); I reviewed the change and ran the tests above.
🤖 Generated with Claude Code