Lost ~45M sats to a power outage during active routing — is recovery truly impossible? Any help deeply appreciated 🙏

[terryas7777-dot]

Summary ( — LND v0.18.5-beta)

LND on an Ubuntu VM (UTM) on a Mac mini, ~4 BTC routing node, no UPS at the time. A power outage caused a hard shutdown while a significant amount was in-flight as HTLCs. I believe I lost ~45M sats and would be grateful for a sanity check on whether any recovery is
possible.

What happened

• 2026-05-27: power outage → hard shutdown while ~47M sats were in-flight (active forwarding).
• On restart, one channel reported local data loss:
  sync failed with local data loss: remote believes our tail height is 10024, while we have 10022
• The remote force-closed it (DLP). The commitment output (~69M sats) was swept fine.

The loss

Using dated balance snapshots, I localized a ~45M sat shortfall to the 5/27–5/31 window, which matches the ~47M that was unsettled at the moment of the outage.

What I checked

• On-chain reconciles to the sat — no unexplained on-chain outflows
• Exchange round-trip (Lightning-in / on-chain-out) deposits = withdrawals, to the sat
• The force-close tx had no HTLC outputs (only commit + anchor)
• No htlc-success / htlc-timeout / contest resolvers in the logs for that period → the in-flight HTLCs appear to have resolved off-chain
• chantools sweepremoteclosed found only ~400k sats of unswept outputs

My current read

Forwards where the outgoing leg settled but the incoming leg failed/expired during the downtime → off-chain loss, not recoverable. I hope I'm wrong.

Questions

1. Is this analysis correct, or is there any recovery path I'm missing (chantools options, peer cooperation, anything at all)?
2. Could the 2-state local data loss itself account for funds beyond the in-flight HTLCs?
3. Beyond UPS + graceful shutdown, anything else to learn for prevention?

---

This is a very significant amount for me, and honestly I've barely slept since I found out. I've spent months building and running this node. If anyone has experience recovering from this kind of outage / data-loss situation — or knows someone who does — I would be
deeply grateful for any pointer, however small. Even a definitive "it's gone" from an expert would help me find peace. Please, if you know anything, reach out. Thank you for reading. 🙏

[yyforyongyu]

It would be better if you could share your logs for analysis, as we need to understand what happened in the two states between 10022 and 10024. In addition, which db are you using? Maybe share your configs too, via DMs or email

[terryas7777-dot]

Thank you so much for looking into this! 🙏

DB backend: bbolt (default — no db.backend in lnd.conf). channel.db ≈ 865 MB.
LND version: v0.18.5-beta

A key detail from the logs: there were two restarts on the outage day:

• 14:54 (power-outage recovery): channel reestablished fine — local commitment height 10012 vs remote 10006 (local ahead, no data loss). The channel then kept routing normally, advancing 10012 → 10022 (local bandwidth grew ~38.2M → ~68.7M sat).
• 17:49 (a second restart ~3h later): this is where data loss was detected —
  [ERR] LNWL: sync failed with local data loss: remote believes our tail height is 10024, while we have 10022!
  So the node was missing the last 2 commitment states (10023, 10024) after the 17:49 restart. The channel was force-closed by the remote (DLP) and I swept the to_remote output ≈ 68,960,965 sat.

Separately, using dated balance snapshots I traced an unexplained ~45M sat shortfall to the 5/27–5/31 window (not this channel's commit balance, which seems to have recovered). I suspect in-flight HTLC forwards that resolved unfavorably during the multiple restarts, but I'd love your
read.

I can send the full logs + lnd.conf privately. What's the best email/handle to reach you? Thank you again 🙏

Thanks a lot!!!

[yyforyongyu]

I've created this PR: lightninglabs/chantools#241
You can try to sweep it via chantools sweephtlc.

[terryas7777-dot]

Thank you so much. I was able to recover the funds I thought were lost forever. I truly couldn't have done it without you.

[terryas7777-dot]

Hi Yong, Thanks again for your help back in June with the power-outage incident. Unfortunately I'm dealing with a much more serious situation now — my node was compromised and an attacker used the RPC interface to force-close 4 channels and sweep the wallet, stealing ~1.4 BTC (140,416,882 sats). Quick summary: - Attacker gained access (entry vector not fully confirmed — likely SSH into a host machine adjacent to the node) - Used CloseChannel RPC with a custom --delivery_addr to redirect settled balances to a single address - Channels with active/pending HTLCs were protected (close request failed with "cannot co-op close channel with active htlcs") — only HTLC-free channels were drained - Logs (auth.log, wtmp, bash_history) were wiped on both the node VM and the host machine shortly after - Funds are still sitting unmoved at the destination address as of now Destination address: bc1qchn3jv87gaalgp2yvnqtcf2p3mx5632atdmmyw https://mempool.space/address/bc1qchn3jv87gaalgp2yvnqtcf2p3mx5632atdmmyw Full timeline with tx IDs, block heights, and log excerpts here: [리포트 링크] Honestly, I still don't know exactly how they got in. I've traced the LND-side behavior pretty thoroughly (RPC calls, the HTLC-blocking pattern, the delivery_addr targeting), but the actual initial access point is still unclear even after going through SSH logs, auth logs, and shell history on both machines — a lot of it had already been wiped or aged out by the time I looked. If this matches any attack pattern you've seen before — macaroon leaks, exposed RPC ports, known LND exploits, whatever — I'd really appreciate hearing about it. Even a "this looks like X" would help me know what to check next. I've already filed a police report. Wondering if you'd be wie lnd.log — specifically curious whether there's anything atthe LND/protocol level that could help identify the RPC client, or any other channel-state forensics I might be missing (I'm not deeply familiar with the internals). No pressure if you're busy, just really

…

On Fri, Jun 19, 2026 at 1:13 AM Yong ***@***.***> wrote: I've created this PR: lightninglabs/chantools#241 <lightninglabs/chantools#241> You can try to sweep it via chantools sweephtlc. — Reply to this email directly, view it on GitHub <#10908?email_source=notifications&email_token=CGIMFBS3Z4EWGKE7RYAM7LT5AQII5A5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZTGU2DENBZUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVRTG633UMVZF6Y3MNFRWW#discussioncomment-17354249>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CGIMFBSD2UYGSWXFW26Q67L5AQII5AVCNFSNUABHKJSXA33TNF2G64TZHM2DSNZWGU2DQMB3IRUXGY3VONZWS33OHMYTAMRXHAYDGNVBOYBA> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/CGIMFBWNZKVQITU6EZYUEGL5AQII5A5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZTGU2DENBZUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVJTG633UMVZF62LPOM> and Android <https://github.com/notifications/mobile/android/CGIMFBTOCWMVTPLGY4QRHNT5AQII5A5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZTGU2DENBZUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVZTG633UMVZF6YLOMRZG62LE>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

[yyforyongyu]

Truly sorry about your loss. Let's move this conversation to a private channel for further investigation.

[terryas7777-dot]

Hi Yong, You already have the details from earlier — thank you, as always, for helping out in urgent situations like this. Attached is an AI-generated summary of the incident, which I hope will be useful for your analysis. Thanks again, Terry

…

On Sun, Aug 9, 2026 at 1:49 PM Yong ***@***.***> wrote: Truly sorry about your loss. Let's move this conversation to a private channel for further investigation. — Reply to this email directly, view it on GitHub <#10908?email_source=notifications&email_token=CGIMFBSKGCYUAODMOIZDPHD5I77G5A5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZZGQ4TGOJUUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVRTG633UMVZF6Y3MNFRWW#discussioncomment-17949394>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CGIMFBQ6M6Y367HNPD5RHLL5I77G5AVCNFSNUABHKJSXA33TNF2G64TZHM2DSNZWGU2DQMB3IRUXGY3VONZWS33OHMYTAMRXHAYDGNVBOYBA> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/CGIMFBSU7YLRXUKHM5VBGLD5I77G5A5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZZGQ4TGOJUUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVJTG633UMVZF62LPOM> and Android <https://github.com/notifications/mobile/android/CGIMFBULGGAS5CPC7ADJVM35I77G5A5CNFSNUABIM5UWIORPF5TWS5BNNB2WEL2ENFZWG5LTONUW63SDN5WW2ZLOOQXTCNZZGQ4TGOJUUZZGKYLTN5XKMYLVORUG64VFMV3GK3TUVZTG633UMVZF6YLOMRZG62LE>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>