stakwork/sphinx-key

`SENDING TO _ on topic lss-msg` deadlock

Closed this issue · 4 comments

irriden commented

Signer had a problem and likely needed a restart while broker executed these lines of code:

sphinx-key/broker/src/looper.rs

Lines 170 to 176 in 6516b20

log::info!("GOT ON {}", res_topic);
let the_res = if res_topic == topics::LSS_RES {
// send reply to LSS to store muts
let lss_reply = self.send_lss(res)?;
log::info!("LSS REPLY LEN {}", &lss_reply.len());
// send to signer for HMAC validation, and get final reply
log::info!("SEND ON {}", topics::LSS_MSG);

Broker is then stuck trying to send the message on topic lss-msg, and signer does not respond.

Manually restarting the broker fixed the issue temporarily.

[2023-09-05T12:30:01.521 hsmd  /sphinx_key_broker::looper ␛[32mINFO␛[0m] loop 42/30: got Ecdh(Ecdh { point: 029e16f02286ae2502480b2dc4cbf43c325820b9a75e34c9b5fdb45bbf1d8320ba })
[2023-09-05T12:30:01.521 hsmd  /sphinx_key_broker::looper ␛[32mINFO␛[0m] SEND ON vls
[2023-09-05T12:30:01.521 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic vls
[2023-09-05T12:30:01.746 hsmd  /sphinx_key_broker::looper ␛[32mINFO␛[0m] GOT ON vls-res
2023-09-05T12:30:01.746Z DEBUG   connectd: Responder: handshake failed act_one_responder2:939
2023-09-05T12:30:25.657Z DEBUG   plugin-bcli: Feerate min_acceptable raised from 4603 perkb to floor of 6518
2023-09-05T12:30:55.700Z DEBUG   plugin-bcli: Feerate min_acceptable raised from 4603 perkb to floor of 6515
2023-09-05T12:31:25.727Z DEBUG   plugin-bcli: Feerate min_acceptable raised from 4603 perkb to floor of 6513
2023-09-05T12:31:55.783Z DEBUG   plugin-bcli: Feerate min_acceptable raised from 4603 perkb to floor of 6509
[2023-09-05T12:32:10.167 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Websocket failed handshake error=Protocol(MissingConnectionUpgradeHeader)
[2023-09-05T12:32:10.310 hsmd  /sphinx_key_broker::looper ␛[32mINFO␛[0m] loop 42/30: got Ecdh(Ecdh { point: 029015639139e88c79a42b1475aca588527e17c529d2642f871d3c5ecac9e6b72a })
[2023-09-05T12:32:10.310 hsmd  /sphinx_key_broker::looper ␛[32mINFO␛[0m] SEND ON vls
[2023-09-05T12:32:10.310 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic vls
[2023-09-05T12:32:10.539 hsmd  /sphinx_key_broker::looper ␛[32mINFO␛[0m] GOT ON vls-res
2023-09-05T12:32:10.540Z DEBUG   connectd: Responder: handshake failed act_one_responder2:939
[2023-09-05T12:32:25.324 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Websocket failed handshake error=Protocol(WrongHttpVersion)
2023-09-05T12:32:25.832Z DEBUG   plugin-bcli: Feerate min_acceptable raised from 4603 perkb to floor of 6506
[2023-09-05T12:32:40.481 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Websocket failed handshake error=Protocol(HttparseError(Token))
[2023-09-05T12:32:52.638 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Websocket failed handshake error=Protocol(HttparseError(Token))
2023-09-05T12:32:55.870Z DEBUG   plugin-bcli: Feerate min_acceptable raised from 4603 perkb to floor of 6502
[2023-09-05T12:33:07.794 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Websocket failed handshake error=Protocol(HttparseError(Token))
[2023-09-05T12:33:22.950 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Websocket failed handshake error=Protocol(HttparseError(Token))
2023-09-05T12:33:25.898Z DEBUG   plugin-bcli: Feerate min_acceptable raised from 4603 perkb to floor of 6500
[2023-09-05T12:33:38.115 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Websocket failed handshake error=Protocol(HttparseError(Token))
[2023-09-05T12:33:53.272 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Websocket failed handshake error=Protocol(HttparseError(Token))
2023-09-05T12:33:53.759Z DEBUG   plugin-bcli: Log pruned 45592 entries (mem 10502740 -> 2021570)
2023-09-05T12:33:54.101Z DEBUG   lightningd: Adding block 806324: 00000000000000000002bc1ad7670fd1b33af3bd8b8fa222117b9b6fee677091
2023-09-05T12:33:54.423Z DEBUG   0379bb7397a354dc8bfb07d3d8a0d24d88d84c0eae2c6a68fd15149aaafbbaca95-chan#1: attempting update blockheight c55652149478eebd4b0c6f0ac2c72a8e190df8d8df31113975984367666f9243
2023-09-05T12:33:55.935Z DEBUG   plugin-bcli: Feerate min_acceptable raised from 4603 perkb to floor of 6498
2023-09-05T12:33:55.935Z DEBUG   lightningd: Feerate estimate for 6 blocks set to 3991 (was 3855)
2023-09-05T12:33:55.935Z DEBUG   lightningd: update_feerates: feerate = 3991, min=1625, max=41670, penalty=3750
[2023-09-05T12:34:00.500 hsmd  /sphinx_key_broker::looper ␛[32mINFO␛[0m] loop 42/33: got ValidateCommitmentTx(ValidateCommitmentTx { tx: WithSize(Transaction { version: 2, lock_time: PackedLockTime(2006b853), input: [TxIn { previous_output: OutPoint { txid: 43926f6667439875391131dfd8f80d198e2ac7c20a6f0c4bbdee7894145256c5, vout: 0 }, script_sig: Script(), sequence: Sequence(8085e4a1), witness: Witness { content: [], witness_elements: 0, last: 0, second_to_last: 0 } }], output: [TxOut { value: 4eae, script_pubkey: Script(OP_0 OP_PUSHBYTES_32 3c8d37c0e5ba5f5e87710575efa8b74338350236acf4e49d49c25ea75cb4db20) }, TxOut { value: 6959, script_pubkey: Script(OP_0 OP_PUSHBYTES_20 d2b845a6442277130488ad72c3475ac7de0fa973) }] }), psbt: WithSize(PartiallySignedTransaction { unsigned_tx: Transaction { version: 2, lock_time: PackedLockTime(2006b853), input: [TxIn { previous_output: OutPoint { txid: 43926f6667439875391131dfd8f80d198e2ac7c20a6f0c4bbdee7894145256c5, vout: 0 }, script_sig: Script(), sequence: Sequence(8085e4a1), witness: Witness { content: [], witness_elements: 0, last: 0, second_to_last: 0 } }], output: [TxOut { value: 4eae, script_pubkey: Script(OP_0 OP_PUSHBYTES_32 3c8d37c0e5ba5f5e87710575efa8b74338350236acf4e49d49c25ea75cb4db20) }, TxOut { value: 6959, script_pubkey: Script(OP_0 OP_PUSHBYTES_20 d2b845a6442277130488ad72c3475ac7de0fa973) }] }, version: 0, xpub: {}, proprietary: {}, unknown: {}, inputs: [Input { non_witness_utxo: None, witness_utxo: Some(TxOut { value: c350, script_pubkey: Script(OP_0 OP_PUSHBYTES_32 2bdc1acc811a9caa7f7f4c86869304760369050b894a7eaf5a6ce616032b3897) }), partial_sigs: {PublicKey { compressed: true, inner: PublicKey(8e30ce2f4bfed39f5253ff9e973f2f15421ed9a0b85cc8e63c967d613051598d968010fcb6094c1a8c0d0ebe38f7db3181a0fbffded48d977bef77c6be9551ce) }: EcdsaSig { sig: 304402200264e3b81bfb451ce5b418d5673c1c830f9eef20d63f72f61740c06e6a5582bd0220541f2de60966f29968c052c37cf3183a873f901cfa9f7c914d57968b74f902b6, hash_ty: All }}, sighash_type: Some(PsbtSighashType { inner: 1 }), redeem_script: None, witness_script: Some(Script(OP_PUSHNUM_2 OP_PUSHBYTES_33 028d595130617d963ce6c85cb8a0d91e42152f3f979eff53529fd3fe4b2fce308e OP_PUSHBYTES_33 03873658e366d2d90f14517c4165c45be870e3bf2d7e5db886dd1011e88ea876c1 OP_PUSHNUM_2 OP_CHECKMULTISIG)), bip32_derivation: {PublicKey(8e30ce2f4bfed39f5253ff9e973f2f15421ed9a0b85cc8e63c967d613051598d968010fcb6094c1a8c0d0ebe38f7db3181a0fbffded48d977bef77c6be9551ce): (0f6dec7a, m/0), PublicKey(c176a88ee81110dd86b85d7e2dbfe370e85bc465417c51140fd9d266e35836873d257692b4fe40e4bd6eb567c4ccee901981b11c1fc9e45d8024246b09f02fa7): (d3219d74, m/0)}, final_script_sig: None, final_script_witness: None, ripemd160_preimages: {}, sha256_preimages: {}, hash160_preimages: {}, hash256_preimages: {}, tap_key_sig: None, tap_script_sigs: {}, tap_scripts: {}, tap_key_origins: {}, tap_internal_key: None, tap_merkle_root: None, proprietary: {}, unknown: {} }], outputs: [Output { redeem_script: None, witness_script: Some(Script(OP_IF OP_PUSHBYTES_33 031c009400c46eab5e15e7482aa2ba5aab575e1e949261d005dcc18dc013e739ee OP_ELSE OP_PUSHBYTES_2 9000 OP_CSV OP_DROP OP_PUSHBYTES_33 03f0a19ca097e7c424b91a132b257ca7f43192ae6450a0266d85df75ece05b16c0 OP_ENDIF OP_CHECKSIG)), bip32_derivation: {}, tap_internal_key: None, tap_tree: None, tap_key_origins: {}, proprietary: {}, unknown: {} }, Output { redeem_script: None, witness_script: None, bip32_derivation: {}, tap_internal_key: None, tap_tree: None, tap_key_origins: {}, proprietary: {}, unknown: {} }] }), htlcs: Array([]), commitment_number: 183, feerate: f97, signature: BitcoinSignature { signature: 0264e3b81bfb451ce5b418d5673c1c830f9eef20d63f72f61740c06e6a5582bd541f2de60966f29968c052c37cf3183a873f901cfa9f7c914d57968b74f902b6, sighash: 1 }, htlc_signatures: Array([]) })
[2023-09-05T12:34:00.501 hsmd  /sphinx_key_broker::looper ␛[32mINFO␛[0m] SEND ON vls
[2023-09-05T12:34:00.501 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic vls
[2023-09-05T12:34:04.506 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic vls
[2023-09-05T12:34:04.559 hsmd  /sphinx_key_broker::looper ␛[32mINFO␛[0m] GOT ON lss-res
[2023-09-05T12:34:04.559 hsmd  /lss_connector::broker ␛[32mINFO␛[0m] HANDLE LSS VlsMuts(SignerMutations { client_hmac: [51, 229, 42, 247, 109, 3, 175, 155, 102, 51, 192, 126, 43, 81, 128, 137, 26, 66, 172, 24, 240, 93, 221, 218, 130, 139, 36, 140, 192, 113, 50, 181], muts: [("channel/037eeaacefb06ae0141fcc3d936a7da0250c9990f16aa01ba60d4ee9449e9196e3/0379bb7397a354dc8bfb07d3d8a0d24d88d84c0eae2c6a68fd15149aaafbbaca950100000000000000", (1167, [123, 34, 99, 104, 97, 110, 110, 101, 108, 95, 118, 97, 108, 117, 101, 95, 115, 97, 116, 111, 115, 104, 105, 115, 34, 58, 53, 48, 48, 48, 48, 44, 34, 99, 104, 97, 110, 110, 101, 108, 95, 115, 101, 116, 117, 112, 34, 58, 123, 34, 105, 115, 95, 111, 117, 116, 98, 111, 117, 110, 100, 34, 58, 102, 97, 108, 115, 101, 44, 34, 99, 104, 97, 110, 110, 101, 108, 95, 118, 97, 108, 117, 101, 95, 115, 97, 116, 34, 58, 53, 48, 48, 48, 48, 44, 34, 112, 117, 115, 104, 95, 118, 97, 108, 117, 101, 95, 109, 115, 97, 116, 34, 58, 50, 48, 48, 48, 48, 44, 34, 102, 117, 110, 100, 105, 110, 103, 95, 111, 117, 116, 112, 111, 105, 110, 116, 34, 58, 123, 34, 116, 120, 105, 100, 34, 58, 34, 99, 53, 53, 54, 53, 50, 49, 52, 57, 52, 55, 56, 101, 101, 98, 100, 52, 98, 48, 99, 54, 102, 48, 97, 99, 50, 99, 55, 50, 97, 56, 101, 49, 57, 48, 100, 102, 56, 100, 56, 100, 102, 51, 49, 49, 49, 51, 57, 55, 53, 57, 56, 52, 51, 54, 55, 54, 54, 54, 102, 57, 50, 52, 51, 34, 44, 34, 118, 111, 117, 116, 34, 58, 48, 125, 44, 34, 104, 111, 108, 100, 101, 114, 95, 115, 101, 108, 101, 99, 116, 101, 100, 95, 99, 111, 110, 116, 101, 115, 116, 95, 100, 101, 108, 97, 121, 34, 58, 49, 52, 52, 44, 34, 104, 111, 108, 100, 101, 114, 95, 115, 104, 117, 116, 100, 111, 119, 110, 95, 115, 99, 114, 105, 112, 116, 34, 58, 110, 117, 108, 108, 44, 34, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 112, 111, 105, 110, 116, 115, 34, 58, 123, 34, 102, 117, 110, 100, 105, 110, 103, 95, 112, 117, 98, 107, 101, 121, 34, 58, 34, 48, 50, 56, 100, 53, 57, 53, 49, 51, 48, 54, 49, 55, 100, 57, 54, 51, 99, 101, 54, 99, 56, 53, 99, 98, 56, 97, 48, 100, 57, 49, 101, 52, 50, 49, 53, 50, 102, 51, 102, 57, 55, 57, 101, 102, 102, 53, 51, 53, 50, 57, 102, 100, 51, 102, 101, 52, 98, 50, 102, 99, 101, 51, 48, 56, 101, 34, 44, 34, 114, 101, 118, 111, 99, 97, 116, 105, 111, 110, 95, 98, 97, 115, 101, 112, 111, 105, 110, 116, 34, 58, 34, 48, 50, 55, 100, 99, 97, 98, 98, 51, 100, 55, 52, 48, 100, 98, 53, 57, 55, 51, 97, 53, 56, 57, 99, 100, 50, 51, 54, 56, 97, 54, 57, 52, 100, 98, 98, 101, 99, 97, 100, 102, 49, 57, 54, 101, 48, 53, 100, 102, 54, 48, 101, 53, 99, 55, 52, 57, 98, 101, 55, 57, 54, 99, 102, 57, 54, 34, 44, 34, 112, 97, 121, 109, 101, 110, 116, 95, 112, 111, 105, 110, 116, 34, 58, 34, 48, 51, 101, 52, 56, 100, 52, 48, 97, 52, 57, 49, 98, 49, 99, 51, 51, 100, 51, 99, 52, 56, 54, 52, 100, 52, 52, 54, 51, 97, 102, 50, 57, 48, 57, 98, 99, 48, 98, 100, 52, 50, 49, 50, 99, 98, 57, 49, 101, 55, 53, 97, 53, 52, 48, 100, 101, 49, 102, 57, 101, 55, 56, 101, 55, 101, 34, 44, 34, 100, 101, 108, 97, 121, 101, 100, 95, 112, 97, 121, 109, 101, 110, 116, 95, 98, 97, 115, 101, 112, 111, 105, 110, 116, 34, 58, 34, 48, 50, 52, 56, 102, 51, 48, 98, 48, 55, 97, 55, 51, 102, 52, 101, 97, 57, 98, 49, 52, 99, 54, 54, 100, 53, 102, 53, 99, 101, 50, 98, 52, 98, 57, 99, 51, 52, 56, 52, 55, 54, 55, 53, 97, 57, 97, 50, 48, 100, 49, 102, 55, 50, 56, 56, 99, 97, 57, 57, 50, 100, 57, 100, 52, 55, 34, 44, 34, 104, 116, 108, 99, 95, 98, 97, 115, 101, 112, 111, 105, 110, 116, 34, 58, 34, 48, 51, 100, 100, 98, 48, 54, 54, 48, 51, 51, 102, 98, 53, 57, 97, 56, 102, 98, 57, 98, 51, 97, 97, 98, 52, 100, 57, 54, 50, 97, 54, 102, 51, 55, 101, 54, 57, 98, 101, 52, 50, 57, 100, 52, 102, 56, 51, 101, 51, 48, 97, 98, 53, 56, 55, 52, 57, 57, 53, 97, 50, 98, 53, 48, 97, 34, 125, 44, 34, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 115, 101, 108, 101, 99, 116, 101, 100, 95, 99, 111, 110, 116, 101, 115, 116, 95, 100, 101, 108, 97, 121, 34, 58, 49, 52, 52, 44, 34, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 115, 104, 117, 116, 100, 111, 119, 110, 95, 115, 99, 114, 105, 112, 116, 34, 58, 110, 117, 108, 108, 44, 34, 99, 111, 109, 109, 105, 116, 109, 101, 110, 116, 95, 116, 121, 112, 101, 34, 58, 34, 83, 116, 97, 116, 105, 99, 82, 101, 109, 111, 116, 101, 75, 101, 121, 34, 125, 44, 34, 105, 100, 34, 58, 34, 48, 51, 55, 57, 98, 98, 55, 51, 57, 55, 97, 51, 53, 52, 100, 99, 56, 98, 102, 98, 48, 55, 100, 51, 100, 56, 97, 48, 100, 50, 52, 100, 56, 56, 100, 56, 52, 99, 48, 101, 97, 101, 50, 99, 54, 97, 54, 56, 102, 100, 49, 53, 49, 52, 57, 97, 97, 97, 102, 98, 98, 97, 99, 97, 57, 53, 48, 49, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 48, 34, 44, 34, 101, 110, 102, 111, 114, 99, 101, 109, 101, 110, 116, 95, 115, 116, 97, 116, 101, 34, 58, 123, 34, 110, 101, 120, 116, 95, 104, 111, 108, 100, 101, 114, 95, 99, 111, 109, 109, 105, 116, 95, 110, 117, 109, 34, 58, 51, 56, 56, 44, 34, 110, 101, 120, 116, 95, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 99, 111, 109, 109, 105, 116, 95, 110, 117, 109, 34, 58, 51, 56, 56, 44, 34, 110, 101, 120, 116, 95, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 114, 101, 118, 111, 107, 101, 95, 110, 117, 109, 34, 58, 51, 56, 55, 44, 34, 99, 117, 114, 114, 101, 110, 116, 95, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 112, 111, 105, 110, 116, 34, 58, 34, 48, 51, 102, 100, 102, 99, 57, 99, 56, 51, 102, 55, 101, 98, 50, 102, 98, 48, 49, 50, 52, 57, 57, 97, 50, 99, 100, 97, 51, 102, 97, 55, 53, 97, 56, 101, 99, 100, 48, 97, 57, 52, 99, 100, 97, 50, 53, 52, 98, 50, 97, 55, 98, 57, 52, 52, 56, 98, 50, 102, 52, 100, 99, 101, 98, 48, 34, 44, 34, 112, 114, 101, 118, 105, 111, 117, 115, 95, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 112, 111, 105, 110, 116, 34, 58, 34, 48, 51, 51, 102, 51, 99, 50, 49, 99, 102, 98, 57, 52, 100, 100, 54, 98, 49, 57, 54, 52, 99, 52, 102, 97, 48, 51, 48, 54, 52, 101, 97, 98, 54, 49, 100, 54, 51, 97, 97, 52, 54, 52, 49, 56, 55, 48, 54, 101, 55, 97, 51, 99, 55, 49, 98, 102, 102, 50, 99, 99, 50, 100, 98, 101, 51, 34, 44, 34, 99, 117, 114, 114, 101, 110, 116, 95, 104, 111, 108, 100, 101, 114, 95, 99, 111, 109, 109, 105, 116, 95, 105, 110, 102, 111, 34, 58, 123, 34, 105, 115, 95, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 98, 114, 111, 97, 100, 99, 97, 115, 116, 101, 114, 34, 58, 102, 97, 108, 115, 101, 44, 34, 116, 111, 95, 99, 111, 117, 110, 116, 101, 114, 115, 105, 103, 110, 101, 114, 95, 112, 117, 98, 107, 101, 121, 34, 58, 34, 48, 51, 101, 52, 56, 100, 52, 48, 97, 52, 57, 49, 98, 49, 99, 51, 51, 100, 51, 99, 52, 56, 54, 52, 100, 52, 52, 54, 51, 97, 102, 50, 57, 48, 57, 98, 99, 48, 98, 100, 52, 50, 49, 50, 99, 98, 57, 49, 101, 55, 53, 97, 53, 52, 48, 100, 101, 49, 102, 57, 101, 55, 56, 101, 55, 101, 34, 44, 34, 116, 111, 95, 99, 111, 117, 110, 116, 101, 114, 115, 105, 103, 110, 101, 114, 95, 118, 97, 108, 117, 101, 95, 115, 97, 116, 34, 58, 50, 54, 57, 54, 57, 44, 34, 114, 101, 118, 111, 99, 97, 116, 105, 111, 110, 95, 112, 117, 98, 107, 101, 121, 34, 58, 34, 48, 51, 49, 99, 48, 48, 57, 52, 48, 48, 99, 52, 54, 101, 97, 98, 53, 101, 49, 53, 101, 55, 52, 56, 50, 97, 97, 50, 98, 97, 53, 97, 97, 98, 53, 55, 53, 101, 49, 101, 57, 52, 57, 50, 54, 49, 100, 48, 48, 53, 100, 99, 99, 49, 56, 100, 99, 48, 49, 51, 101, 55, 51, 57, 101, 101, 34, 44, 34, 116, 111, 95, 98, 114, 111, 97, 100, 99, 97, 115, 116, 101, 114, 95, 100, 101, 108, 97, 121, 101, 100, 95, 112, 117, 98, 107, 101, 121, 34, 58, 34, 48, 51, 102, 48, 97, 49, 57, 99, 97, 48, 57, 55, 101, 55, 99, 52, 50, 52, 98, 57, 49, 97, 49, 51, 50, 98, 50, 53, 55, 99, 97, 55, 102, 52, 51, 49, 57, 50, 97, 101, 54, 52, 53, 48, 97, 48, 50, 54, 54, 100, 56, 53, 100, 102, 55, 53, 101, 99, 101, 48, 53, 98, 49, 54, 99, 48, 34, 44, 34, 116, 111, 95, 98, 114, 111, 97, 100, 99, 97, 115, 116, 101, 114, 95, 118, 97, 108, 117, 101, 95, 115, 97, 116, 34, 58, 50, 48, 49, 52, 50, 44, 34, 116, 111, 95, 115, 101, 108, 102, 95, 100, 101, 108, 97, 121, 34, 58, 49, 52, 52, 44, 34, 111, 102, 102, 101, 114, 101, 100, 95, 104, 116, 108, 99, 115, 34, 58, 91, 93, 44, 34, 114, 101, 99, 101, 105, 118, 101, 100, 95, 104, 116, 108, 99, 115, 34, 58, 91, 93, 44, 34, 102, 101, 101, 114, 97, 116, 101, 95, 112, 101, 114, 95, 107, 119, 34, 58, 51, 57, 57, 49, 125, 44, 34, 99, 117, 114, 114, 101, 110, 116, 95, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 115, 105, 103, 110, 97, 116, 117, 114, 101, 115, 34, 58, 91, 34, 51, 48, 52, 52, 48, 50, 50, 48, 48, 50, 54, 52, 101, 51, 98, 56, 49, 98, 102, 98, 52, 53, 49, 99, 101, 53, 98, 52, 49, 56, 100, 53, 54, 55, 51, 99, 49, 99, 56, 51, 48, 102, 57, 101, 101, 102, 50, 48, 100, 54, 51, 102, 55, 50, 102, 54, 49, 55, 52, 48, 99, 48, 54, 101, 54, 97, 53, 53, 56, 50, 98, 100, 48, 50, 50, 48, 53, 52, 49, 102, 50, 100, 101, 54, 48, 57, 54, 54, 102, 50, 57, 57, 54, 56, 99, 48, 53, 50, 99, 51, 55, 99, 102, 51, 49, 56, 51, 97, 56, 55, 51, 102, 57, 48, 49, 99, 102, 97, 57, 102, 55, 99, 57, 49, 52, 100, 53, 55, 57, 54, 56, 98, 55, 52, 102, 57, 48, 50, 98, 54, 34, 44, 91, 93, 93, 44, 34, 99, 117, 114, 114, 101, 110, 116, 95, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 99, 111, 109, 109, 105, 116, 95, 105, 110, 102, 111, 34, 58, 123, 34, 105, 115, 95, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 98, 114, 111, 97, 100, 99, 97, 115, 116, 101, 114, 34, 58, 116, 114, 117, 101, 44, 34, 116, 111, 95, 99, 111, 117, 110, 116, 101, 114, 115, 105, 103, 110, 101, 114, 95, 112, 117, 98, 107, 101, 121, 34, 58, 34, 48, 51, 55, 50, 55, 101, 57, 100, 97, 98, 56, 48, 98, 52, 99, 56, 56, 54, 55, 55, 100, 53, 55, 102, 102, 98, 57, 102, 52, 49, 100, 52, 98, 50, 100, 57, 53, 48, 51, 102, 53, 100, 53, 50, 54, 99, 100, 52, 97, 50, 55, 48, 57, 51, 52, 102, 49, 55, 57, 55, 53, 102, 99, 102, 49, 48, 34, 44, 34, 116, 111, 95, 99, 111, 117, 110, 116, 101, 114, 115, 105, 103, 110, 101, 114, 95, 118, 97, 108, 117, 101, 95, 115, 97, 116, 34, 58, 50, 48, 49, 52, 50, 44, 34, 114, 101, 118, 111, 99, 97, 116, 105, 111, 110, 95, 112, 117, 98, 107, 101, 121, 34, 58, 34, 48, 50, 97, 54, 57, 53, 102, 98, 100, 51, 56, 51, 53, 52, 50, 97, 100, 53, 51, 99, 99, 53, 48, 57, 50, 101, 101, 53, 52, 100, 49, 55, 97, 98, 97, 56, 98, 99, 101, 99, 55, 54, 56, 48, 57, 50, 101, 51, 51, 56, 97, 49, 57, 57, 50, 50, 50, 51, 54, 48, 100, 57, 98, 97, 56, 97, 34, 44, 34, 116, 111, 95, 98, 114, 111, 97, 100, 99, 97, 115, 116, 101, 114, 95, 100, 101, 108, 97, 121, 101, 100, 95, 112, 117, 98, 107, 101, 121, 34, 58, 34, 48, 50, 101, 56, 53, 98, 51, 100, 56, 55, 99, 101, 56, 98, 54, 48, 102, 51, 56, 57, 52, 52, 55, 48, 99, 102, 101, 99, 53, 49, 48, 102, 49, 99, 55, 51, 55, 55, 56, 52, 49, 51, 53, 101, 50, 100, 49, 50, 48, 50, 50, 54, 101, 52, 49, 51, 49, 100, 51, 55, 50, 53, 56, 49, 55, 101, 34, 44, 34, 116, 111, 95, 98, 114, 111, 97, 100, 99, 97, 115, 116, 101, 114, 95, 118, 97, 108, 117, 101, 95, 115, 97, 116, 34, 58, 50, 55, 49, 52, 51, 44, 34, 116, 111, 95, 115, 101, 108, 102, 95, 100, 101, 108, 97, 121, 34, 58, 49, 52, 52, 44, 34, 111, 102, 102, 101, 114, 101, 100, 95, 104, 116, 108, 99, 115, 34, 58, 91, 93, 44, 34, 114, 101, 99, 101, 105, 118, 101, 100, 95, 104, 116, 108, 99, 115, 34, 58, 91, 93, 44, 34, 102, 101, 101, 114, 97, 116, 101, 95, 112, 101, 114, 95, 107, 119, 34, 58, 51, 55, 53, 48, 125, 44, 34, 112, 114, 101, 118, 105, 111, 117, 115, 95, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 99, 111, 109, 109, 105, 116, 95, 105, 110, 102, 111, 34, 58, 110, 117, 108, 108, 44, 34, 99, 104, 97, 110, 110, 101, 108, 95, 99, 108, 111, 115, 101, 100, 34, 58, 102, 97, 108, 115, 101, 44, 34, 105, 110, 105, 116, 105, 97, 108, 95, 104, 111, 108, 100, 101, 114, 95, 118, 97, 108, 117, 101, 34, 58, 50, 48, 44, 34, 99, 111, 117, 110, 116, 101, 114, 112, 97, 114, 116, 121, 95, 115, 101, 99, 114, 101, 116, 115, 34, 58, 123, 34, 111, 108, 100, 95, 115, 101, 99, 114, 101, 116, 115, 34, 58, 91, 91, 91, 57, 44, 50, 51, 51, 44, 50, 49, 55, 44, 49, 54, 57, 44, 52, 44, 55, 53, 44, 50, 51, 54, 44, 49, 55, 53, 44, 49, 53, 57, 44, 49, 51, 56, 44, 49, 48, 54, 44, 56, 52, 44, 49, 56, 51, 44, 55, 51, 44, 49, 50, 57, 44, 56, 57, 44, 49, 50, 57, 44, 49, 57, 57, 44, 49, 51, 51, 44, 52, 54, 44, 49, 55, 53, 44, 50, 49, 50, 44, 49, 44, 49, 56, 52, 44, 50, 52, 48, 44, 51, 54, 44, 49, 51, 55, 44, 49, 48, 50, 44, 52, 52, 44, 49, 53, 53, 44, 49, 54, 51, 44, 54, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 50, 54, 57, 93, 44, 91, 91, 49, 51, 53, 44, 49, 54, 55, 44, 49, 52, 56, 44, 49, 52, 49, 44, 50, 54, 44, 49, 54, 51, 44, 49, 54, 53, 44, 50, 51, 56, 44, 55, 53, 44, 54, 55, 44, 49, 57, 44, 50, 50, 57, 44, 50, 49, 49, 44, 50, 49, 48, 44, 49, 55, 44, 50, 48, 52, 44, 49, 57, 53, 44, 49, 55, 49, 44, 50, 49, 44, 50, 51, 53, 44, 49, 48, 51, 44, 51, 57, 44, 56, 53, 44, 50, 49, 56, 44, 52, 54, 44, 49, 54, 51, 44, 49, 51, 48, 44, 57, 53, 44, 51, 55, 44, 53, 51, 44, 49, 53, 57, 44, 49, 53, 51, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 50, 55, 48, 93, 44, 91, 91, 49, 50, 48, 44, 50, 49, 57, 44, 50, 52, 54, 44, 49, 51, 51, 44, 50, 51, 48, 44, 50, 50, 49, 44, 49, 51, 49, 44, 50, 51, 52, 44, 50, 49, 57, 44, 51, 54, 44, 49, 53, 57, 44, 54, 57, 44, 52, 55, 44, 55, 50, 44, 50, 49, 49, 44, 50, 49, 55, 44, 49, 52, 52, 44, 50, 52, 53, 44, 49, 57, 44, 49, 51, 44, 49, 49, 55, 44, 52, 53, 44, 50, 48, 55, 44, 49, 52, 50, 44, 49, 55, 52, 44, 55, 53, 44, 55, 49, 44, 49, 50, 51, 44, 57, 48, 44, 56, 55, 44, 50, 48, 54, 44, 49, 53, 56, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 50, 55, 54, 93, 44, 91, 91, 52, 56, 44, 52, 50, 44, 50, 50, 57, 44, 55, 54, 44, 49, 50, 52, 44, 50, 49, 53, 44, 49, 51, 54, 44, 50, 49, 57, 44, 57, 48, 44, 53, 52, 44, 54, 57, 44, 50, 49, 52, 44, 50, 52, 52, 44, 52, 56, 44, 53, 49, 44, 50, 48, 51, 44, 49, 53, 52, 44, 49, 51, 55, 44, 52, 51, 44, 49, 50, 55, 44, 55, 49, 44, 54, 48, 44, 50, 52, 57, 44, 49, 51, 52, 44, 57, 57, 44, 49, 55, 50, 44, 54, 49, 44, 50, 49, 50, 44, 57, 48, 44, 51, 56, 44, 49, 50, 50, 44, 50, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 50, 56, 48, 93, 44, 91, 91, 49, 54, 53, 44, 49, 57, 49, 44, 49, 50, 57, 44, 49, 56, 48, 44, 50, 49, 48, 44, 52, 51, 44, 50, 50, 57, 44, 49, 56, 49, 44, 55, 53, 44, 55, 56, 44, 49, 49, 53, 44, 50, 52, 52, 44, 49, 56, 52, 44, 54, 57, 44, 49, 51, 51, 44, 50, 52, 48, 44, 55, 56, 44, 52, 53, 44, 50, 48, 48, 44, 54, 54, 44, 49, 53, 56, 44, 49, 50, 51, 44, 49, 49, 55, 44, 51, 52, 44, 50, 52, 51, 44, 50, 44, 49, 54, 57, 44, 52, 54, 44, 51, 52, 44, 52, 53, 44, 57, 50, 44, 53, 56, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 50, 56, 56, 93, 44, 91, 91, 49, 53, 48, 44, 53, 55, 44, 50, 51, 49, 44, 49, 53, 57, 44, 50, 52, 53, 44, 49, 55, 56, 44, 51, 48, 44, 49, 49, 53, 44, 50, 52, 57, 44, 50, 48, 49, 44, 50, 51, 54, 44, 49, 50, 54, 44, 49, 54, 48, 44, 49, 53, 52, 44, 49, 53, 52, 44, 49, 51, 51, 44, 51, 55, 44, 49, 57, 57, 44, 50, 48, 57, 44, 50, 52, 52, 44, 49, 50, 54, 44, 49, 51, 48, 44, 56, 51, 44, 57, 54, 44, 50, 49, 52, 44, 49, 57, 49, 44, 54, 56, 44, 52, 53, 44, 50, 51, 56, 44, 49, 48, 44, 49, 49, 49, 44, 53, 49, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 51, 54, 56, 93, 44, 91, 91, 49, 54, 57, 44, 49, 49, 49, 44, 50, 50, 52, 44, 49, 50, 51, 44, 50, 49, 53, 44, 49, 57, 53, 44, 49, 54, 51, 44, 49, 53, 53, 44, 56, 51, 44, 49, 50, 49, 44, 50, 48, 49, 44, 57, 51, 44, 49, 51, 50, 44, 49, 55, 49, 44, 49, 52, 53, 44, 54, 44, 50, 52, 53, 44, 57, 53, 44, 52, 53, 44, 50, 52, 48, 44, 49, 51, 48, 44, 53, 53, 44, 49, 57, 48, 44, 57, 50, 44, 51, 44, 50, 51, 51, 44, 49, 50, 55, 44, 52, 54, 44, 57, 53, 44, 49, 57, 50, 44, 50, 48, 54, 44, 57, 56, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 51, 51, 54, 93, 44, 91, 91, 49, 51, 57, 44, 49, 50, 54, 44, 49, 57, 53, 44, 51, 54, 44, 57, 49, 44, 49, 55, 51, 44, 51, 56, 44, 56, 48, 44, 49, 54, 49, 44, 50, 44, 49, 48, 50, 44, 49, 53, 48, 44, 50, 48, 48, 44, 51, 56, 44, 49, 48, 48, 44, 50, 51, 53, 44, 50, 49, 56, 44, 57, 53, 44, 49, 56, 55, 44, 49, 51, 53, 44, 49, 50, 53, 44, 49, 51, 51, 44, 49, 56, 50, 44, 50, 52, 52, 44, 49, 51, 51, 44, 49, 54, 52, 44, 50, 52, 44, 53, 55, 44, 52, 49, 44, 53, 50, 44, 50, 49, 57, 44, 49, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 50, 55, 50, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 44, 91, 91, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 44, 48, 93, 44, 50, 56, 49, 52, 55, 52, 57, 55, 54, 55, 49, 48, 54, 53, 54, 93, 93, 125, 125, 44, 34, 98, 108, 111, 99, 107, 104, 101, 105, 103, 104, 116, 34, 58, 110, 117, 108, 108, 125]))] })
[2023-09-05T12:34:04.561 hsmd  /lss_connector::broker ␛[32mINFO␛[0m] MSG TO SIGNER: Stored(BrokerMutations { server_hmac: Some([121, 146, 109, 124, 63, 117, 228, 64, 223, 190, 86, 194, 94, 214, 27, 232, 101, 7, 250, 3, 62, 120, 240, 56, 52, 16, 12, 207, 83, 171, 234, 108]), muts: [] })
[2023-09-05T12:34:04.561 hsmd  /sphinx_key_broker::looper ␛[32mINFO␛[0m] LSS REPLY LEN 61
[2023-09-05T12:34:04.561 hsmd  /sphinx_key_broker::looper ␛[32mINFO␛[0m] SEND ON lss-msg
[2023-09-05T12:34:04.561 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
[2023-09-05T12:34:08.429 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Websocket failed handshake error=Protocol(HttparseError(Token))
[2023-09-05T12:34:08.562 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
[2023-09-05T12:34:12.539 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Disconnected!! error=Network(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }))
[2023-09-05T12:34:12.562 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
[2023-09-05T12:34:12.734 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] remote_link; tenant_id=None
[2023-09-05T12:34:13.995 hsmd  /sphinx_key_broker ␛[32mINFO␛[0m] => connection status: Gs7AI7qKqUlY: true
[2023-09-05T12:34:13.995 hsmd  /sphinx_key_broker::lss ␛[32mINFO␛[0m] CLIENT Gs7AI7qKqUlY connected!
[2023-09-05T12:34:13.996 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic init-1-msg
[2023-09-05T12:34:14.992 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic init-2-msg
[2023-09-05T12:34:17.000 hsmd  /sphinx_key_broker ␛[32mINFO␛[0m] adding client to the list? true
[2023-09-05T12:34:17.562 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
[2023-09-05T12:34:21.562 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
[2023-09-05T12:34:23.586 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Websocket failed handshake error=Protocol(HttparseError(Token))
[2023-09-05T12:34:25.562 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
2023-09-05T12:34:25.964Z DEBUG   plugin-bcli: Feerate min_acceptable raised from 4603 perkb to floor of 6494
[2023-09-05T12:34:29.562 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
[2023-09-05T12:34:33.563 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
[2023-09-05T12:34:37.563 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
[2023-09-05T12:34:38.742 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Websocket failed handshake error=Protocol(HttparseError(Token))
[2023-09-05T12:34:41.563 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
[2023-09-05T12:34:45.563 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
[2023-09-05T12:34:49.563 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
[2023-09-05T12:34:53.563 hsmd  /sphinx_key_broker::mqtt ␛[32mINFO␛[0m] SENDING TO Gs7AI7qKqUlY on topic lss-msg
[2023-09-05T12:34:53.899 hsmd  /rumqttd::server::broker ␛[31mERROR␛[0m] Websocket failed handshake error=Protocol(HttparseError(Token))
2023-09-05T12:34:55.998Z DEBUG   plugin-bcli: Feerate min_acceptable raised from 4603 perkb to floor of 6491
Evanfeenstra commented

ok... to recap...

  1. signer got CLN message and signed it
  2. signer sent back LSS changes to store
  3. signer crashed for some reason (or restarted)
  4. broker keeps sending LSS res... but signer ignores it until LSS init is done
  5. then signer is getting "should be previous msg bytes" error, since it doesn't have the previous msgs in memory
  6. so signer is never responding... broker never gets the CLN response either...

Solution possibilities

  • signer keeps the previous msgs in non-volatile storage (SD card?). mobile signers would also have to implement something here....
  • broker gives up after a while, and goes back to retrying the previous CLN message (but then the signer is signing the same thing twice?)
  • refactor the message order so that the CLN response is also sent with the LSS stuff? But its not actually sent to CLN until the LSS stuff is stored? This maybe breaks the "atomic" nature of LSS updates though?
irriden commented

This works well for single signer, but doesn't work well in case a signer goes offline at this same point.

Broker tries the other signer, and keeps trying to send lss message, signer receives it, restores from sd card, but gets an E (297856) sphinx_key::core::events: Invalid server hmac because the message it just restored from the sd card doesn't match.

Evanfeenstra commented

Yes... this is what i was thinking yesterday... we need to refactor the way broker switches between signers. But lets merge the latest VLS stuff first

irriden commented

Fixed in 80a23cb

Key is the following: in the case where signer crashes before sending an LSS response, broker shouldn't try the LSS response again and again, but rather go back to sending the previous VLS message, and attempt to restart the show at that point.