golemfactory/ya-runtime-ai

Runtime downloads a model when there is no enough storage space.

Opened this issue · 0 comments

pwalski commented

Not related directly to Automatic. Scenario of downloading model when there is no enough free storage space.

Runtime logs:

[2024-04-10T12:20:29.220+0200 ERROR ya_transfer::file] Error writing to file [E:/modules/golem-data/provider/exe-unit/cache/tmp/4d44bf1c5ec6b78a909227dc98983bb474322e24fa3869476171fd86]: IO error: There is not enough space on the disk. (os error 112)
[2024-04-10T12:20:29.223+0200 ERROR ya_runtime_ai] ExeScript failure: Service("DeployImage failed: Send error: send failed because receiver is gone")
[2024-04-10T12:20:29.223+0200 DEBUG rustls::common_state] Sending warning alert CloseNotify
[2024-04-10T12:20:29.330+0200 DEBUG ya_runtime_ai] Successfully sent activity usage message
[2024-04-10T12:20:29.357+0200 INFO  ya_runtime_ai] CTRL-BREAK received, Shutting down runtime...

This time Agreement was terminated by Requestor:

[2024-04-10T12:20:29.356+0200 INFO  ya_provider::market::provider_market] Agreement [3a5c1a5c6a23096176a053af5be22b2b95565948af2744c9de8e303248785088] terminated by Requestor. Reason: Work cancelled
[2024-04-10T12:20:29.356+0200 DEBUG ya_client::web] doing GET on http://127.0.0.1:12502/market-api/v1/agreementEvents?timeout=20&afterTimestamp=2024-04-10T10%3A20%3A29.354732%2B00%3A00&maxEvents=15&appSessionId=ya-provider-160060
[2024-04-10T12:20:29.356+0200 WARN  ya_provider::execution::task_runner] Activity [406eb92e260b4d59be1ad603c4506ef9] will be destroyed, because of terminated agreement [3a5c1a5c6a23096176a053af5be22b2b95565948af2744c9de8e303248785088].
[2024-04-10T12:20:29.357+0200 INFO  ya_provider::payments::payments] Payments - agreement [3a5c1a5c6a23096176a053af5be22b2b95565948af2744c9de8e303248785088] closed. Computing cost summary...
[2024-04-10T12:20:29.357+0200 INFO  ya_provider::execution::task_runner] Destroying activity [406eb92e260b4d59be1ad603c4506ef9].
[2024-04-10T12:20:29.357+0200 INFO  ya_provider::execution::exeunit_instance] Terminating ExeUnit [automatic]... pid: 165168
[2024-04-10T12:20:29.357+0200 INFO  ya_provider::market::negotiator::builtin::max_agreements] Negotiator: 1 free slot(s) for agreements.
[2024-04-10T12:20:29.357+0200 INFO  ya_provider::market::provider_market] Re-negotiating all demands
[2024-04-10T12:20:29.357+0200 DEBUG ya_provider::payments::agreement] Waiting for all activities to finish.
[2024-04-10T12:20:29.361+0200 INFO  ya_provider::execution::task_runner] ExeUnit process exited with status Finished - exit code: 0, agreement [3a5c1a5c6a23096176a053af5be22b2b95565948af2744c9de8e303248785088], activity [406eb92e260b4d59be1ad603c4506ef9].
[2024-04-10T12:20:29.361+0200 INFO  ya_provider::execution::task_runner] ExeUnit for activity terminated: [406eb92e260b4d59be1ad603c4506ef9].

Provider logs are full of

[2024-04-10T12:21:31.733+0200 WARN  ya_provider::payments::payments] Error sending debit note: Failed to issue debit note for activity [406eb92e260b4d59be1ad603c4506ef9]. HTTP error requesting POST http://127.0.0.1:12502/payment-api/v1/debitNotes: 400 Bad Request; msg: 'Cannot perform this operation while a transaction is open' Retry in 24.774295497s.

Activity probably should be immediately terminated when there is not enough storage to download model.

Originally posted by @pwalski in #79