Patchlog server starts re-syncing
Closed this issue · 13 comments
In our setup (see issue #154) we see now all over sudden, the patch log servers trying to resync. Applying all patch log. We didn't have a new Fuseki database or anything else and don't understand what triggers the re-sync. Since we have quite some patches, this takes very long
Any thoughts?
We just saw that after the patchlog servers did reach the latest patch they start all over again
@tomkxy - not sure what's going on here - it is not something I've seen myself or seen any reports.
Do you have the log of a patch log server doing this? Is it happening when the patch log server starts/restarts? Were there any Zookeeper incidents around the time this happened?
Given this and #154, could you run as a test a simpler setup (not production) with the file-backed patch log server?
@afs - Yes, we saw that after a restart. However, what I don't get it really starts from version 0 and we have right now 292234 patch versions.
How is the mechanism working anyhow.. would the Fuseki inquire a re-sync up to the latest version, here 292234 in a single request? If yes, could it be that this then fails because that is just too much data for a single request? Can you elaborate a bit more how the process works between Fuseki and the Patchlog Server works.
Btw, did you consider my request in #154, we desperately looking forward to somebody having a look into your config and helping us to figure out the root cause of the problems we are facing.
some additional infos:
The Fuseki server which is syching has the following last two log entry:
[2022-08-08 15:43:53] Delta INFO [id:803e4a:PATCHLOG] Sync: Versions [<init>, ver:293234]
[2022-08-08 16:47:48] Delta INFO [id:803e4a:PATCHLOG] Sync: Versions [ver:293230, ver:293234]
Obviously, it started from version 0, we saw that the state file looked like:
{
"version" : 0 ,
"id" : "6dd0b96b-3fda-49c0-b896-47a55f4ac1b5" ,
"name" : "PATCHLOG" ,
"datasource" : "803e4a44-66a9-4c22-a1ed-a7d0c6b61a6c" ,
"storage" : "TDB2" ,
"uri" : "delta:PATCHLOG"
}
Now after a couple of hours, it has synced up to version 293230, four patches are missing.
{
"version" : 293230 ,
"id" : "6dd0b96b-3fda-49c0-b896-47a55f4ac1b5" ,
"name" : "PATCHLOG" ,
"datasource" : "803e4a44-66a9-4c22-a1ed-a7d0c6b61a6c" ,
"storage" : "TDB2" ,
"uri" : "delta:PATCHLOG"
}
However, the patch log server is showing that it fetches starting from version 0, so it goes over all patches again.
@afs Today, I was able to re-produce that behavior. It turns out, that while a Fuseki server is syncing any new query hitting the server will trigger a resync with the same initial and target version of the currently running sync.
It is easy to imagine, if there is a longer running sync, this easily ends what looks like an endless syncing loop.
Hi @tomkxy , that's sounds like something I can investigate. I'll look at setting up a test case.
Could you attach the Fuseki configuration file for the server to this PR please?
Hi @afs, I re-produced the problem with a rather simple setup. One patchlog server, one Fuseki instance.
I started the Fuseki instance, with a version number 0 to force it into a sync during startup (that is what we see in our environments quite often, that the Fuseki start syncing from 0). Before I populated the database with enough patches so that the sync takes long enough to fire the query in between. Btw. this was done using V1.1.0. We rolled back to that version after we had all the issues, though obviously it didn't cause them.
Fuseki Log (see the two entries: INFO [id:ac5a53:ABC] Sync: Versions [, ver:268]). The second one was triggered by the query while the first one was running.
While it was syncing, I issued a SPARQL query to the Fuseki (pls note, the two consecutive patch fetching runs).
`A71396588@T000ad93b4 rdf-delta-1.1.0 % ./dcmd fuseki --port 3031 --conf fuseki-config1.ttl
[2022-08-10 12:59:56] Delta INFO Delta Patch Log Servers: [http://localhost:1066/]
[2022-08-10 12:59:56] Zone INFO Connection : /Users/A71396588/Documents/projects/rdf-delta-1.1.0/Zone1/ABC
[2022-08-10 12:59:56] Delta INFO [id:ac5a53:ABC] Sync: Versions [, ver:268]
[2022-08-10 12:59:56] Server INFO Apache Jena Fuseki 4.3.2
[2022-08-10 12:59:56] Server INFO Configuration file: fuseki-config1.ttl
[2022-08-10 12:59:56] Server INFO Path = /ds
[2022-08-10 12:59:56] Server INFO System
[2022-08-10 12:59:56] Server INFO Memory: 4,0 GiB
[2022-08-10 12:59:56] Server INFO Java: 17.0.2
[2022-08-10 12:59:56] Server INFO OS: Mac OS X 12.4 aarch64
[2022-08-10 12:59:56] Server INFO PID: 9009
[2022-08-10 12:59:56] Server INFO Start Fuseki (http=3031)
[2022-08-10 12:59:56] Fuseki INFO [1] POST http://localhost:3031/ds
[2022-08-10 12:59:56] Fuseki INFO [1] Query = SELECT ?s ?o ?p where {?s ?o ?p} LIMIT 10
[2022-08-10 12:59:56] Delta INFO [id:ac5a53:ABC] Sync: Versions [, ver:268]
[2022-08-10 12:59:58] Fuseki INFO [1] 200 OK (1,802 s)
[2022-08-10 12:59:59] Fuseki INFO [2] POST http://localhost:3031/ds
[2022-08-10 12:59:59] Fuseki INFO [2] Query = SELECT ?s ?o ?p where {?s ?o ?p} LIMIT 10
[2022-08-10 12:59:59] Fuseki INFO [2] 200 OK (8 ms)
`
PatchServer Log:
A71396588@T000ad93b4 rdf-delta-1.1.0 % ./dcmd server --store PatchStore [2022-08-10 12:59:38] Delta DEBUG Delta initialization (level 0) [2022-08-10 12:59:38] Delta DEBUG Delta initialization (level 1) [2022-08-10 12:59:38] Delta DEBUG Provider: file [2022-08-10 12:59:38] Delta DEBUG Provider: rdb [2022-08-10 12:59:38] Delta DEBUG Provider: mem [2022-08-10 12:59:38] Delta DEBUG Provider: zk [2022-08-10 12:59:38] Delta DEBUG Provider: Local [2022-08-10 12:59:38] Config DEBUG Setup for provider: local[PatchStore] [2022-08-10 12:59:38] Delta DEBUG Register datasource: id:ac5a53 [2022-08-10 12:59:38] Delta DEBUG Register datasource: id:ac5a53 [2022-08-10 12:59:38] Delta INFO Provider: Local [2022-08-10 12:59:38] Delta DEBUG [id:ac5a53, ABC, <delta:ABC>] [2022-08-10 12:59:38] Delta DEBUG DataSources: Local : [[id:ac5a53, ABC, <delta:ABC>]] [2022-08-10 12:59:38] Config INFO Delta Server port=1066 [2022-08-10 12:59:38] Delta INFO RDF Delta 1.1.0 2022-08-10T07:33:49+0000 [2022-08-10 12:59:38] Delta DEBUG Server start: port=1066 [2022-08-10 12:59:38] Delta INFO Data source: [id:ac5a53, ABC, <delta:ABC>] version [1,268] type=file [2022-08-10 12:59:56] RPC DEBUG [1] ping { } => {"value" : "2022-08-10T12:59:56.399+02:00" } [2022-08-10 12:59:56] RPC DEBUG [2] ping { } => {"value" : "2022-08-10T12:59:56.413+02:00" } [2022-08-10 12:59:56] RPC DEBUG [3] describe_datasource {"name" : "ABC" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" } [2022-08-10 12:59:56] RPC DEBUG [4] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 12:59:56] Patch INFO [5] Patch:fetch Dest=id:ac5a53, Patch=ver:1 [2022-08-10 12:59:56] Patch INFO [6] Patch:fetch Dest=id:ac5a53, Patch=ver:2 [2022-08-10 12:59:56] Patch INFO [7] Patch:fetch Dest=id:ac5a53, Patch=ver:3 [2022-08-10 12:59:56] Patch INFO [8] Patch:fetch Dest=id:ac5a53, Patch=ver:4 [2022-08-10 12:59:56] Patch INFO [9] Patch:fetch Dest=id:ac5a53, Patch=ver:5 [2022-08-10 12:59:56] Patch INFO [10] Patch:fetch Dest=id:ac5a53, Patch=ver:6 [2022-08-10 12:59:56] Patch INFO [11] Patch:fetch Dest=id:ac5a53, Patch=ver:7 [2022-08-10 12:59:56] Patch INFO [12] Patch:fetch Dest=id:ac5a53, Patch=ver:8 [2022-08-10 12:59:56] Patch INFO [13] Patch:fetch Dest=id:ac5a53, Patch=ver:9 [2022-08-10 12:59:56] Patch INFO [14] Patch:fetch Dest=id:ac5a53, Patch=ver:10 [2022-08-10 12:59:56] Patch INFO [15] Patch:fetch Dest=id:ac5a53, Patch=ver:11 [2022-08-10 12:59:56] Patch INFO [16] Patch:fetch Dest=id:ac5a53, Patch=ver:12 [2022-08-10 12:59:56] Patch INFO [17] Patch:fetch Dest=id:ac5a53, Patch=ver:13 [2022-08-10 12:59:56] Patch INFO [18] Patch:fetch Dest=id:ac5a53, Patch=ver:14 [2022-08-10 12:59:56] Patch INFO [19] Patch:fetch Dest=id:ac5a53, Patch=ver:15 [2022-08-10 12:59:56] Patch INFO [20] Patch:fetch Dest=id:ac5a53, Patch=ver:16 [2022-08-10 12:59:56] Patch INFO [21] Patch:fetch Dest=id:ac5a53, Patch=ver:17 [2022-08-10 12:59:56] Patch INFO [22] Patch:fetch Dest=id:ac5a53, Patch=ver:18 [2022-08-10 12:59:56] Patch INFO [23] Patch:fetch Dest=id:ac5a53, Patch=ver:19 [2022-08-10 12:59:56] Patch INFO [24] Patch:fetch Dest=id:ac5a53, Patch=ver:20 [2022-08-10 12:59:56] Patch INFO [25] Patch:fetch Dest=id:ac5a53, Patch=ver:21 [2022-08-10 12:59:56] Patch INFO [26] Patch:fetch Dest=id:ac5a53, Patch=ver:22 [2022-08-10 12:59:56] Patch INFO [27] Patch:fetch Dest=id:ac5a53, Patch=ver:23 [2022-08-10 12:59:56] Patch INFO [28] Patch:fetch Dest=id:ac5a53, Patch=ver:24 [2022-08-10 12:59:56] Patch INFO [29] Patch:fetch Dest=id:ac5a53, Patch=ver:25 [2022-08-10 12:59:56] Patch INFO [30] Patch:fetch Dest=id:ac5a53, Patch=ver:26 [2022-08-10 12:59:56] Patch INFO [31] Patch:fetch Dest=id:ac5a53, Patch=ver:27 [2022-08-10 12:59:56] Patch INFO [32] Patch:fetch Dest=id:ac5a53, Patch=ver:28 [2022-08-10 12:59:56] Patch INFO [33] Patch:fetch Dest=id:ac5a53, Patch=ver:29 [2022-08-10 12:59:56] Patch INFO [34] Patch:fetch Dest=id:ac5a53, Patch=ver:30 [2022-08-10 12:59:56] Patch INFO [35] Patch:fetch Dest=id:ac5a53, Patch=ver:31 [2022-08-10 12:59:56] Patch INFO [36] Patch:fetch Dest=id:ac5a53, Patch=ver:32 [2022-08-10 12:59:56] Patch INFO [37] Patch:fetch Dest=id:ac5a53, Patch=ver:33 [2022-08-10 12:59:56] Patch INFO [38] Patch:fetch Dest=id:ac5a53, Patch=ver:34 [2022-08-10 12:59:56] Patch INFO [39] Patch:fetch Dest=id:ac5a53, Patch=ver:35 [2022-08-10 12:59:56] Patch INFO [40] Patch:fetch Dest=id:ac5a53, Patch=ver:36 [2022-08-10 12:59:56] Patch INFO [41] Patch:fetch Dest=id:ac5a53, Patch=ver:37 [2022-08-10 12:59:56] Patch INFO [42] Patch:fetch Dest=id:ac5a53, Patch=ver:38 [2022-08-10 12:59:56] Patch INFO [43] Patch:fetch Dest=id:ac5a53, Patch=ver:39 [2022-08-10 12:59:56] Patch INFO [44] Patch:fetch Dest=id:ac5a53, Patch=ver:40 [2022-08-10 12:59:56] Patch INFO [45] Patch:fetch Dest=id:ac5a53, Patch=ver:41 [2022-08-10 12:59:56] Patch INFO [46] Patch:fetch Dest=id:ac5a53, Patch=ver:42 [2022-08-10 12:59:56] Patch INFO [47] Patch:fetch Dest=id:ac5a53, Patch=ver:43 [2022-08-10 12:59:56] Patch INFO [48] Patch:fetch Dest=id:ac5a53, Patch=ver:44 [2022-08-10 12:59:56] Patch INFO [49] Patch:fetch Dest=id:ac5a53, Patch=ver:45 [2022-08-10 12:59:56] Patch INFO [50] Patch:fetch Dest=id:ac5a53, Patch=ver:46 [2022-08-10 12:59:56] Patch INFO [51] Patch:fetch Dest=id:ac5a53, Patch=ver:47 [2022-08-10 12:59:56] Patch INFO [52] Patch:fetch Dest=id:ac5a53, Patch=ver:48 [2022-08-10 12:59:56] Patch INFO [53] Patch:fetch Dest=id:ac5a53, Patch=ver:49 [2022-08-10 12:59:56] Patch INFO [54] Patch:fetch Dest=id:ac5a53, Patch=ver:50 [2022-08-10 12:59:56] Patch INFO [55] Patch:fetch Dest=id:ac5a53, Patch=ver:51 [2022-08-10 12:59:56] Patch INFO [56] Patch:fetch Dest=id:ac5a53, Patch=ver:52 [2022-08-10 12:59:56] Patch INFO [57] Patch:fetch Dest=id:ac5a53, Patch=ver:53 [2022-08-10 12:59:56] Patch INFO [58] Patch:fetch Dest=id:ac5a53, Patch=ver:54 [2022-08-10 12:59:56] Patch INFO [59] Patch:fetch Dest=id:ac5a53, Patch=ver:55 [2022-08-10 12:59:56] Patch INFO [60] Patch:fetch Dest=id:ac5a53, Patch=ver:56 [2022-08-10 12:59:56] Patch INFO [61] Patch:fetch Dest=id:ac5a53, Patch=ver:57 [2022-08-10 12:59:56] Patch INFO [62] Patch:fetch Dest=id:ac5a53, Patch=ver:58 [2022-08-10 12:59:56] Patch INFO [63] Patch:fetch Dest=id:ac5a53, Patch=ver:59 [2022-08-10 12:59:56] Patch INFO [64] Patch:fetch Dest=id:ac5a53, Patch=ver:60 [2022-08-10 12:59:56] Patch INFO [65] Patch:fetch Dest=id:ac5a53, Patch=ver:61 [2022-08-10 12:59:56] Patch INFO [66] Patch:fetch Dest=id:ac5a53, Patch=ver:62 [2022-08-10 12:59:56] Patch INFO [67] Patch:fetch Dest=id:ac5a53, Patch=ver:63 [2022-08-10 12:59:56] Patch INFO [68] Patch:fetch Dest=id:ac5a53, Patch=ver:64 [2022-08-10 12:59:56] Patch INFO [69] Patch:fetch Dest=id:ac5a53, Patch=ver:65 [2022-08-10 12:59:56] Patch INFO [70] Patch:fetch Dest=id:ac5a53, Patch=ver:66 [2022-08-10 12:59:56] Patch INFO [71] Patch:fetch Dest=id:ac5a53, Patch=ver:67 [2022-08-10 12:59:56] Patch INFO [72] Patch:fetch Dest=id:ac5a53, Patch=ver:68 [2022-08-10 12:59:56] Patch INFO [73] Patch:fetch Dest=id:ac5a53, Patch=ver:69 [2022-08-10 12:59:56] Patch INFO [74] Patch:fetch Dest=id:ac5a53, Patch=ver:70 [2022-08-10 12:59:56] Patch INFO [75] Patch:fetch Dest=id:ac5a53, Patch=ver:71 [2022-08-10 12:59:56] Patch INFO [76] Patch:fetch Dest=id:ac5a53, Patch=ver:72 [2022-08-10 12:59:56] Patch INFO [77] Patch:fetch Dest=id:ac5a53, Patch=ver:73 [2022-08-10 12:59:56] Patch INFO [78] Patch:fetch Dest=id:ac5a53, Patch=ver:74 [2022-08-10 12:59:56] Patch INFO [79] Patch:fetch Dest=id:ac5a53, Patch=ver:75 [2022-08-10 12:59:56] Patch INFO [80] Patch:fetch Dest=id:ac5a53, Patch=ver:76 [2022-08-10 12:59:56] Patch INFO [81] Patch:fetch Dest=id:ac5a53, Patch=ver:77 [2022-08-10 12:59:56] Patch INFO [82] Patch:fetch Dest=id:ac5a53, Patch=ver:78 [2022-08-10 12:59:56] Patch INFO [83] Patch:fetch Dest=id:ac5a53, Patch=ver:79 [2022-08-10 12:59:56] Patch INFO [84] Patch:fetch Dest=id:ac5a53, Patch=ver:80 [2022-08-10 12:59:56] Patch INFO [85] Patch:fetch Dest=id:ac5a53, Patch=ver:81 [2022-08-10 12:59:56] Patch INFO [86] Patch:fetch Dest=id:ac5a53, Patch=ver:82 [2022-08-10 12:59:56] Patch INFO [87] Patch:fetch Dest=id:ac5a53, Patch=ver:83 [2022-08-10 12:59:56] Patch INFO [88] Patch:fetch Dest=id:ac5a53, Patch=ver:84 [2022-08-10 12:59:56] Patch INFO [89] Patch:fetch Dest=id:ac5a53, Patch=ver:85 [2022-08-10 12:59:56] Patch INFO [90] Patch:fetch Dest=id:ac5a53, Patch=ver:86 [2022-08-10 12:59:56] Patch INFO [91] Patch:fetch Dest=id:ac5a53, Patch=ver:87 [2022-08-10 12:59:56] Patch INFO [92] Patch:fetch Dest=id:ac5a53, Patch=ver:88 [2022-08-10 12:59:56] Patch INFO [93] Patch:fetch Dest=id:ac5a53, Patch=ver:89 [2022-08-10 12:59:56] Patch INFO [94] Patch:fetch Dest=id:ac5a53, Patch=ver:90 [2022-08-10 12:59:56] Patch INFO [95] Patch:fetch Dest=id:ac5a53, Patch=ver:91 [2022-08-10 12:59:56] Patch INFO [96] Patch:fetch Dest=id:ac5a53, Patch=ver:92 [2022-08-10 12:59:56] Patch INFO [97] Patch:fetch Dest=id:ac5a53, Patch=ver:93 [2022-08-10 12:59:56] Patch INFO [98] Patch:fetch Dest=id:ac5a53, Patch=ver:94 [2022-08-10 12:59:56] Patch INFO [99] Patch:fetch Dest=id:ac5a53, Patch=ver:95 [2022-08-10 12:59:56] Patch INFO [100] Patch:fetch Dest=id:ac5a53, Patch=ver:96 [2022-08-10 12:59:56] RPC DEBUG [101] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 12:59:56] Patch INFO [102] Patch:fetch Dest=id:ac5a53, Patch=ver:97 [2022-08-10 12:59:56] Patch INFO [103] Patch:fetch Dest=id:ac5a53, Patch=ver:98 [2022-08-10 12:59:56] Patch INFO [104] Patch:fetch Dest=id:ac5a53, Patch=ver:99 [2022-08-10 12:59:56] Patch INFO [105] Patch:fetch Dest=id:ac5a53, Patch=ver:100 [2022-08-10 12:59:56] Patch INFO [106] Patch:fetch Dest=id:ac5a53, Patch=ver:101 [2022-08-10 12:59:56] Patch INFO [107] Patch:fetch Dest=id:ac5a53, Patch=ver:102 [2022-08-10 12:59:56] Patch INFO [108] Patch:fetch Dest=id:ac5a53, Patch=ver:103 [2022-08-10 12:59:56] Patch INFO [109] Patch:fetch Dest=id:ac5a53, Patch=ver:104 [2022-08-10 12:59:56] Patch INFO [110] Patch:fetch Dest=id:ac5a53, Patch=ver:105 [2022-08-10 12:59:56] Patch INFO [111] Patch:fetch Dest=id:ac5a53, Patch=ver:106 [2022-08-10 12:59:56] Patch INFO [112] Patch:fetch Dest=id:ac5a53, Patch=ver:107 [2022-08-10 12:59:56] Patch INFO [113] Patch:fetch Dest=id:ac5a53, Patch=ver:108 [2022-08-10 12:59:56] Patch INFO [114] Patch:fetch Dest=id:ac5a53, Patch=ver:109 [2022-08-10 12:59:56] Patch INFO [115] Patch:fetch Dest=id:ac5a53, Patch=ver:110 [2022-08-10 12:59:56] Patch INFO [116] Patch:fetch Dest=id:ac5a53, Patch=ver:111 [2022-08-10 12:59:56] Patch INFO [117] Patch:fetch Dest=id:ac5a53, Patch=ver:112 [2022-08-10 12:59:56] Patch INFO [118] Patch:fetch Dest=id:ac5a53, Patch=ver:113 [2022-08-10 12:59:56] Patch INFO [119] Patch:fetch Dest=id:ac5a53, Patch=ver:114 [2022-08-10 12:59:56] Patch INFO [120] Patch:fetch Dest=id:ac5a53, Patch=ver:115 [2022-08-10 12:59:56] Patch INFO [121] Patch:fetch Dest=id:ac5a53, Patch=ver:116 [2022-08-10 12:59:56] Patch INFO [122] Patch:fetch Dest=id:ac5a53, Patch=ver:117 [2022-08-10 12:59:56] Patch INFO [123] Patch:fetch Dest=id:ac5a53, Patch=ver:118 [2022-08-10 12:59:56] Patch INFO [124] Patch:fetch Dest=id:ac5a53, Patch=ver:119 [2022-08-10 12:59:56] Patch INFO [125] Patch:fetch Dest=id:ac5a53, Patch=ver:120 [2022-08-10 12:59:56] Patch INFO [126] Patch:fetch Dest=id:ac5a53, Patch=ver:121 [2022-08-10 12:59:56] Patch INFO [127] Patch:fetch Dest=id:ac5a53, Patch=ver:122 [2022-08-10 12:59:56] Patch INFO [128] Patch:fetch Dest=id:ac5a53, Patch=ver:123 [2022-08-10 12:59:56] Patch INFO [129] Patch:fetch Dest=id:ac5a53, Patch=ver:124 [2022-08-10 12:59:56] Patch INFO [130] Patch:fetch Dest=id:ac5a53, Patch=ver:125 [2022-08-10 12:59:56] Patch INFO [131] Patch:fetch Dest=id:ac5a53, Patch=ver:126 [2022-08-10 12:59:56] Patch INFO [132] Patch:fetch Dest=id:ac5a53, Patch=ver:127 [2022-08-10 12:59:56] Patch INFO [133] Patch:fetch Dest=id:ac5a53, Patch=ver:128 [2022-08-10 12:59:56] Patch INFO [134] Patch:fetch Dest=id:ac5a53, Patch=ver:129 [2022-08-10 12:59:56] Patch INFO [135] Patch:fetch Dest=id:ac5a53, Patch=ver:130 [2022-08-10 12:59:56] Patch INFO [136] Patch:fetch Dest=id:ac5a53, Patch=ver:131 [2022-08-10 12:59:56] Patch INFO [137] Patch:fetch Dest=id:ac5a53, Patch=ver:132 [2022-08-10 12:59:56] Patch INFO [138] Patch:fetch Dest=id:ac5a53, Patch=ver:133 [2022-08-10 12:59:56] Patch INFO [139] Patch:fetch Dest=id:ac5a53, Patch=ver:134 [2022-08-10 12:59:56] Patch INFO [140] Patch:fetch Dest=id:ac5a53, Patch=ver:135 [2022-08-10 12:59:56] Patch INFO [141] Patch:fetch Dest=id:ac5a53, Patch=ver:136 [2022-08-10 12:59:56] Patch INFO [142] Patch:fetch Dest=id:ac5a53, Patch=ver:137 [2022-08-10 12:59:56] Patch INFO [143] Patch:fetch Dest=id:ac5a53, Patch=ver:138 [2022-08-10 12:59:56] Patch INFO [144] Patch:fetch Dest=id:ac5a53, Patch=ver:139 [2022-08-10 12:59:56] Patch INFO [145] Patch:fetch Dest=id:ac5a53, Patch=ver:140 [2022-08-10 12:59:56] Patch INFO [146] Patch:fetch Dest=id:ac5a53, Patch=ver:141 [2022-08-10 12:59:56] Patch INFO [147] Patch:fetch Dest=id:ac5a53, Patch=ver:142 [2022-08-10 12:59:56] Patch INFO [148] Patch:fetch Dest=id:ac5a53, Patch=ver:143 [2022-08-10 12:59:56] Patch INFO [149] Patch:fetch Dest=id:ac5a53, Patch=ver:144 [2022-08-10 12:59:56] Patch INFO [150] Patch:fetch Dest=id:ac5a53, Patch=ver:145 [2022-08-10 12:59:56] Patch INFO [151] Patch:fetch Dest=id:ac5a53, Patch=ver:146 [2022-08-10 12:59:56] Patch INFO [152] Patch:fetch Dest=id:ac5a53, Patch=ver:147 [2022-08-10 12:59:56] Patch INFO [153] Patch:fetch Dest=id:ac5a53, Patch=ver:148 [2022-08-10 12:59:56] Patch INFO [154] Patch:fetch Dest=id:ac5a53, Patch=ver:149 [2022-08-10 12:59:56] Patch INFO [155] Patch:fetch Dest=id:ac5a53, Patch=ver:150 [2022-08-10 12:59:56] Patch INFO [156] Patch:fetch Dest=id:ac5a53, Patch=ver:151 [2022-08-10 12:59:56] Patch INFO [157] Patch:fetch Dest=id:ac5a53, Patch=ver:152 [2022-08-10 12:59:56] Patch INFO [158] Patch:fetch Dest=id:ac5a53, Patch=ver:153 [2022-08-10 12:59:56] Patch INFO [159] Patch:fetch Dest=id:ac5a53, Patch=ver:154 [2022-08-10 12:59:56] Patch INFO [160] Patch:fetch Dest=id:ac5a53, Patch=ver:155 [2022-08-10 12:59:56] Patch INFO [161] Patch:fetch Dest=id:ac5a53, Patch=ver:156 [2022-08-10 12:59:56] Patch INFO [162] Patch:fetch Dest=id:ac5a53, Patch=ver:157 [2022-08-10 12:59:56] Patch INFO [163] Patch:fetch Dest=id:ac5a53, Patch=ver:158 [2022-08-10 12:59:56] Patch INFO [164] Patch:fetch Dest=id:ac5a53, Patch=ver:159 [2022-08-10 12:59:56] Patch INFO [165] Patch:fetch Dest=id:ac5a53, Patch=ver:160 [2022-08-10 12:59:56] Patch INFO [166] Patch:fetch Dest=id:ac5a53, Patch=ver:161 [2022-08-10 12:59:56] Patch INFO [167] Patch:fetch Dest=id:ac5a53, Patch=ver:162 [2022-08-10 12:59:56] Patch INFO [168] Patch:fetch Dest=id:ac5a53, Patch=ver:163 [2022-08-10 12:59:56] Patch INFO [169] Patch:fetch Dest=id:ac5a53, Patch=ver:164 [2022-08-10 12:59:57] Patch INFO [170] Patch:fetch Dest=id:ac5a53, Patch=ver:165 [2022-08-10 12:59:57] Patch INFO [171] Patch:fetch Dest=id:ac5a53, Patch=ver:166 [2022-08-10 12:59:57] Patch INFO [172] Patch:fetch Dest=id:ac5a53, Patch=ver:167 [2022-08-10 12:59:57] Patch INFO [173] Patch:fetch Dest=id:ac5a53, Patch=ver:168 [2022-08-10 12:59:57] Patch INFO [174] Patch:fetch Dest=id:ac5a53, Patch=ver:169 [2022-08-10 12:59:57] Patch INFO [175] Patch:fetch Dest=id:ac5a53, Patch=ver:170 [2022-08-10 12:59:57] Patch INFO [176] Patch:fetch Dest=id:ac5a53, Patch=ver:171 [2022-08-10 12:59:57] Patch INFO [177] Patch:fetch Dest=id:ac5a53, Patch=ver:172 [2022-08-10 12:59:57] Patch INFO [178] Patch:fetch Dest=id:ac5a53, Patch=ver:173 [2022-08-10 12:59:57] Patch INFO [179] Patch:fetch Dest=id:ac5a53, Patch=ver:174 [2022-08-10 12:59:57] Patch INFO [180] Patch:fetch Dest=id:ac5a53, Patch=ver:175 [2022-08-10 12:59:57] Patch INFO [181] Patch:fetch Dest=id:ac5a53, Patch=ver:176 [2022-08-10 12:59:57] Patch INFO [182] Patch:fetch Dest=id:ac5a53, Patch=ver:177 [2022-08-10 12:59:57] Patch INFO [183] Patch:fetch Dest=id:ac5a53, Patch=ver:178 [2022-08-10 12:59:57] Patch INFO [184] Patch:fetch Dest=id:ac5a53, Patch=ver:179 [2022-08-10 12:59:57] Patch INFO [185] Patch:fetch Dest=id:ac5a53, Patch=ver:180 [2022-08-10 12:59:57] Patch INFO [186] Patch:fetch Dest=id:ac5a53, Patch=ver:181 [2022-08-10 12:59:57] Patch INFO [187] Patch:fetch Dest=id:ac5a53, Patch=ver:182 [2022-08-10 12:59:57] Patch INFO [188] Patch:fetch Dest=id:ac5a53, Patch=ver:183 [2022-08-10 12:59:57] Patch INFO [189] Patch:fetch Dest=id:ac5a53, Patch=ver:184 [2022-08-10 12:59:57] Patch INFO [190] Patch:fetch Dest=id:ac5a53, Patch=ver:185 [2022-08-10 12:59:57] Patch INFO [191] Patch:fetch Dest=id:ac5a53, Patch=ver:186 [2022-08-10 12:59:57] Patch INFO [192] Patch:fetch Dest=id:ac5a53, Patch=ver:187 [2022-08-10 12:59:57] Patch INFO [193] Patch:fetch Dest=id:ac5a53, Patch=ver:188 [2022-08-10 12:59:57] Patch INFO [194] Patch:fetch Dest=id:ac5a53, Patch=ver:189 [2022-08-10 12:59:57] Patch INFO [195] Patch:fetch Dest=id:ac5a53, Patch=ver:190 [2022-08-10 12:59:57] Patch INFO [196] Patch:fetch Dest=id:ac5a53, Patch=ver:191 [2022-08-10 12:59:57] Patch INFO [197] Patch:fetch Dest=id:ac5a53, Patch=ver:192 [2022-08-10 12:59:57] Patch INFO [198] Patch:fetch Dest=id:ac5a53, Patch=ver:193 [2022-08-10 12:59:57] Patch INFO [199] Patch:fetch Dest=id:ac5a53, Patch=ver:194 [2022-08-10 12:59:57] Patch INFO [200] Patch:fetch Dest=id:ac5a53, Patch=ver:195 [2022-08-10 12:59:57] Patch INFO [201] Patch:fetch Dest=id:ac5a53, Patch=ver:196 [2022-08-10 12:59:57] Patch INFO [202] Patch:fetch Dest=id:ac5a53, Patch=ver:197 [2022-08-10 12:59:57] Patch INFO [203] Patch:fetch Dest=id:ac5a53, Patch=ver:198 [2022-08-10 12:59:57] Patch INFO [204] Patch:fetch Dest=id:ac5a53, Patch=ver:199 [2022-08-10 12:59:57] Patch INFO [205] Patch:fetch Dest=id:ac5a53, Patch=ver:200 [2022-08-10 12:59:57] Patch INFO [206] Patch:fetch Dest=id:ac5a53, Patch=ver:201 [2022-08-10 12:59:57] Patch INFO [207] Patch:fetch Dest=id:ac5a53, Patch=ver:202 [2022-08-10 12:59:57] Patch INFO [208] Patch:fetch Dest=id:ac5a53, Patch=ver:203 [2022-08-10 12:59:57] Patch INFO [209] Patch:fetch Dest=id:ac5a53, Patch=ver:204 [2022-08-10 12:59:57] Patch INFO [210] Patch:fetch Dest=id:ac5a53, Patch=ver:205 [2022-08-10 12:59:57] Patch INFO [211] Patch:fetch Dest=id:ac5a53, Patch=ver:206 [2022-08-10 12:59:57] Patch INFO [212] Patch:fetch Dest=id:ac5a53, Patch=ver:207 [2022-08-10 12:59:57] Patch INFO [213] Patch:fetch Dest=id:ac5a53, Patch=ver:208 [2022-08-10 12:59:57] Patch INFO [214] Patch:fetch Dest=id:ac5a53, Patch=ver:209 [2022-08-10 12:59:57] Patch INFO [215] Patch:fetch Dest=id:ac5a53, Patch=ver:210 [2022-08-10 12:59:57] Patch INFO [216] Patch:fetch Dest=id:ac5a53, Patch=ver:211 [2022-08-10 12:59:57] Patch INFO [217] Patch:fetch Dest=id:ac5a53, Patch=ver:212 [2022-08-10 12:59:57] Patch INFO [218] Patch:fetch Dest=id:ac5a53, Patch=ver:213 [2022-08-10 12:59:57] Patch INFO [219] Patch:fetch Dest=id:ac5a53, Patch=ver:214 [2022-08-10 12:59:57] Patch INFO [220] Patch:fetch Dest=id:ac5a53, Patch=ver:215 [2022-08-10 12:59:57] Patch INFO [221] Patch:fetch Dest=id:ac5a53, Patch=ver:216 [2022-08-10 12:59:57] Patch INFO [222] Patch:fetch Dest=id:ac5a53, Patch=ver:217 [2022-08-10 12:59:57] Patch INFO [223] Patch:fetch Dest=id:ac5a53, Patch=ver:218 [2022-08-10 12:59:57] Patch INFO [224] Patch:fetch Dest=id:ac5a53, Patch=ver:219 [2022-08-10 12:59:57] Patch INFO [225] Patch:fetch Dest=id:ac5a53, Patch=ver:220 [2022-08-10 12:59:57] Patch INFO [226] Patch:fetch Dest=id:ac5a53, Patch=ver:221 [2022-08-10 12:59:57] Patch INFO [227] Patch:fetch Dest=id:ac5a53, Patch=ver:222 [2022-08-10 12:59:57] Patch INFO [228] Patch:fetch Dest=id:ac5a53, Patch=ver:223 [2022-08-10 12:59:57] Patch INFO [229] Patch:fetch Dest=id:ac5a53, Patch=ver:224 [2022-08-10 12:59:57] Patch INFO [230] Patch:fetch Dest=id:ac5a53, Patch=ver:225 [2022-08-10 12:59:57] Patch INFO [231] Patch:fetch Dest=id:ac5a53, Patch=ver:226 [2022-08-10 12:59:57] Patch INFO [232] Patch:fetch Dest=id:ac5a53, Patch=ver:227 [2022-08-10 12:59:57] Patch INFO [233] Patch:fetch Dest=id:ac5a53, Patch=ver:228 [2022-08-10 12:59:57] Patch INFO [234] Patch:fetch Dest=id:ac5a53, Patch=ver:229 [2022-08-10 12:59:57] Patch INFO [235] Patch:fetch Dest=id:ac5a53, Patch=ver:230 [2022-08-10 12:59:57] Patch INFO [236] Patch:fetch Dest=id:ac5a53, Patch=ver:231 [2022-08-10 12:59:57] Patch INFO [237] Patch:fetch Dest=id:ac5a53, Patch=ver:232 [2022-08-10 12:59:57] Patch INFO [238] Patch:fetch Dest=id:ac5a53, Patch=ver:233 [2022-08-10 12:59:57] Patch INFO [239] Patch:fetch Dest=id:ac5a53, Patch=ver:234 [2022-08-10 12:59:57] Patch INFO [240] Patch:fetch Dest=id:ac5a53, Patch=ver:235 [2022-08-10 12:59:57] Patch INFO [241] Patch:fetch Dest=id:ac5a53, Patch=ver:236 [2022-08-10 12:59:57] Patch INFO [242] Patch:fetch Dest=id:ac5a53, Patch=ver:237 [2022-08-10 12:59:57] Patch INFO [243] Patch:fetch Dest=id:ac5a53, Patch=ver:238 [2022-08-10 12:59:57] Patch INFO [244] Patch:fetch Dest=id:ac5a53, Patch=ver:239 [2022-08-10 12:59:57] Patch INFO [245] Patch:fetch Dest=id:ac5a53, Patch=ver:240 [2022-08-10 12:59:57] Patch INFO [246] Patch:fetch Dest=id:ac5a53, Patch=ver:241 [2022-08-10 12:59:57] Patch INFO [247] Patch:fetch Dest=id:ac5a53, Patch=ver:242 [2022-08-10 12:59:57] Patch INFO [248] Patch:fetch Dest=id:ac5a53, Patch=ver:243 [2022-08-10 12:59:57] Patch INFO [249] Patch:fetch Dest=id:ac5a53, Patch=ver:244 [2022-08-10 12:59:57] Patch INFO [250] Patch:fetch Dest=id:ac5a53, Patch=ver:245 [2022-08-10 12:59:57] Patch INFO [251] Patch:fetch Dest=id:ac5a53, Patch=ver:246 [2022-08-10 12:59:57] Patch INFO [252] Patch:fetch Dest=id:ac5a53, Patch=ver:247 [2022-08-10 12:59:57] Patch INFO [253] Patch:fetch Dest=id:ac5a53, Patch=ver:248 [2022-08-10 12:59:57] Patch INFO [254] Patch:fetch Dest=id:ac5a53, Patch=ver:249 [2022-08-10 12:59:57] Patch INFO [255] Patch:fetch Dest=id:ac5a53, Patch=ver:250 [2022-08-10 12:59:57] Patch INFO [256] Patch:fetch Dest=id:ac5a53, Patch=ver:251 [2022-08-10 12:59:57] Patch INFO [257] Patch:fetch Dest=id:ac5a53, Patch=ver:252 [2022-08-10 12:59:57] Patch INFO [258] Patch:fetch Dest=id:ac5a53, Patch=ver:253 [2022-08-10 12:59:57] Patch INFO [259] Patch:fetch Dest=id:ac5a53, Patch=ver:254 [2022-08-10 12:59:57] Patch INFO [260] Patch:fetch Dest=id:ac5a53, Patch=ver:255 [2022-08-10 12:59:57] Patch INFO [261] Patch:fetch Dest=id:ac5a53, Patch=ver:256 [2022-08-10 12:59:57] Patch INFO [262] Patch:fetch Dest=id:ac5a53, Patch=ver:257 [2022-08-10 12:59:57] Patch INFO [263] Patch:fetch Dest=id:ac5a53, Patch=ver:258 [2022-08-10 12:59:57] Patch INFO [264] Patch:fetch Dest=id:ac5a53, Patch=ver:259 [2022-08-10 12:59:57] Patch INFO [265] Patch:fetch Dest=id:ac5a53, Patch=ver:260 [2022-08-10 12:59:57] Patch INFO [266] Patch:fetch Dest=id:ac5a53, Patch=ver:261 [2022-08-10 12:59:57] Patch INFO [267] Patch:fetch Dest=id:ac5a53, Patch=ver:262 [2022-08-10 12:59:57] Patch INFO [268] Patch:fetch Dest=id:ac5a53, Patch=ver:263 [2022-08-10 12:59:57] Patch INFO [269] Patch:fetch Dest=id:ac5a53, Patch=ver:264 [2022-08-10 12:59:57] Patch INFO [270] Patch:fetch Dest=id:ac5a53, Patch=ver:265 [2022-08-10 12:59:57] Patch INFO [271] Patch:fetch Dest=id:ac5a53, Patch=ver:266 [2022-08-10 12:59:57] Patch INFO [272] Patch:fetch Dest=id:ac5a53, Patch=ver:267 [2022-08-10 12:59:57] Patch INFO [273] Patch:fetch Dest=id:ac5a53, Patch=ver:268 [2022-08-10 12:59:57] Patch INFO [274] Patch:fetch Dest=id:ac5a53, Patch=ver:1 [2022-08-10 12:59:57] Patch INFO [275] Patch:fetch Dest=id:ac5a53, Patch=ver:2 [2022-08-10 12:59:57] Patch INFO [276] Patch:fetch Dest=id:ac5a53, Patch=ver:3 [2022-08-10 12:59:57] Patch INFO [277] Patch:fetch Dest=id:ac5a53, Patch=ver:4 [2022-08-10 12:59:57] Patch INFO [278] Patch:fetch Dest=id:ac5a53, Patch=ver:5 [2022-08-10 12:59:57] Patch INFO [279] Patch:fetch Dest=id:ac5a53, Patch=ver:6 [2022-08-10 12:59:57] Patch INFO [280] Patch:fetch Dest=id:ac5a53, Patch=ver:7 [2022-08-10 12:59:57] Patch INFO [281] Patch:fetch Dest=id:ac5a53, Patch=ver:8 [2022-08-10 12:59:57] Patch INFO [282] Patch:fetch Dest=id:ac5a53, Patch=ver:9 [2022-08-10 12:59:57] Patch INFO [283] Patch:fetch Dest=id:ac5a53, Patch=ver:10 [2022-08-10 12:59:57] Patch INFO [284] Patch:fetch Dest=id:ac5a53, Patch=ver:11 [2022-08-10 12:59:57] Patch INFO [285] Patch:fetch Dest=id:ac5a53, Patch=ver:12 [2022-08-10 12:59:57] Patch INFO [286] Patch:fetch Dest=id:ac5a53, Patch=ver:13 [2022-08-10 12:59:57] Patch INFO [287] Patch:fetch Dest=id:ac5a53, Patch=ver:14 [2022-08-10 12:59:57] Patch INFO [288] Patch:fetch Dest=id:ac5a53, Patch=ver:15 [2022-08-10 12:59:57] Patch INFO [289] Patch:fetch Dest=id:ac5a53, Patch=ver:16 [2022-08-10 12:59:57] Patch INFO [290] Patch:fetch Dest=id:ac5a53, Patch=ver:17 [2022-08-10 12:59:57] Patch INFO [291] Patch:fetch Dest=id:ac5a53, Patch=ver:18 [2022-08-10 12:59:57] Patch INFO [292] Patch:fetch Dest=id:ac5a53, Patch=ver:19 [2022-08-10 12:59:57] Patch INFO [293] Patch:fetch Dest=id:ac5a53, Patch=ver:20 [2022-08-10 12:59:57] Patch INFO [294] Patch:fetch Dest=id:ac5a53, Patch=ver:21 [2022-08-10 12:59:57] Patch INFO [295] Patch:fetch Dest=id:ac5a53, Patch=ver:22 [2022-08-10 12:59:57] Patch INFO [296] Patch:fetch Dest=id:ac5a53, Patch=ver:23 [2022-08-10 12:59:57] Patch INFO [297] Patch:fetch Dest=id:ac5a53, Patch=ver:24 [2022-08-10 12:59:57] Patch INFO [298] Patch:fetch Dest=id:ac5a53, Patch=ver:25 [2022-08-10 12:59:57] Patch INFO [299] Patch:fetch Dest=id:ac5a53, Patch=ver:26 [2022-08-10 12:59:57] Patch INFO [300] Patch:fetch Dest=id:ac5a53, Patch=ver:27 [2022-08-10 12:59:57] Patch INFO [301] Patch:fetch Dest=id:ac5a53, Patch=ver:28 [2022-08-10 12:59:57] Patch INFO [302] Patch:fetch Dest=id:ac5a53, Patch=ver:29 [2022-08-10 12:59:57] Patch INFO [303] Patch:fetch Dest=id:ac5a53, Patch=ver:30 [2022-08-10 12:59:57] Patch INFO [304] Patch:fetch Dest=id:ac5a53, Patch=ver:31 [2022-08-10 12:59:57] Patch INFO [305] Patch:fetch Dest=id:ac5a53, Patch=ver:32 [2022-08-10 12:59:57] Patch INFO [306] Patch:fetch Dest=id:ac5a53, Patch=ver:33 [2022-08-10 12:59:57] Patch INFO [307] Patch:fetch Dest=id:ac5a53, Patch=ver:34 [2022-08-10 12:59:57] Patch INFO [308] Patch:fetch Dest=id:ac5a53, Patch=ver:35 [2022-08-10 12:59:57] Patch INFO [309] Patch:fetch Dest=id:ac5a53, Patch=ver:36 [2022-08-10 12:59:57] Patch INFO [310] Patch:fetch Dest=id:ac5a53, Patch=ver:37 [2022-08-10 12:59:57] Patch INFO [311] Patch:fetch Dest=id:ac5a53, Patch=ver:38 [2022-08-10 12:59:57] Patch INFO [312] Patch:fetch Dest=id:ac5a53, Patch=ver:39 [2022-08-10 12:59:57] Patch INFO [313] Patch:fetch Dest=id:ac5a53, Patch=ver:40 [2022-08-10 12:59:57] Patch INFO [314] Patch:fetch Dest=id:ac5a53, Patch=ver:41 [2022-08-10 12:59:57] Patch INFO [315] Patch:fetch Dest=id:ac5a53, Patch=ver:42 [2022-08-10 12:59:57] Patch INFO [316] Patch:fetch Dest=id:ac5a53, Patch=ver:43 [2022-08-10 12:59:57] Patch INFO [317] Patch:fetch Dest=id:ac5a53, Patch=ver:44 [2022-08-10 12:59:57] Patch INFO [318] Patch:fetch Dest=id:ac5a53, Patch=ver:45 [2022-08-10 12:59:57] Patch INFO [319] Patch:fetch Dest=id:ac5a53, Patch=ver:46 [2022-08-10 12:59:57] Patch INFO [320] Patch:fetch Dest=id:ac5a53, Patch=ver:47 [2022-08-10 12:59:57] Patch INFO [321] Patch:fetch Dest=id:ac5a53, Patch=ver:48 [2022-08-10 12:59:57] Patch INFO [322] Patch:fetch Dest=id:ac5a53, Patch=ver:49 [2022-08-10 12:59:57] Patch INFO [323] Patch:fetch Dest=id:ac5a53, Patch=ver:50 [2022-08-10 12:59:57] Patch INFO [324] Patch:fetch Dest=id:ac5a53, Patch=ver:51 [2022-08-10 12:59:57] Patch INFO [325] Patch:fetch Dest=id:ac5a53, Patch=ver:52 [2022-08-10 12:59:57] Patch INFO [326] Patch:fetch Dest=id:ac5a53, Patch=ver:53 [2022-08-10 12:59:57] Patch INFO [327] Patch:fetch Dest=id:ac5a53, Patch=ver:54 [2022-08-10 12:59:57] Patch INFO [328] Patch:fetch Dest=id:ac5a53, Patch=ver:55 [2022-08-10 12:59:57] Patch INFO [329] Patch:fetch Dest=id:ac5a53, Patch=ver:56 [2022-08-10 12:59:57] Patch INFO [330] Patch:fetch Dest=id:ac5a53, Patch=ver:57 [2022-08-10 12:59:57] Patch INFO [331] Patch:fetch Dest=id:ac5a53, Patch=ver:58 [2022-08-10 12:59:57] Patch INFO [332] Patch:fetch Dest=id:ac5a53, Patch=ver:59 [2022-08-10 12:59:57] Patch INFO [333] Patch:fetch Dest=id:ac5a53, Patch=ver:60 [2022-08-10 12:59:57] Patch INFO [334] Patch:fetch Dest=id:ac5a53, Patch=ver:61 [2022-08-10 12:59:57] Patch INFO [335] Patch:fetch Dest=id:ac5a53, Patch=ver:62 [2022-08-10 12:59:57] Patch INFO [336] Patch:fetch Dest=id:ac5a53, Patch=ver:63 [2022-08-10 12:59:57] Patch INFO [337] Patch:fetch Dest=id:ac5a53, Patch=ver:64 [2022-08-10 12:59:57] Patch INFO [338] Patch:fetch Dest=id:ac5a53, Patch=ver:65 [2022-08-10 12:59:57] Patch INFO [339] Patch:fetch Dest=id:ac5a53, Patch=ver:66 [2022-08-10 12:59:57] Patch INFO [340] Patch:fetch Dest=id:ac5a53, Patch=ver:67 [2022-08-10 12:59:57] Patch INFO [341] Patch:fetch Dest=id:ac5a53, Patch=ver:68 [2022-08-10 12:59:57] Patch INFO [342] Patch:fetch Dest=id:ac5a53, Patch=ver:69 [2022-08-10 12:59:57] Patch INFO [343] Patch:fetch Dest=id:ac5a53, Patch=ver:70 [2022-08-10 12:59:57] Patch INFO [344] Patch:fetch Dest=id:ac5a53, Patch=ver:71 [2022-08-10 12:59:57] Patch INFO [345] Patch:fetch Dest=id:ac5a53, Patch=ver:72 [2022-08-10 12:59:57] Patch INFO [346] Patch:fetch Dest=id:ac5a53, Patch=ver:73 [2022-08-10 12:59:57] Patch INFO [347] Patch:fetch Dest=id:ac5a53, Patch=ver:74 [2022-08-10 12:59:57] Patch INFO [348] Patch:fetch Dest=id:ac5a53, Patch=ver:75 [2022-08-10 12:59:57] Patch INFO [349] Patch:fetch Dest=id:ac5a53, Patch=ver:76 [2022-08-10 12:59:57] Patch INFO [350] Patch:fetch Dest=id:ac5a53, Patch=ver:77 [2022-08-10 12:59:57] Patch INFO [351] Patch:fetch Dest=id:ac5a53, Patch=ver:78 [2022-08-10 12:59:57] Patch INFO [352] Patch:fetch Dest=id:ac5a53, Patch=ver:79 [2022-08-10 12:59:57] Patch INFO [353] Patch:fetch Dest=id:ac5a53, Patch=ver:80 [2022-08-10 12:59:57] Patch INFO [354] Patch:fetch Dest=id:ac5a53, Patch=ver:81 [2022-08-10 12:59:57] Patch INFO [355] Patch:fetch Dest=id:ac5a53, Patch=ver:82 [2022-08-10 12:59:57] Patch INFO [356] Patch:fetch Dest=id:ac5a53, Patch=ver:83 [2022-08-10 12:59:57] Patch INFO [357] Patch:fetch Dest=id:ac5a53, Patch=ver:84 [2022-08-10 12:59:57] Patch INFO [358] Patch:fetch Dest=id:ac5a53, Patch=ver:85 [2022-08-10 12:59:57] Patch INFO [359] Patch:fetch Dest=id:ac5a53, Patch=ver:86 [2022-08-10 12:59:57] Patch INFO [360] Patch:fetch Dest=id:ac5a53, Patch=ver:87 [2022-08-10 12:59:57] Patch INFO [361] Patch:fetch Dest=id:ac5a53, Patch=ver:88 [2022-08-10 12:59:57] Patch INFO [362] Patch:fetch Dest=id:ac5a53, Patch=ver:89 [2022-08-10 12:59:57] Patch INFO [363] Patch:fetch Dest=id:ac5a53, Patch=ver:90 [2022-08-10 12:59:57] Patch INFO [364] Patch:fetch Dest=id:ac5a53, Patch=ver:91 [2022-08-10 12:59:57] Patch INFO [365] Patch:fetch Dest=id:ac5a53, Patch=ver:92 [2022-08-10 12:59:57] Patch INFO [366] Patch:fetch Dest=id:ac5a53, Patch=ver:93 [2022-08-10 12:59:57] Patch INFO [367] Patch:fetch Dest=id:ac5a53, Patch=ver:94 [2022-08-10 12:59:57] Patch INFO [368] Patch:fetch Dest=id:ac5a53, Patch=ver:95 [2022-08-10 12:59:57] Patch INFO [369] Patch:fetch Dest=id:ac5a53, Patch=ver:96 [2022-08-10 12:59:57] Patch INFO [370] Patch:fetch Dest=id:ac5a53, Patch=ver:97 [2022-08-10 12:59:57] Patch INFO [371] Patch:fetch Dest=id:ac5a53, Patch=ver:98 [2022-08-10 12:59:57] Patch INFO [372] Patch:fetch Dest=id:ac5a53, Patch=ver:99 [2022-08-10 12:59:57] Patch INFO [373] Patch:fetch Dest=id:ac5a53, Patch=ver:100 [2022-08-10 12:59:57] Patch INFO [374] Patch:fetch Dest=id:ac5a53, Patch=ver:101 [2022-08-10 12:59:57] Patch INFO [375] Patch:fetch Dest=id:ac5a53, Patch=ver:102 [2022-08-10 12:59:57] Patch INFO [376] Patch:fetch Dest=id:ac5a53, Patch=ver:103 [2022-08-10 12:59:57] Patch INFO [377] Patch:fetch Dest=id:ac5a53, Patch=ver:104 [2022-08-10 12:59:57] Patch INFO [378] Patch:fetch Dest=id:ac5a53, Patch=ver:105 [2022-08-10 12:59:57] Patch INFO [379] Patch:fetch Dest=id:ac5a53, Patch=ver:106 [2022-08-10 12:59:57] Patch INFO [380] Patch:fetch Dest=id:ac5a53, Patch=ver:107 [2022-08-10 12:59:57] Patch INFO [381] Patch:fetch Dest=id:ac5a53, Patch=ver:108 [2022-08-10 12:59:57] Patch INFO [382] Patch:fetch Dest=id:ac5a53, Patch=ver:109 [2022-08-10 12:59:57] Patch INFO [383] Patch:fetch Dest=id:ac5a53, Patch=ver:110 [2022-08-10 12:59:57] Patch INFO [384] Patch:fetch Dest=id:ac5a53, Patch=ver:111 [2022-08-10 12:59:57] Patch INFO [385] Patch:fetch Dest=id:ac5a53, Patch=ver:112 [2022-08-10 12:59:57] Patch INFO [386] Patch:fetch Dest=id:ac5a53, Patch=ver:113 [2022-08-10 12:59:57] Patch INFO [387] Patch:fetch Dest=id:ac5a53, Patch=ver:114 [2022-08-10 12:59:57] Patch INFO [388] Patch:fetch Dest=id:ac5a53, Patch=ver:115 [2022-08-10 12:59:57] Patch INFO [389] Patch:fetch Dest=id:ac5a53, Patch=ver:116 [2022-08-10 12:59:57] Patch INFO [390] Patch:fetch Dest=id:ac5a53, Patch=ver:117 [2022-08-10 12:59:57] Patch INFO [391] Patch:fetch Dest=id:ac5a53, Patch=ver:118 [2022-08-10 12:59:57] Patch INFO [392] Patch:fetch Dest=id:ac5a53, Patch=ver:119 [2022-08-10 12:59:57] Patch INFO [393] Patch:fetch Dest=id:ac5a53, Patch=ver:120 [2022-08-10 12:59:57] Patch INFO [394] Patch:fetch Dest=id:ac5a53, Patch=ver:121 [2022-08-10 12:59:57] Patch INFO [395] Patch:fetch Dest=id:ac5a53, Patch=ver:122 [2022-08-10 12:59:57] Patch INFO [396] Patch:fetch Dest=id:ac5a53, Patch=ver:123 [2022-08-10 12:59:57] Patch INFO [397] Patch:fetch Dest=id:ac5a53, Patch=ver:124 [2022-08-10 12:59:57] Patch INFO [398] Patch:fetch Dest=id:ac5a53, Patch=ver:125 [2022-08-10 12:59:57] Patch INFO [399] Patch:fetch Dest=id:ac5a53, Patch=ver:126 [2022-08-10 12:59:57] Patch INFO [400] Patch:fetch Dest=id:ac5a53, Patch=ver:127 [2022-08-10 12:59:57] Patch INFO [401] Patch:fetch Dest=id:ac5a53, Patch=ver:128 [2022-08-10 12:59:57] Patch INFO [402] Patch:fetch Dest=id:ac5a53, Patch=ver:129 [2022-08-10 12:59:57] Patch INFO [403] Patch:fetch Dest=id:ac5a53, Patch=ver:130 [2022-08-10 12:59:57] Patch INFO [404] Patch:fetch Dest=id:ac5a53, Patch=ver:131 [2022-08-10 12:59:57] Patch INFO [405] Patch:fetch Dest=id:ac5a53, Patch=ver:132 [2022-08-10 12:59:57] Patch INFO [406] Patch:fetch Dest=id:ac5a53, Patch=ver:133 [2022-08-10 12:59:57] Patch INFO [407] Patch:fetch Dest=id:ac5a53, Patch=ver:134 [2022-08-10 12:59:57] Patch INFO [408] Patch:fetch Dest=id:ac5a53, Patch=ver:135 [2022-08-10 12:59:57] Patch INFO [409] Patch:fetch Dest=id:ac5a53, Patch=ver:136 [2022-08-10 12:59:57] Patch INFO [410] Patch:fetch Dest=id:ac5a53, Patch=ver:137 [2022-08-10 12:59:57] Patch INFO [411] Patch:fetch Dest=id:ac5a53, Patch=ver:138 [2022-08-10 12:59:57] Patch INFO [412] Patch:fetch Dest=id:ac5a53, Patch=ver:139 [2022-08-10 12:59:57] Patch INFO [413] Patch:fetch Dest=id:ac5a53, Patch=ver:140 [2022-08-10 12:59:57] Patch INFO [414] Patch:fetch Dest=id:ac5a53, Patch=ver:141 [2022-08-10 12:59:57] Patch INFO [415] Patch:fetch Dest=id:ac5a53, Patch=ver:142 [2022-08-10 12:59:57] Patch INFO [416] Patch:fetch Dest=id:ac5a53, Patch=ver:143 [2022-08-10 12:59:57] Patch INFO [417] Patch:fetch Dest=id:ac5a53, Patch=ver:144 [2022-08-10 12:59:57] Patch INFO [418] Patch:fetch Dest=id:ac5a53, Patch=ver:145 [2022-08-10 12:59:57] Patch INFO [419] Patch:fetch Dest=id:ac5a53, Patch=ver:146 [2022-08-10 12:59:57] Patch INFO [420] Patch:fetch Dest=id:ac5a53, Patch=ver:147 [2022-08-10 12:59:57] Patch INFO [421] Patch:fetch Dest=id:ac5a53, Patch=ver:148 [2022-08-10 12:59:57] Patch INFO [422] Patch:fetch Dest=id:ac5a53, Patch=ver:149 [2022-08-10 12:59:57] Patch INFO [423] Patch:fetch Dest=id:ac5a53, Patch=ver:150 [2022-08-10 12:59:57] Patch INFO [424] Patch:fetch Dest=id:ac5a53, Patch=ver:151 [2022-08-10 12:59:57] Patch INFO [425] Patch:fetch Dest=id:ac5a53, Patch=ver:152 [2022-08-10 12:59:57] Patch INFO [426] Patch:fetch Dest=id:ac5a53, Patch=ver:153 [2022-08-10 12:59:57] Patch INFO [427] Patch:fetch Dest=id:ac5a53, Patch=ver:154 [2022-08-10 12:59:57] Patch INFO [428] Patch:fetch Dest=id:ac5a53, Patch=ver:155 [2022-08-10 12:59:57] Patch INFO [429] Patch:fetch Dest=id:ac5a53, Patch=ver:156 [2022-08-10 12:59:57] Patch INFO [430] Patch:fetch Dest=id:ac5a53, Patch=ver:157 [2022-08-10 12:59:57] Patch INFO [431] Patch:fetch Dest=id:ac5a53, Patch=ver:158 [2022-08-10 12:59:57] Patch INFO [432] Patch:fetch Dest=id:ac5a53, Patch=ver:159 [2022-08-10 12:59:57] Patch INFO [433] Patch:fetch Dest=id:ac5a53, Patch=ver:160 [2022-08-10 12:59:57] Patch INFO [434] Patch:fetch Dest=id:ac5a53, Patch=ver:161 [2022-08-10 12:59:58] Patch INFO [435] Patch:fetch Dest=id:ac5a53, Patch=ver:162 [2022-08-10 12:59:58] Patch INFO [436] Patch:fetch Dest=id:ac5a53, Patch=ver:163 [2022-08-10 12:59:58] Patch INFO [437] Patch:fetch Dest=id:ac5a53, Patch=ver:164 [2022-08-10 12:59:58] Patch INFO [438] Patch:fetch Dest=id:ac5a53, Patch=ver:165 [2022-08-10 12:59:58] Patch INFO [439] Patch:fetch Dest=id:ac5a53, Patch=ver:166 [2022-08-10 12:59:58] Patch INFO [440] Patch:fetch Dest=id:ac5a53, Patch=ver:167 [2022-08-10 12:59:58] Patch INFO [441] Patch:fetch Dest=id:ac5a53, Patch=ver:168 [2022-08-10 12:59:58] Patch INFO [442] Patch:fetch Dest=id:ac5a53, Patch=ver:169 [2022-08-10 12:59:58] Patch INFO [443] Patch:fetch Dest=id:ac5a53, Patch=ver:170 [2022-08-10 12:59:58] Patch INFO [444] Patch:fetch Dest=id:ac5a53, Patch=ver:171 [2022-08-10 12:59:58] Patch INFO [445] Patch:fetch Dest=id:ac5a53, Patch=ver:172 [2022-08-10 12:59:58] Patch INFO [446] Patch:fetch Dest=id:ac5a53, Patch=ver:173 [2022-08-10 12:59:58] Patch INFO [447] Patch:fetch Dest=id:ac5a53, Patch=ver:174 [2022-08-10 12:59:58] Patch INFO [448] Patch:fetch Dest=id:ac5a53, Patch=ver:175 [2022-08-10 12:59:58] Patch INFO [449] Patch:fetch Dest=id:ac5a53, Patch=ver:176 [2022-08-10 12:59:58] Patch INFO [450] Patch:fetch Dest=id:ac5a53, Patch=ver:177 [2022-08-10 12:59:58] Patch INFO [451] Patch:fetch Dest=id:ac5a53, Patch=ver:178 [2022-08-10 12:59:58] Patch INFO [452] Patch:fetch Dest=id:ac5a53, Patch=ver:179 [2022-08-10 12:59:58] Patch INFO [453] Patch:fetch Dest=id:ac5a53, Patch=ver:180 [2022-08-10 12:59:58] Patch INFO [454] Patch:fetch Dest=id:ac5a53, Patch=ver:181 [2022-08-10 12:59:58] Patch INFO [455] Patch:fetch Dest=id:ac5a53, Patch=ver:182 [2022-08-10 12:59:58] Patch INFO [456] Patch:fetch Dest=id:ac5a53, Patch=ver:183 [2022-08-10 12:59:58] Patch INFO [457] Patch:fetch Dest=id:ac5a53, Patch=ver:184 [2022-08-10 12:59:58] Patch INFO [458] Patch:fetch Dest=id:ac5a53, Patch=ver:185 [2022-08-10 12:59:58] Patch INFO [459] Patch:fetch Dest=id:ac5a53, Patch=ver:186 [2022-08-10 12:59:58] Patch INFO [460] Patch:fetch Dest=id:ac5a53, Patch=ver:187 [2022-08-10 12:59:58] Patch INFO [461] Patch:fetch Dest=id:ac5a53, Patch=ver:188 [2022-08-10 12:59:58] Patch INFO [462] Patch:fetch Dest=id:ac5a53, Patch=ver:189 [2022-08-10 12:59:58] Patch INFO [463] Patch:fetch Dest=id:ac5a53, Patch=ver:190 [2022-08-10 12:59:58] Patch INFO [464] Patch:fetch Dest=id:ac5a53, Patch=ver:191 [2022-08-10 12:59:58] Patch INFO [465] Patch:fetch Dest=id:ac5a53, Patch=ver:192 [2022-08-10 12:59:58] Patch INFO [466] Patch:fetch Dest=id:ac5a53, Patch=ver:193 [2022-08-10 12:59:58] Patch INFO [467] Patch:fetch Dest=id:ac5a53, Patch=ver:194 [2022-08-10 12:59:58] Patch INFO [468] Patch:fetch Dest=id:ac5a53, Patch=ver:195 [2022-08-10 12:59:58] Patch INFO [469] Patch:fetch Dest=id:ac5a53, Patch=ver:196 [2022-08-10 12:59:58] Patch INFO [470] Patch:fetch Dest=id:ac5a53, Patch=ver:197 [2022-08-10 12:59:58] Patch INFO [471] Patch:fetch Dest=id:ac5a53, Patch=ver:198 [2022-08-10 12:59:58] Patch INFO [472] Patch:fetch Dest=id:ac5a53, Patch=ver:199 [2022-08-10 12:59:58] Patch INFO [473] Patch:fetch Dest=id:ac5a53, Patch=ver:200 [2022-08-10 12:59:58] Patch INFO [474] Patch:fetch Dest=id:ac5a53, Patch=ver:201 [2022-08-10 12:59:58] Patch INFO [475] Patch:fetch Dest=id:ac5a53, Patch=ver:202 [2022-08-10 12:59:58] Patch INFO [476] Patch:fetch Dest=id:ac5a53, Patch=ver:203 [2022-08-10 12:59:58] Patch INFO [477] Patch:fetch Dest=id:ac5a53, Patch=ver:204 [2022-08-10 12:59:58] Patch INFO [478] Patch:fetch Dest=id:ac5a53, Patch=ver:205 [2022-08-10 12:59:58] Patch INFO [479] Patch:fetch Dest=id:ac5a53, Patch=ver:206 [2022-08-10 12:59:58] Patch INFO [480] Patch:fetch Dest=id:ac5a53, Patch=ver:207 [2022-08-10 12:59:58] Patch INFO [481] Patch:fetch Dest=id:ac5a53, Patch=ver:208 [2022-08-10 12:59:58] Patch INFO [482] Patch:fetch Dest=id:ac5a53, Patch=ver:209 [2022-08-10 12:59:58] Patch INFO [483] Patch:fetch Dest=id:ac5a53, Patch=ver:210 [2022-08-10 12:59:58] Patch INFO [484] Patch:fetch Dest=id:ac5a53, Patch=ver:211 [2022-08-10 12:59:58] Patch INFO [485] Patch:fetch Dest=id:ac5a53, Patch=ver:212 [2022-08-10 12:59:58] Patch INFO [486] Patch:fetch Dest=id:ac5a53, Patch=ver:213 [2022-08-10 12:59:58] Patch INFO [487] Patch:fetch Dest=id:ac5a53, Patch=ver:214 [2022-08-10 12:59:58] Patch INFO [488] Patch:fetch Dest=id:ac5a53, Patch=ver:215 [2022-08-10 12:59:58] Patch INFO [489] Patch:fetch Dest=id:ac5a53, Patch=ver:216 [2022-08-10 12:59:58] Patch INFO [490] Patch:fetch Dest=id:ac5a53, Patch=ver:217 [2022-08-10 12:59:58] Patch INFO [491] Patch:fetch Dest=id:ac5a53, Patch=ver:218 [2022-08-10 12:59:58] Patch INFO [492] Patch:fetch Dest=id:ac5a53, Patch=ver:219 [2022-08-10 12:59:58] Patch INFO [493] Patch:fetch Dest=id:ac5a53, Patch=ver:220 [2022-08-10 12:59:58] Patch INFO [494] Patch:fetch Dest=id:ac5a53, Patch=ver:221 [2022-08-10 12:59:58] Patch INFO [495] Patch:fetch Dest=id:ac5a53, Patch=ver:222 [2022-08-10 12:59:58] Patch INFO [496] Patch:fetch Dest=id:ac5a53, Patch=ver:223 [2022-08-10 12:59:58] Patch INFO [497] Patch:fetch Dest=id:ac5a53, Patch=ver:224 [2022-08-10 12:59:58] Patch INFO [498] Patch:fetch Dest=id:ac5a53, Patch=ver:225 [2022-08-10 12:59:58] Patch INFO [499] Patch:fetch Dest=id:ac5a53, Patch=ver:226 [2022-08-10 12:59:58] Patch INFO [500] Patch:fetch Dest=id:ac5a53, Patch=ver:227 [2022-08-10 12:59:58] Patch INFO [501] Patch:fetch Dest=id:ac5a53, Patch=ver:228 [2022-08-10 12:59:58] Patch INFO [502] Patch:fetch Dest=id:ac5a53, Patch=ver:229 [2022-08-10 12:59:58] Patch INFO [503] Patch:fetch Dest=id:ac5a53, Patch=ver:230 [2022-08-10 12:59:58] Patch INFO [504] Patch:fetch Dest=id:ac5a53, Patch=ver:231 [2022-08-10 12:59:58] Patch INFO [505] Patch:fetch Dest=id:ac5a53, Patch=ver:232 [2022-08-10 12:59:58] Patch INFO [506] Patch:fetch Dest=id:ac5a53, Patch=ver:233 [2022-08-10 12:59:58] Patch INFO [507] Patch:fetch Dest=id:ac5a53, Patch=ver:234 [2022-08-10 12:59:58] Patch INFO [508] Patch:fetch Dest=id:ac5a53, Patch=ver:235 [2022-08-10 12:59:58] Patch INFO [509] Patch:fetch Dest=id:ac5a53, Patch=ver:236 [2022-08-10 12:59:58] Patch INFO [510] Patch:fetch Dest=id:ac5a53, Patch=ver:237 [2022-08-10 12:59:58] Patch INFO [511] Patch:fetch Dest=id:ac5a53, Patch=ver:238 [2022-08-10 12:59:58] Patch INFO [512] Patch:fetch Dest=id:ac5a53, Patch=ver:239 [2022-08-10 12:59:58] Patch INFO [513] Patch:fetch Dest=id:ac5a53, Patch=ver:240 [2022-08-10 12:59:58] Patch INFO [514] Patch:fetch Dest=id:ac5a53, Patch=ver:241 [2022-08-10 12:59:58] Patch INFO [515] Patch:fetch Dest=id:ac5a53, Patch=ver:242 [2022-08-10 12:59:58] Patch INFO [516] Patch:fetch Dest=id:ac5a53, Patch=ver:243 [2022-08-10 12:59:58] Patch INFO [517] Patch:fetch Dest=id:ac5a53, Patch=ver:244 [2022-08-10 12:59:58] Patch INFO [518] Patch:fetch Dest=id:ac5a53, Patch=ver:245 [2022-08-10 12:59:58] Patch INFO [519] Patch:fetch Dest=id:ac5a53, Patch=ver:246 [2022-08-10 12:59:58] Patch INFO [520] Patch:fetch Dest=id:ac5a53, Patch=ver:247 [2022-08-10 12:59:58] Patch INFO [521] Patch:fetch Dest=id:ac5a53, Patch=ver:248 [2022-08-10 12:59:58] Patch INFO [522] Patch:fetch Dest=id:ac5a53, Patch=ver:249 [2022-08-10 12:59:58] Patch INFO [523] Patch:fetch Dest=id:ac5a53, Patch=ver:250 [2022-08-10 12:59:58] Patch INFO [524] Patch:fetch Dest=id:ac5a53, Patch=ver:251 [2022-08-10 12:59:58] Patch INFO [525] Patch:fetch Dest=id:ac5a53, Patch=ver:252 [2022-08-10 12:59:58] Patch INFO [526] Patch:fetch Dest=id:ac5a53, Patch=ver:253 [2022-08-10 12:59:58] Patch INFO [527] Patch:fetch Dest=id:ac5a53, Patch=ver:254 [2022-08-10 12:59:58] Patch INFO [528] Patch:fetch Dest=id:ac5a53, Patch=ver:255 [2022-08-10 12:59:58] Patch INFO [529] Patch:fetch Dest=id:ac5a53, Patch=ver:256 [2022-08-10 12:59:58] Patch INFO [530] Patch:fetch Dest=id:ac5a53, Patch=ver:257 [2022-08-10 12:59:58] Patch INFO [531] Patch:fetch Dest=id:ac5a53, Patch=ver:258 [2022-08-10 12:59:58] Patch INFO [532] Patch:fetch Dest=id:ac5a53, Patch=ver:259 [2022-08-10 12:59:58] Patch INFO [533] Patch:fetch Dest=id:ac5a53, Patch=ver:260 [2022-08-10 12:59:58] Patch INFO [534] Patch:fetch Dest=id:ac5a53, Patch=ver:261 [2022-08-10 12:59:58] Patch INFO [535] Patch:fetch Dest=id:ac5a53, Patch=ver:262 [2022-08-10 12:59:58] Patch INFO [536] Patch:fetch Dest=id:ac5a53, Patch=ver:263 [2022-08-10 12:59:58] Patch INFO [537] Patch:fetch Dest=id:ac5a53, Patch=ver:264 [2022-08-10 12:59:58] Patch INFO [538] Patch:fetch Dest=id:ac5a53, Patch=ver:265 [2022-08-10 12:59:58] Patch INFO [539] Patch:fetch Dest=id:ac5a53, Patch=ver:266 [2022-08-10 12:59:58] Patch INFO [540] Patch:fetch Dest=id:ac5a53, Patch=ver:267 [2022-08-10 12:59:58] Patch INFO [541] Patch:fetch Dest=id:ac5a53, Patch=ver:268 [2022-08-10 12:59:59] RPC DEBUG [542] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 13:04:56] RPC DEBUG [543] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 13:09:56] RPC DEBUG [544] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 13:14:56] RPC DEBUG [545] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 13:19:56] RPC DEBUG [546] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 13:24:56] RPC DEBUG [547] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 13:29:56] RPC DEBUG [548] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 13:34:56] RPC DEBUG [549] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 13:39:56] RPC DEBUG [550] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 13:44:56] RPC DEBUG [551] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 13:49:56] RPC DEBUG [552] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 13:54:56] RPC DEBUG [553] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 13:59:56] RPC DEBUG [554] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 14:04:56] RPC DEBUG [555] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 14:09:56] RPC DEBUG [556] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 14:14:56] RPC DEBUG [557] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 14:19:56] RPC DEBUG [558] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 14:24:56] RPC DEBUG [559] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 14:29:56] RPC DEBUG [560] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 14:34:56] RPC DEBUG [561] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" } [2022-08-10 14:39:56] RPC DEBUG [562] describe_log {"datasource" : "ac5a5305-91c3-4ac5-a266-45c3284cfafe" } => { "id" : "id:ac5a5305-91c3-4ac5-a266-45c3284cfafe" , "name" : "ABC" , "uri" : "delta:ABC" , "min_version" : 1 , "max_version" : 268 , "latest" : "id:4b770a10-462d-4f96-8568-a6745ae77545" }
Query:
`curl -X POST http://localhost:3031/ds -H "Content-Type: application/x-www-form-urlencoded" -H "Accept:application/sparql-results+json" --data-urlencode 'format=json' --data-urlencode 'query=SELECT ?s ?o ?p where {?s ?o ?p} LIMIT 10 '
`
Fuseki config:
more fuseki-config1.ttl
## Licensed under the terms of http://www.apache.org/licenses/LICENSE-2.0
## Example RDF Delta + Fuseki configuration.
PREFIX : <#>
PREFIX fuseki: <http://jena.apache.org/fuseki#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX ja: <http://jena.hpl.hp.com/2005/11/Assembler#>
PREFIX delta: <http://jena.apache.org/rdf-delta#>
[] rdf:type fuseki:Server ;
.
<#service1> rdf:type fuseki:Service ;
fuseki:name "ds" ;
fuseki:serviceQuery "sparql" ;
fuseki:serviceQuery "query" ;
fuseki:serviceUpdate "update" ;
fuseki:serviceUpload "upload" ;
fuseki:serviceReadWriteGraphStore "data" ;
fuseki:serviceReadGraphStore "get" ;
fuseki:dataset <#dataset> ;
.
<#dataset> rdf:type delta:DeltaDataset ;
delta:changes "http://localhost:1066/" ;
delta:patchlog "ABC";
##delta:storage "mem";
## Other storage options:
delta:storage "tdb2";
## delta:storage "tdb";
## ## Fuskei server, Delta client area - must not be shared between servers.
delta:zone "Zone1";
Hi @tomkxy
Thanks for that. I think I know where to look.
Not directly related:
When you restart Fuseki servers, is it on a clean machine?
Or is it with a copy of the the database and an existing zone file?
Hi @afs,
the Fuseki servers are running as stateful set in a Kubernetes cluster with a PVC mounted where the database resides. The database and the ZONE files are usually filled.
With respect to syncing, we see that this is being triggered by something which we don't understand yet. My suspicion is that it is due to communication issues with the patchlog servers. We even might have the situation that Kubernetes decides to re-start a Fuseki server.
Anyhow, thinking about that: If a Fuseki node is in initial sync it is not responsive (not sure how it is if it if just catching up for a couple of patches). In a HA setup with load balancers in front of the Fusekis that is really an issue (since this usually takes longer). I think, the Fuseki server should have an endpoint which returns whether they are able to process requests. So if the Fuseki goes into an initial sync, the endpoint would indicate that, e.g. returning a 5xx or 4xx. The implementation must be very lightweight since the endpoint would be called frequently.
This could then be easily by a load balancer to determine whether the node can process requests.e e.g. in Kubernetes a readiness probe could be configured. Kubernetes would not direct any traffic to a pod which is not in ready state. What do you think about it?
Hi @afs update to the sync topic:
I could verify now that the same behavior (triggering another sync) can be observed when the Fuseki is syncing from any version number.
Hi @tomkxy
I have found the place where overlapping sync's occur. With long initial sync, that's going to cause overlap when the first requests come in.
Thought - if the problem of #154 is that patch servers are not in-step or close, then there may be large jumps especially around restart because a Fuseki server may be allocated to a different patch server.
The issue about being unresponsive to requests if starting with a very large sync - certainly something can be done. Please raise a separate issue for this. There are choices - whether to build into Fuseki or have a separate subcommand of dcmd to sync the database before starting Fuseki. The latter would be useful to keep a "nearly ready-to-go" database copy for example, sync'ing it once a day and is not fronted by Fuseki.
Each Fuseki server should have it's own database storage and zone. There is some process checking but if PVCs are shared, the single-copy lock on a running TDB database can't be guaranteed.
Hi @afs ,
thanks for looking into that.
The issue about being unresponsive to requests if starting with a very large sync - certainly something can be done. Please raise a separate issue for this.
I raised a new issue. However, as long as we cannot avoid that a re-sync is just started in the middle of its operation, a separate sync would not help us.
Thought - if the problem of #154 is that patch servers are not in-step or close, then there may be large jumps especially around restart because a Fuseki server may be allocated to a different patch server.
What happens if a Fuseki server is allocated to a different patch server? Why could cause that a problem.
I am asking because in our setup, the PatchLog Servers are setup as a service (3 instances are started). The Fusekis will communicate to them through a load balancer. Could this be a or the problem??
Because what we see with that initial sync is that it happens even so the latest version number in the state file says that the Fuseki is up to date, but then for whatever reason it resets the version number and starts the re-sync.
Each Fuseki server should have it's own database storage and zone. There is some process checking but if PVCs are shared, the single-copy lock on a running TDB database can't be guaranteed.
The Fusekis are all having a separate db and ZONE file.
But as I wrote above, the PatchLog servers are configure as a service which a load balancer in front of it. The PatchLog servers have their own Zookeeper file mounted through a PVC but are sharing the S3 based patch file .
Release 1.1.2 done - contains this fix.