Enhancement request - sync jobs and other features
Closed this issue · 126 comments
The following is inteface / feature ideas you may consider for future development. The proposed interface is inspired by an old nifty program called "allway sync". The main feature allows creating sync jobs for each book. Each job resides in its own tab where it can be configured and run.
The second file show the sync direction configurator. This is how it is implemented in allway sync. Extremely flexible since one can select the sync direction, participating devices, as well as weather or not to propogate changes, deletions, or both.
The "Analyze" button produces the summary of changes for each device and the "Synchoronize" button - executes the job.
I hope this is in line with the direction you are taking your great little program. Thanks again for your service to the community!
The following shows some of the sync configuration options that one can change upon clicking on the "change" button:
Thank you for your ideas and suggestions.
Unfortunately, syncing highlights was not in the original plan for this app.
It was made just to read the highlights on the PC.
Syncing appeared as an afterthought.
But since it exists, these ideas might help me to make it more user friendly.
The main difference of KOHighlights and this sync app (which obviously is created mainly for syncing), is that KOHighlights is totally device-agnostic.
It can compare between, and work with, only two data parts:
What is loaded from a directory, and a database.
Sure, I can save some preset paths for the mount points of different devices, but I can have only one list of books at a time.
And also only one database.
Hmm..
I have to think more about it..
If you have any more ideas, add them to the pile.. 😉
If syncing appeared as an afterthought, it was a great afterthought indeed ! :)
I could be wrong but imho the difference between two way sync and three way (or more) sync is doing it in sequence. Once the first pair finishes, the second pair starts, etc.
I could be wrong but imho the difference between two way sync and three way (or more) sync is doing it in sequence. Once the first pair finishes, the second pair starts, etc.
The difference for me is to find the different elements to sync (the second pair, the third pair..)
I have two lists of books as I wrote. The file list and the database list.
Those pairs are books that exist in deferent file lists or databases.
The problem I'm thinking is how to create a GUI way to let the user select all those different destinations.
Without recreating half of the application again.. 😉
Don't worry, I'll think of something..
The problem I'm thinking is how to create a GUI way to let the user select all those different destinations.
Without recreating half of the application again.. 😉
Well, I didn't succeed.. I almost rewrote the app.
This enhancement coincided with a major change in the KOReader's metadata format (not yet on current KOReader) and a major addition of GUI changes (dark theme and icons)
So, this is what I came up with:
Not finished yet, so if you have any suggestions..
Wow, this looks like a total re-write!
I like the linear way you present the information and the dark theme looks cool.
Correct me if I am wrong, it appears that your "sync groups" are the same as "jobs", right? If so does it mean that they are going to be persistent or must be re-created every time I open Kohighlights?
RE: the new menu icons. They look great. At the same time when I put on a hat of a new user (someone who has never used Kohighlights before), I would welcome more descriptive naming (or additional hint on mouse over). I would appreciate more info on what the buttons do exactly.
A) E.g. The "power" button on the left - am I correct to assume that it toggles the option to include the given group into the Group Sync?
B) What are the "select" / "plus" / "minus" buttons?
C) What would happen if the sync is attempted while neither "sync position", "merge highlights" or "update to archive" is checked?
D) what would happen if the sync is attempted while some of the devices / sync paths for the given group are not connected / accessible (assuming the groups are persistent)?
Ideally, on the program-level menu (the main menu tool bar), I would expect to see the buttons that modify the groups - add, remove or sync (selected) groups.
The rest of the buttons that modify inside the groups should reside on the group- level sub-menus (currently this sub-menu has a single button called "Sync this group" as well as three check boxes) . I would find it more intuitive should these sub-menus had the "plus" and "minus" buttons allowing additions or removals of the existing (selected) sync paths respectively.
On a different note, there is a potential risk inherent in all sync programs. I am referring to a scenario where one of the sync sources accidently gets deleted and the deletions are then propagated to all destinations resulting in a total data loss. I understand to this end you have the "archive" feature.
However, since currently archiving is a manual process I wonder if you would consider turning the archiving on by default. Ideally, the archive should be viewed as the historic version of the current dataset existing for each sync group on the "server".
That means that Kohighlights database would be included in every sync group by default as a server copy. In case of accidental data corruption one would be able to revert to the last good copy.
In the future a cloud version of the program would make a good sense whereas the user would create a server copy as the very first step by synching his device to the server. Just an idea.
Thanks for your great work! I know this can be a lonely jorney. Let me know if you need any help with testing.
Wow, this looks like a total re-write! I like the linear way you present the information and the dark theme looks cool. Correct me if I am wrong, it appears that your "sync groups" are the same as "jobs", right? If so does it mean that they are going to be persistent or must be re-created every time I open Kohighlights?
They are persistent. Any change of any group is saved immediately,
RE: the new menu icons. They look great. At the same time when I put on a hat of a new user (someone who has never used Kohighlights before), I would welcome more descriptive naming (or additional hint on mouse over). I would appreciate more info on what the buttons do exactly.
The new GUI has the option to select between the old bitmap (colored) icons or the minimalistic ones (menu bottom-right) and of course every button has a tooltip.. 😛
A) E.g. The "power" button on the left - am I correct to assume that it toggles the option to include the given group into the Group Sync?
This button enables/disables this Sync group.
A Sync group consists of various Sync paths.
This let you sync all the enabled Sync groups with one click of the main Merge/Sync toolbar button.
You can also sync the individual Sync groups one at a time, by pressing their "Sync this group" button.
B) What are the "select" / "plus" / "minus" buttons?
The select button lets you select the Sync path's address by browsing you drive.
The plus button adds a new Sync path to the group and the minus removes the Sync path from the group.
C) What would happen if the sync is attempted while neither "sync position", "merge highlights" or "update to archive" is checked?
A popup will inform you that there is nothing to sync.
D) what would happen if the sync is attempted while some of the devices / sync paths for the given group are not connected / accessible (assuming the groups are persistent)?
When a sync path is not accessible or the metadata don't match (with the first) or has one of various problems that is checked against, it will be marked red and will not take part in the syncing process.
Ideally, on the program-level menu (the main menu tool bar), I would expect to see the buttons that modify the groups - add, remove or sync (selected) groups.
Every Sync group contains all the buttons that modify it.
The main toolbar's buttons modify the Sync View (the table that holds the Sync groups)
The rest of the buttons that modify inside the groups should reside on the group- level sub-menus (currently this sub-menu has a single button called "Sync this group" as well as three check boxes) . I would find it more intuitive should these sub-menus had the "plus" and "minus" buttons allowing additions or removals of the existing (selected) sync paths respectively.
See previous item.
On a different note, there is a potential risk inherent in all sync programs. I am referring to a scenario where one of the sync sources accidently gets deleted and the deletions are then propagated to all destinations resulting in a total data loss.
There is not such a risk.
If a metadata files get deleted or corrupted or …, it will not get used at all.
I understand to this end you have the "archive" feature. However, since currently archiving is a manual process I wonder if you would consider turning the archiving on by default. Ideally, the archive should be viewed as the historic version of the current dataset existing for each sync group on the "server". That means that Kohighlights database would be included in every sync group by default as a server copy. In case of accidental data corruption one would be able to revert to the last good copy.
For me, the archived version are for the highlights that you want to have access to, without having to connect your reader.
Something like locally saved to use whenever I need them.
And with every sync it gets a copy of the most recent metadata.
In the future a cloud version of the program would make a good sense whereas the user would create a server copy as the very first step by synching his device to the server. Just an idea.
Hmm.. That will need a server (or more), and a knowledge of things I'm not currently interested in, and money (that I'm not making of this app).
I think that, for an app that I made just to copy my highlights to the PC, it came a long way.. 😃
Thanks for your great work! I know this can be a lonely jorney. Let me know if you need any help with testing.
Thank you. I can send you a link to a beta version, when it's a little more on the ready side, to check (carefully)...
Thanks for clarifying. This is really helpful.
There is not such a risk.
If a metadata files get deleted or corrupted or …, it will not get used at all.
Unfortunately, there is also a human error. It happened to me more than once when I accidently messed things up and deleted many annotations. Maybe it is because I am a professional pessimist I tend to view things through the prysm of recoverability or lack thereoff (I work in disaster recovery)... Lol
For me, the archived version are for the highlights that you want to have access to, without having to connect your reader.
Something like locally saved to use whenever I need them.
And with every sync it gets a copy of the most recent metadata.
I have not fully understood the last sentence above. Do you mean the program automatically updates the archive every time the group sync is run? Will there be a way to use the last good copy to recover should the said risk materializes?
Thank you. I can send you a link to a beta version, when it's a little more on the ready side, to check (carefully)...
Sure !
Unfortunately, there is also a human error. It happened to me more than once when I accidently messed things up and deleted many annotations.
In KOHighlights?
How did you do it? Manually?
You can only delete individual annotations one by one.
I couldn't think that this could happen by accident.
There are always warning popups for the "human error".
And with every sync it gets a copy of the most recent metadata.
I have not fully understood the last sentence above. Do you mean the program automatically updates the archive every time the group sync is run? Will there be a way to use the last good copy to recover should the said risk materializes?
The metadata files that take part in the sync have different modified times.
The annotations from the most recent is copied to the archive (of course only if you have the "Update to archive" checked).
This copy is used as one part of the sync. Like the metadata files.
That means that if you accidentally delete your annotations from your reader, syncing with the archive copy will recover them.
Maybe I should name the checkbox "Sync with archive" or "Sync with archived"..
Edit: Changed it to "Sync with archived"
How did you do it? Manually?
Mostly manually. Don't underestimate the human propensity to screw things up and the speed with which the fingers can move ahead of the brain... lol. On a different occassion I lost nearly all comments (had to recreate most of them manually) after I moved away from kepub to epub on my kobo reader. I re-uploaded the new epubs from Calibre but could not restore the comments from the metadata files which was created for the kepubs...
That means that if you accidentally delete your annotations from your reader, syncing with the archive copy will recover them.
Maybe I should name the checkbox "Sync with archive" or "Sync with archived"..
Edit: Changed it to "Sync with archived"
Ok, that makes sense.
Here is a beta version that has all the new features to test.
Be careful with your files (always backup).
The installer will remove v1.x.x installation files, but the settings will be preserved.
Have fun.. 😉
Great! I have installed the beta and started testing. How do you want me to report the issues?
Here is a couple so far:
- The program doesn't see my android phone and I am not able to add it to sync (the phone is connected to windows via USB but the windows doesn't map it as a drive. Instead it shows in the explorer as "Card" and "Phone" for the SD card and the phone's internal storage respectively. (this was also an issue with the previous version of the program.)
- I am unable to name / rename the groups. Right click / rename group does nothing. The dialog box doesn't provide a way to enter or edit the name
Great! I have installed the beta and started testing. How do you want me to report the issues? Here is a couple so far:
Like you do now.. 😉
Just add them here and will fix it as we go..
- The program doesn't see my android phone and I am not able to add it to sync (the phone is connected to windows via USB but the windows doesn't map it as a drive. Instead it shows in the explorer as "Card" and "Phone" for the SD card and the phone's internal storage respectively. (this was also an issue with the previous version of the program.)
This is a known issue, see #21
- I am unable to name / rename the groups. Right click / rename group does nothing.
That's a nice catch.. 😃
If you select a path it will automatically get the title from the book, and then you can change it.
The same applies if you create a group from a book in the Books view.
But I forgot about a new entry.. 👎
Tomorrow will upload a new beta..
Also don't leave a totally empty group while you're closing the app, or it will create a ghost group that won't be possible to change afterwards.
Will be fixed in the next beta..
Actually, I just uploaded a new beta with the fixes..
Just installed the new beta 1.9.9.1
The program did not save the old profiles (so I will need to recreate them manually).
RE: not recognizing android devices. I am using a work-around: installed an FTP app on the phone and then mapped the connection as a local drive using NetDrive. Works well.
Going forward for clarity I will refer to my WSA Koreader instance as "Device 1", My Kobo Libra2 reader as "Device 2" and my Android phone instance as "Device 3"). Or as D1, D2, D3 respectively.
Issues with 1.9.9.0 (will re-test this on 1.9.9.1 tomorrow):
Issue 1: New highlights do not show in the bookmark and comments are not synced
Starting point: D1 and D2 were in sync with each other, but not D3
Sync group created with the D1, D2, and D3. All the 3 check boxes checked (i.e. "sync positions", "merge highlights", and "sync with archived")
Result:
A dialog box informing that syncing of the group was completed.
1.1 D3 shows the new highlights from D1 / D2 but not the comments attached to the highlights.
1.2 D3 Koreader's bookmark menu doesn't show the new highlights or comments (only the old ones that were created on D3).
Issue 2: New highlights / comments are deleted from the originating device by subsequent sync and not propagated to other devices in the group.
Starting point: D1, D2 and D3 synced via Kohighlights 1.9.9.0
Sync group created with the D1, D2, and D3. All the 3 check boxes checked (i.e. "sync positions", "merge highlights", and "sync with archived")
New highlights with comments made on D1, then Koreader on D1 is exited and the sync for the group is executed.
Result
A dialog box informing that syncing of the group was completed.
Re-opening the book on D1 shows that the newly made comment disappeared (tested twice, the same result)
Opening the book on D2 and D3 doesn't show the new highlight / comments at all
The program did not save the old profiles (so I will need to recreate them manually).
By profiles you mean Sync groups?
If so, these are located in the settings folder of windows and are not erased.
The same is true for the other settings of the app.
BUT there was a bug that prevented the loading of all Sync groups if a Sync path was not existing.
That is fixed in 1.9.9.2
Point 1.A general advice regarding android/WSA
I spend hours trying to understand why I was not syncing some files to Dropbox from my android mobile, only to found out that the changed file was changed only in the cache of the OS and was not committed to the disk.
After that I had to explicitly check the files to be sure they were written before were send with any app.
This is true for all Linux systems (random stack overflow comment)
So, unfortunately you have to double check the existence of the changes in the.luafiles..Point 2.Some general info about metadata formats.
The current KOReader metadata format is going to be replaced (probably after the next update) with a "new" much better and cleaner format. I'll refer to the current format as the "old" format.
This "old" metadata format had two different places to storehighlightsandbookmarks.
It was build one new feature at the time, and this created a patchwork that was a little confusing and redundant.
One issue that is currently affecting KOHighlights (that will be solved with the "new" format) is that the comments of the highlights are stored in thebookmarksplace.
So, when I find an existing highlight, the app continue without knowing if the comment is different.
This is changed with the "new" format .
Also note that "old format" bookmarks were not synced.
Only highlights. (KOHighlights was also build with one feature at a time.. 😃 )
Issue 1: New highlights do not show in the bookmark and comments are not synced
If we rule our Point 1, then the culprit should be a bug that is fixed in1.9.9.2.
Most of the new "merge" code was for the "new" format, but somehow, the old one was affected too.. 😞
Issue 2: New highlights / comments are deleted from the originating device by subsequent sync and not propagated to other devices in the group.
For the missing highlights, if we rule out the Point 1 from above, most probably happened because of the previous bug.
For the missing/not updated comments, see Point 2.
This is the new 1.9.9.2.beta, with more squashed bugs..
Thank you for your time. 🙏
Another day, another beta... version 1.9.9.3 😉
Point 1. A general advice regarding android/WSA
I spend hours trying to understand why I was not syncing some files to Dropbox from my android mobile, only to found out that the changed file was changed only in the cache of the OS and was not committed to the disk.
After that I had to explicitly check the files to be sure they were written before were send with any app.
This is true for all Linux systems (random stack overflow comment)
So, unfortunately you have to double check the existence of the changes in the .lua files..
I understand this has to do with how Koreader updates the metadata files. I was struggling with same issue before eventually learning that you can force the update by exiting Koreader from the exit menu. This causes Koreader to write the latest changes into the metadata.epub.lua (works on WSA and android phone)
Installed 1.9.9.3
Here are the issues so far:
The program updated ok, but after I recreated the sync groups and executed the first sync group, the screen blinked, and the newly created groups disappeared.
Issues 1 and 2 remain unresolved.
Issue 3: No notification dialog is displayed upon execution of Merge/Sync
Starting point: D1, D2 and D3 synced via Kohighlights 1.9.9.0
Sync group created with the D1, D2, and D3. All the 3 check boxes checked (i.e. "sync positions", "merge highlights", and "sync with archived")
New highlights with comments made on D1, then Koreader on D1 is exited to force metadata update and the sync for the group is executed.
Result
No dialog box is displayed to notify about the sync results.
Re-opening the book on D1 shows that the newly made comment disappeared
Opening the book on D2 and D3 doesn't show the new highlight / comments at all
Issue 4: The "Books /Loaded" view fails to update the archive
Starting point: D1, D2 and D3 synced via Kohighlights 1.9.9.0
New highlights with comments made on D1, then Koreader on D1 is exited and the sync for the group is executed.
- Go to the "Books /Loaded" view.
- Use Scan Directory to navigate to the .sdr folder for a book (that is already in archive).
- Select sync with archive
- Go to the "Books/Archive" view and select the same book to see if the comment made on D1 did sync with the archive
Result:
The comment is not synced with archive
-
Navigate back to "Books /Loaded" view
Result:
The book that was used to sync with the archive is no longer visible (disappeared from the view). I tried to use "scan directory" to redisplay it but it fails to see the book. -
The comment made on D1 has disappeared (the Issue 2 persists)
Suggestion, consider changing the "Scan Directory" to "Select File" for consistency. In the Sync Group view one must select the specific .lua files, while in the Books/Loaded view the program scans the .sdr directory and selects the file automatically (or doesn't as per the issue above).
Starting point: D1, D2 and D3 synced via Kohighlights 1.9.9.0
I'll try to recreate your test cases, but I have to warn you that this starting point is wrong.
Version 1.9.9.0 had a bug that missed some highlights/bookmarks.
These metadata are problematic.
You have to start with the original metadata.
I'll post about the rest of your points a little later..
I'll try to recreate your test cases, but I have to warn you that this starting point is wrong.
Version 1.9.9.0 had a bug that missed some highlights/bookmarks.
These metadata are problematic.
You have to start with the original metadata.
Agree.
Let me reboot the machine and start from the beginning (I will restore the .sdrs from backup)
There was a problem with the v1.9.9.3 too. 😳
Too many changes at the same time!
Please, use this 1.9.9.4 beta instead..
The program updated ok, but after I recreated the sync groups and executed the first sync group, the screen blinked, and the newly created groups disappeared.
This, I can't explain. Never seen anything like that.
If you can replicate it, note down every step, so I can create it myself too.
Issues 1 and 2 remain unresolved.
These I think, have to do with corrupted metadata files.
If it persists with 1.9.9.4 I'll check again.
Same with Issue 3.
The lack of info dialog at the end is fishy.
Even if nothing was merged it should inform you about it.
The Issue 4:
What happened to the file? Was it deleted?
If it was in the sdr folder, it should be scanned and discovered again.
The only explanation I can think of is that some part of the app was crashed and it needed a restart to work properly again.
Something is wrong here, but I have to wait for you to test it with 1.9.9.4 and normal metadata files, before starting a total rewrite..
I will probably need your metadata files too, if possible.
Suggestion, consider changing the "Scan Directory" to "Select File" for consistency. In the Sync Group view one must select the specific .lua files, while in the Books/Loaded view the program scans the .sdr directory and selects the file automatically (or doesn't as per the issue above).
The "Scan Directory" is a folder selector dialog that selects the root folder of your book library (or the root of your reader's exposed drive) to scan all the books' metadata files recursively.
These can be hundreds or more.
Selecting one file at the time (since usually there is only one metadata file in each .sdr) would make users spend a lot of time just to open a few books.
The Sync groups on the other hand can have just a couple of files.
Metadata files can have different names and not be alone in their own folder, depending on the device (some times many of them are located in a "History" folder), so choosing a folder is not enough to specify which file will get synced.
I'm anxiously waiting for your results..
Performed full uninstall of all older versions using Revo, Also removed all archive data, configs and cleaned up the registry.
Removed all .sdr folders from the test books.
Restored the .sdr from backup.
Installed 1.9.9.4
Installation went ok. Was able to switch from / to new or old icons and themes.
The new comments or highilights are not syncing. Both the Issue 1 and 2 persist. The newly made comments are not syncing to the destinations and are deleted from the origin
Here is another strange one: the same book on D1 is being recognized as a different book, so I am unable to sync this particular group. I have uploaded and re-apploaded it (using Calibre) trying to make it work, but no luck.
Issue 5: The metadata file of the same book is rejected as being from a different book
The three sdrs are attached. See if you are able to sync it.
Book not recognized as same.zip
Here is another strange one: the same book on D1 is being recognized as a different book, so I am unable to sync this particular group. I have uploaded and re-apploaded it (using Calibre) trying to make it work, but no luck.
Issue 5: The metadata file of the same book is rejected as being from a different book
The three sdrs are attached. See if you are able to sync it.
One possible reason for the books not recognized as the same, might be Calibre's reader.
It used to be that if you opened an epub with Calibre's reader, it stored the reading position inside the epub.
I don't know if this is still true, but if it is, the epub was changed and afterward, because of the different md5 checksums, KOHighlights identifies them as not the same books.
I'll check the metadata and come back to you.
Thanks.
More answers..
As I thought, the D1 book's md5 checksum is different from the others.
That means that is somehow edited.
About the not syncing comments, the answer was in Point 2 of my previous comment.
In a nutshell, comment editing can't be synced between the "old" (current) formats, but will be possible with the upcoming "new" metadata format.
What it will be most helpful are the metadata files that don't merge from the Issue 1 & 2, because whatever I do here, highlights are always merge as expected (except the comments of course)..
Maybe we should wait until Koreader releases the new version based on the new metadata format and then resume testing? Do the devs provide an ETA?
I also hope they implement a framework that allows to sync comments irrespective of the screen resolution / size differences
D1 book's md5 checksum is different from the others.
Not sure why this happens. It did happen before as well (with a different book in the old version). How can I make this file recongized? It is the same book, lol...
Maybe we should wait until Koreader releases the new version based on the new metadata format and then resume testing?
I have to release KOHighlights before the KOReader release, so the users don't get confused.
Anyway, the two formats have different code that must exist in the app for users of older KOReader versions.
This has nothing to do with the Sync groups. These are a different feature all together.
Do the devs provide an ETA?
The release after the next one.
I also hope they implement a framework that allows to sync comments irrespective of the screen resolution / size differences
As I said the "new" format support it.
And that's what they did. They re-wrote the metadata format structure.
Its me that have to implement the merging on KOHighlights.
I just said that it's a lot of work to support it for the "old" format too.
I might do it, but for now I'm working on this Sync group feature.
Edit: Done it..
So, can you send me the not merging metadata to see what's going on with that issues?
I'm sorry for all the work I make you to do, but the beta testing is really appreciated.. 🙏
A new 1.9.9.5.beta version is uploaded.
I finally succeed to merge the different comments of the same highlight.
The newer one will prevail and if the newest action was to delete the whole comment, it will get deleted from all.
So time for a recapitulation.
- Issue 1.
It should be resolved since the missing highlights were due to a 1.9.9.0 bug and the comments are synced now (from 1.9.9.5+) - Issue 2.
Same as Issue 1. - Issue 3.
This is something I can't reproduce. I need the specific steps. - Issue 4.
Books View merging works exactly as the Sync Groups merging.
You didn't answer me about what happened to the lua file that was not scanned.
I can't reproduce this. - Issue 5.
This is resolved, since the book file was edited, and it wasn't the same as the others.
Did I miss anything?
If something from the above is wrong, please elaborate.
Try the comment syncing to be sure..
Thanks 🙏
I will install and re-test it tonight.
Thanks!
Performed full uninstall of 1.9.9.4. Also removed all archive data, configs and cleaned up the registry.
Removed all .sdr folders from the test books.
Restored the .sdr from backup.
Installed 1.9.9.5
Recreated sync groups. I see that the book names for the sync groups are now populated automatically - this is great!
Issue 1 - appears to be fixed. New highlights and comments do propagate and displayed under the bookmarks
Issue 2 - appears to be fixed. The new highlights are no longer deleted from the originating device
However, there is a related problem which might be related (so I will insert it here as
Issue 2A - The deleted highlights are not propagated and restored back on subsequent sync
Starting point:
Completed sync on a sync group made of D1, D2 and D3. All devices are in sync.
- Open the book on D3 and delete one of the highlights.
- Exit the Koreader to force sync
- Return to KOHighlights and sync the group again.
- Check the devices for the deleted highlight
Result:
The deleted highlight is back on D3 and is not removed from D2 and D1
Moving on:
Issue 3 - has never reoccurred since. I would close this as well
The Issue 4:
What happened to the file? Was it deleted?
If it was in the sdr folder, it should be scanned and discovered again.
The only explanation I can think of is that some part of the app was crashed and it needed a restart to work properly again.
No the file was there, however the program could not see it. What you are saying above make sense. It is likely was caused by a crush of some part of the program. It can also be related to Issue 6 (see below). So I would close it for now.
Issue 4 - closed
Issue 5 - closed
I still have no explanation why this book changed md5 checksum... I had to copy the book from D3 to D1 and recreated the .sdr So it works now.
Issue 6 : After each sync the Books/Archived view fails to refresh showing duplicates of the same book
Starting point:
- Clean install of the program v 1.9.9.5 (old version completely uninstalled, no settings preserved)
- Create a sync group for a book on D1, D2, D3
- Execute sync (with one checkbox checked)
- Navigate to Books/Archived view
Result:
The synced book's title is displayed.
-
Navigate back to the Sync Group views
-
Select another check box
-
Execute sync (with two checkboxes checked)
-
Navigate to Books/Archived view
Result:
The synced book is displayed twice -
Navigate back to the Sync Group views
-
Select the third check box
-
Execute sync (with three checkboxes checked)
-
Navigate to Books/Archived view
Result:
The synced book is displayed three times -
Now switch to the "Loaded view" and back to the "Archived"
Result: the screen refreshes and correctly displays a single book
PS. There is no need to actually select one check box at a time. You can sync the same group several times - the result will be the same.
Issue 7 (suggestion, not a bug) By default check all options under "Show in Highlights" menu
Starting point - loaded or archived book selected (on a freshly installed program)
The program's default setting for "Show in Highlights" are blank. Accordingly, the highlights pane only displays the chapter headings (see Issue 8)
Issue 8. Chapter headings cannot be removed from display even if everything is unchecked under "Show in Highlights"
Ideally, it would be great to define how many levels of headings one could display since this determines how the exported text looks in in .md format among other things.
Update on Issue 2A
The same issue occurs (i.e. the deleted highlight is restored) when using a manual sync from the Loaded view with the archived. After the comment was deleted on D3 I exited Koreader and loaded the .sdr via the Books/Loaded view. It correctly showed that the highlight no longer was present. After syncing with the archived the deleted comment was restored...
Issue 2A - The deleted highlights are not propagated and restored back on subsequent sync
Unfortunately this is something that can't be fixed.
There is no way for the app to know if this missing highlight is because you purposely removed it or is not there yet and needs to get merged.
Update on Issue 2A
The same issue occurs (i.e. the deleted highlight is restored) when using a manual sync from the Loaded view with the archived. After the comment was deleted on D3 I exited Koreader and loaded the .sdr via the Books/Loaded view. It correctly showed that the highlight no longer was present. After syncing with the archived the deleted comment was restored...
Are we talking about a deleted highlight or a deleted comment in a highlight?
A deleted highlight will be merged from the other versions of the book.
A deleted comment will get deleted from all other versions (if the deletion was made last)
I still have no explanation why this book changed md5 checksum...
Did you open it with Calibre?
Issue 6 : After each sync the Books/Archived view fails to refresh showing duplicates of the same book
I can reproduce this..
Really silly of my. Fix on it's way 😋
Issue 7 (suggestion, not a bug) By default check all options under "Show in Highlights" menu
Well, I already do this. All options should be checked..
Something is broken there.. I'll investigate.
This must be connected with the Issue 8 too.
Very good findings, thank you.
Issue 2A - The deleted highlights are not propagated and restored back on subsequent sync
Unfortunately this is something that can't be fixed.
There is no way for the app to know if this missing highlight is because you purposely removed it or is not there yet and needs > to get merged.
I wonder if this could be achieved via uni-directional sync. One day maybe? ...😏
Update on Issue 2A
The same issue occurs (i.e. the deleted highlight is restored) when using a manual sync from the Loaded view with the archived. After the comment was deleted on D3 I exited Koreader and loaded the .sdr via the Books/Loaded view. It correctly showed that the highlight no longer was present. After syncing with the archived the deleted comment was restored...
Are we talking about a deleted highlight or a deleted comment in a highlight?
A deleted highlight will be merged from the other versions of the book.
A deleted comment will get deleted from all other versions (if the deletion was made last)
I have tested this with highlights only. My bad. I should not have used "highlights" interchangeably with "comments"
I still have no explanation why this book changed md5 checksum...
Did you open it with Calibre?
I used Calibre to copy the book from the Calibre library to Kobo Libra 2 as well as to Calibre Companion on WSA as well as on the phone
Very good findings, thank you.
I am glad to be of help. Thank you for all your work on KOhighlights. Keep up the good work!
I find the "Highlights" view somewhat difficult to use given that one cannot use it one book at a time (i.e. highlights from all books are presented in one big table). In the way of a suggestion it would be great if you could create a book selector pane similar to one used for the "Books" view? The book level metadata such as Title, Author, Type, Modified, etc. should stay in the book selector pane, and the Highlights table should contain the Highlihgts, Comments, Date, Page, Chapter (and possibly other levels of headings as per Issue 8)
Unfortunately the Issue 2 has returned after I have added the third sync group (I am not saying that this caused it since it might be just a coincidence). I have rebooted the machine and reopened the app. The sync groups survived but the issues persist. Interestingly enough, the new highlight (made on D2) shows correctly under the Books/Loaded view when I select its .sdr folder. Then it syncs correctly with the archived. Once the highlight or comment exists in the archived, the sync correctly propagates to other destinations (in this case D1 and D3). If it is not in the archived, it won't propagate and gets deleted from the originating device. 😟
New 1.9.9.6.beta uploaded..
I wonder if this could be achieved via uni-directional sync. One day maybe? ...😏
😄
I used Calibre to copy the book from the Calibre library to Kobo Libra 2 as well as to Calibre Companion on WSA as well as on the phone
Well, something edited the book file for sure.
You can see at the Book Info panel that there is an extra "-2 Testing" tag and a different language code.
Issue 6 : After each sync the Books/Archived view fails to refresh showing duplicates of the same book
Fixed in 1.9.9.6
Issue 7 (suggestion, not a bug) By default check all options under "Show in Highlights" menu
Issue 8. Chapter headings cannot be removed from display even if everything is unchecked under "Show in Highlights"
There was a bug there too! Affected both. Fixed in 1.9.9.6
I find the "Highlights" view somewhat difficult to use given that one cannot use it one book at a time (i.e. highlights from all books are presented in one big table).
You can already do that if you use the Filter to limit the entries to that of a specific book.
Interestingly enough, the new highlight (made on D2) shows correctly under the Books/Loaded view when I select its .sdr folder. Then it syncs correctly with the archived. Once the highlight or comment exists in the archived, the sync correctly propagates to other destinations (in this case D1 and D3). If it is not in the archived, it won't propagate and gets deleted from the originating device. 😟
This I'm not sure I understand.
You have an extra highlight in D2.
If it is synced with the archived you see it in both loaded and archived.
If you then merge it with the others, the highlight is copied to the others.
If not synced with the archived and you merge D2 with the others, this highlight is deleted from all?
Is this what you are saying?
I can't replicate it by doing these steps here.
Merging the books here always creates the new highlight in all of them, archived or not..
Am I missing something?
Interestingly enough, the new highlight (made on D2) shows correctly under the Books/Loaded view when I select its .sdr folder. Then it syncs correctly with the archived. Once the highlight or comment exists in the archived, the sync correctly propagates to other destinations (in this case D1 and D3). If it is not in the archived, it won't propagate and gets deleted from the originating device. 😟
This I'm not sure I understand.
You have an extra highlight in D2.
If it is synced with the archived you see it in both loaded and archived.
If you then merge it with the others, the highlight is copied to the others.
If not synced with the archived and you merge D2 with the others, this highlight is deleted from all?
Is this what you are saying?
I can't replicate it by doing these steps here.
Merging the books here always creates the new highlight in all of them, archived or not..
Am I missing something?
This was in the context of the Issue 2 that has become an issue again. It occurred when syncing via the "sync group" however, when using Books/Loaded/Sync with Archived the highlight would sync with the "archived" correctly. If one uses "sync group" after that (that is when the "archived" contains the new highlight or comment) it would then correctly propagate to other devices in the group.
I will install 1.9.9.6 and test again. This time I wil let the program to update from the old version to make sure that the sync groups and other settings are preserved correctly.
Issue 9 - Installation of 1.9.9.6 triggered defender warning.
The program finished the install, but the sync groups are not preserved. Not sure if this was caused by the defender or not...
Issue 10 - The sync group settings are not preserved following the program update to new version
Manually restoring the "quarantined threat" and re-launching the program did not seem to change anything
I have re-created the sync groups manually.
The Issue 2 is still causing an impact. Highlights / Comments are not propagated to other devices and are erased from the originating device
Issue 9 - Installation of 1.9.9.6 triggered defender warning.
Unfortunately, nothing I can do here..
The way that I package all those setups is the same, using the same version of PyInstaller.
From your screenshot I can see that the affected item is the KOHighlights Portable.exe, which is a packed .bat file that contains only the command ..\KOHighlights\KOHighlights.exe -p to start the app in portable mode.
Its the exact same file that you had in every one of the betas.
That means that Defender is loosing its marbles..
The sync group settings are not preserved following the program update to new version
Sync group setting are stored in the C:\Users\--USER_NAME--\AppData\Roaming\KoHighlights folder, in a json file called sync_groups.json.
The only way that the app is deleting this file is if the user selects to "Remove any existing user settings" during the uninstallation of the app.
During a normal install/update, nothing changes in this folder.
You can keep track of this by yourself.
I have re-created the sync groups manually.
The Issue 2 is still causing an impact. Highlights / Comments are not propagated to other devices and are erased from the originating device
Since I can't reproduce this, is it possible for you to send me these 3 metadata files (with one having this extra highlight) to check it here?
The only way that the app is deleting this file is if the user selects to "Remove any existing user settings" during the uninstallation of the app.
During a normal install/update, nothing changes in this folder.
You can keep track of this by yourself.
Not sure where I can select "Remove any existing user settings" or not to remove... here is the screen shot.
And here is what I see when I click "Next"
Not sure where I can select "Remove any existing user settings" or not to remove..
"Remove any existing user settings" is an option that you see if you Uninstall the app.
During installation you only have the option to remove the "Program files" directory..
"Remove any existing user settings" is an option that you see if you Uninstall the app.
During installation you only have the option to remove the "Program files" directory..
I see what you mean. However, most users are not used to this type of "orderly" process when one is expected to uninstall the app first and only then install the new version. People are conditioned to run the new version on top of the old one and expect that the program will do the rest - i.e. unistall the old version preserving the settings and install the update. Unless the program follows the expected behavior I am afraid many users would find themselves in the same situation as I did - i.e. needing to recreate the sync groups, etc.
People are conditioned to run the new version on top of the old one and expect that the program will do the rest - i.e. unistall the old version preserving the settings and install the update.
And this is exactly what the setup is doing. Nothing more.
I use the same installer in many programs.
Not once were the user settings affected by the installation of a newer version.
I just did an new install, created a Sync group, closed the app.
Run the setup again, open the app, the group is there.
Again, more groups, more installations, the settings persist.
I don't understand what is happening there.
Windows are going bonkers!! 🙄
I have re-created the sync groups manually.
The Issue 2 is still causing an impact. Highlights / Comments are not propagated to other devices and are erased from the originating device
Since I can't reproduce this, is it possible for you to send me these 3 metadata files (with one having this extra highlight) to check it here?
For clarity, this only impacts the highlights. The comments seem to work ok.
The attached are the 3 .sdrs. The D2 has a couple of highlights not present in D1 and D3. The sync erases the highlight from D2 instead of propagating them to D1 and D3. This happens to other books as well so you should be able to replicate
Apr 24 - highlights deleted from origin not synced.zip
I don't understand what is happening there.
Windows are going bonkers!! 🙄
I agree, windows is getting worse and worse.
I just did an new install, created a Sync group, closed the app.
Run the setup again, open the app, the group is there.
Again, more groups, more installations, the settings persist.
I wonder if this was caused by the Defender that kicked in during my previous installation... Let me try again
Apr 24 - highlights deleted from origin not synced.zip
These files have exactly the same highlights
You should send me the ones that aren't synced..
These files have exactly the same highlights
You should send me the ones that aren't synced..
I don't know what is going on. Before sending it to you I made sure that D2 was different (I did not open the .lua directly, but instead used the Books/Loaded view to scan the .sdr. It then showed the two extra highlights. I then sent the files to you. Now when I looked at the zipped .sdrs sent to you - they are exactly the same... 😟😟😟
Upon further reflection. It appears that this can happen because of either:
a) Koreader did not update the metadata file, so even the highlight was present on the screen, but not yet written in metadata.epub.lua
b) KOHighlights is somehow using a catched verison of metadata and not refreshes the file prior to sync
Regarding (a): Koreader has a setting that impacts how frequently metatada file is refreshed. However, my understanding is that it always updates it before the program is exited. This is what I have been using all this time to force metadata updates.
Regarding (b): I don't how KOhighlights does it - so maybe you could see if there is a potential issue there...
Any other ideas?
Regarding (b): I don't how KOhighlights does it - so maybe you could see if there is a potential issue there... Any other ideas?
Every change that the sync/merge does to the metadata is in-place (in the memory).
So the metadata are always up to date.
I manually update the table's column values, so that they are also up to date, without the need of a new scanning of the files.
Of course the scanning of the file while is loaded is the most reliable..
Every change that the sync/merge does to the metadata is in-place (in the memory).
So the metadata are always up to date.
I manually update the table's column values, so that they are also up to date, without the need of a new scanning of the files.
I can be wrong but re-scanning the files before every sync/merge is albeit a tiny bit slower, but would be a lot more reliable, especially when the changes are made while Kohighlights is running in the background.
This new 1.9.9.7.beta might be the last one, since I don't think we have any standing issues (apart from the Windows quirks).
Installed the latest beta 1.9.9.7 . For the first time the program displayed a dialog box asking if I wanted to remove the settings. This is certainly a positive. The initial loading of the program took a long time. The sync groups were present, however, some of the entries were in red (even though all devices were connected). Since there is no "refresh" button the only way for the program to recognize the connected devices was to remove them from the sync groups and add them back.
When for the first time I switched to the Books/Archived view the program displayed a dialog informing that it is loading the database and the program appeared to be frozen for a long time (certainly more than a minute, may be close to a minute and half) when it finally displayed the archived view. But here is the odd thing (see the pic):
For whatever reason the archived view doesn't see two out of three books. I made highlights for those books but they failed to sync. And the archived view still show a warning saying that "epub is missing"
I noticed that the two failed ones have a diffrent path which I don't know how to interpret this since the path of the archived view is supposed to point to the database, not a book location on a drive.
The last but not least: when I went back to the Sync Groups view this is what I see 😟😟😟
Question: do you really need the program to be backwardly compatible with the [old] metadata framework that Koreader is replacing? In my view trying to accommodate both standards in a single program introduced an extra layer of complexity. From my limited understanding of the domain (please pardon me for being overly simplistic here) it would appear that it maybe best to keep the old working version of Kohighlights as is, and develop the new functionality as a separate branch tailoring it to the new metadata format in the upcoming version of Koreader. You can call it differently as well - e.g. "KoHighSync" or something like that to differentiate from the old one... Just a thought.
Installed the latest beta 1.9.9.7 . For the first time the program displayed a dialog box asking if I wanted to remove the settings. This is certainly a positive.
This dialog was always displayed when you uninstall the app.
What is new, is that now there is another dialog that asks you if you want to uninstall the app.
This will appear only when updating from 1.x.x.x versions to 2.x.x.x.
The beta is still 1.9.9.x, so you will see this with every beta install.
The initial loading of the program took a long time. The sync groups were present, however, some of the entries were in red (even though all devices were connected).
Positioning the mouse over the red text will provide a tooltip that describes the specific error.
If the error is a missing path, then the specific path was missing at the time the groups were loaded.
But it might be a different error.
Since there is no "refresh" button the only way for the program to recognize the connected devices was to remove them from the sync groups and add them back.
You could close/re-open the app for the groups to get reloaded.
No need to recreate them..
When for the first time I switched to the Books/Archived view the program displayed a dialog informing that it is loading the database and the program appeared to be frozen for a long time (certainly more than a minute, may be close to a minute and half) when it finally displayed the archived view.
This should be almost instantaneous for that many entries.
Something else was blocking the disk.
But here is the odd thing (see the pic):
For whatever reason the archived view doesn't see two out of three books.
You can always check the Path column to see if the specific path exist in explorer.
If it says its missing, well, it must be missing.. 😉
I made highlights for those books but they failed to sync.
I don't understand what you're saying here.
I need some more info..
And the archived view still show a warning saying that "epub is missing" I noticed that the two failed ones have a diffrent path which I don't know how to interpret this since the path of the archived view is supposed to point to the database, not a book location on a drive.
The data of the book is in the database.
The path is the path to the book that is copied to the database.
That means that when you archived this book, the actual book was located in an path that now does not exist.
The last but not least: when I went back to the Sync Groups view this is what I see 😟😟😟
That means that there was something wrong with the data of the specific Sync group.
If you send me the sync_groups.json that is located in the C:\Users\--YOUR_USER_NAME--\AppData\Roaming\KoHighlights directory, I can tell you exactly what the error is..
Just uploaded this 1.9.9.8.beta version that adds a refresh button on every Sync group.
I had it there from the beginning, but I didn't thought that anybody would need it.. 😏
I have tested 1.9.9.8 beta. Unfortunately, Issue 2 is not yet resolved. 😟
adds a refresh button on every Sync group.
I had it there from the beginning, but I didn't thought that anybody would need it.. 😏
This is useful - thanks!
I have tested 1.9.9.8 beta. Unfortunately, Issue 2 is not yet resolved. 😟
Issue 2: New highlights / comments are deleted from the originating device by subsequent sync and not propagated to other devices in the group.
As we already discussed, highlights that are deleted from a book will get re-created after merge, because there is no way of knowing if the highlight was deleted or not yet merged.
OTOH, comments that are deleted from one device should get deleted from all the devices.
If this is not happening I'll need some files that reproduce this, so I can investigate..
As we already discussed, highlights that are deleted from a book will get re-created after merge, because there is no way of knowing if the highlight was deleted or not yet merged.
I did not make my self clear. I was refering to comments and highlights that were (1) made on the originating device and (2) deleted from that device after sync. Also there is no propagation to other devices in the sync group. That is the behavior first described under Issue 2.
If you are referring to this..
I have re-created the sync groups manually.
The Issue 2 is still causing an impact. Highlights / Comments are not propagated to other devices and are erased from the originating device
Since I can't reproduce this, is it possible for you to send me these 3 metadata files (with one having this extra highlight) to check it here?
I still don't have any files that reproduce this.
The ones that you send me was already synced.
Without files that exhibit this problem, there is nothing I can do, since my trials here do not behave this way..
I still don't have any files that reproduce this.
The ones that you send me was already synced.
Without files that exhibit this problem, there is nothing I can do, since my trials here do not behave this way..
The problem is seem to be occurring during the run time. It is hard for me to explain so I am going to record the screen session to illustrate. Here is the first recording - unfortunately the sync groups disappeared again so I will recreate them shortly and continue. In the meantime here is what I have:
For clarity, I did try closing / openning the program, and even reinstalling it. The groups are gone.
disappering.groups.mp4
Can you please send me the sync_groups.json that is located in the C:\Users\--YOUR_USER_NAME--\AppData\Roaming\KoHighlights directory, and are not showing up?
I have already recreated the groups so the .json might be reflecting this. But, it is attached
sync_groups.json
Here is the video illustrating the Issue 2
Issue.2.illustrated.mp4
Here is a better illustration
Issue.2.-.another.example.mp4
OK. One picture is 1000 words.
So imagine what a video is.. 😃
Now I know what happens, so I will describe the steps of the events so you can understand it too.
- KOHighlights opens.
- The Sync Groups are loaded and read the metadata files from the disk to memory.
- KOReader is opened.
- A new highlight is made.
- KOReader closes and the metadata file is saved on the disk.
At this point the highlight is written in the metadata file (that file I wanted).
- Switching to KOHighlights that doesn't know about the new addition.
- Start the syncing of the group.
- KOHighlights uses the already loaded data to synchronize the files and saves them to disk.
At this point the files are over-written and are not containing the additional highlight.
- Re-opening KOReader shows no highlight as expected.
The solution to this might be the refresh button (I'll have to check it though).
Did you press it?
If, after the creation of the new highlight you were pressing "Refresh" the groups should be reloaded and the data would be updated.
Any syncing after this should work as expected.
I have to test this more.
But the most reliable solution of course would be to close and re-open KOHighlights.
And an of topic question:
What is the ftp app you are using to show the Android directories to the PC?
Just checked Refresh and it works as it should.
If you press it after any external change, the data of the files are updated to memory and the sync keeps the new highlights..
OK. One picture is 1000 words.
So imagine what a video is.. 😃
It sure is better than words... 🤩
Switching to KOHighlights that doesn't know about the new addition.
yep, I think this is it!
The solution to this might be the refresh button (I'll have to check it though).
Did you press it?
If, after the creation of the new highlight you were pressing "Refresh" the groups should be reloaded and the data would be updated.
Any syncing after this should work as expected.
I have to test this more.
I have not used the refresh. I should do some more testing as well.
ust checked Refresh and it works as it should.
If you press it after any external change, the data of the files are updated to memory and the sync keeps the new highlights..
Great. Although you should consider incorporating the refresh into the sync routine
And an of topic question:
What is the ftp app you are using to show the Android directories to the PC?
On Android I am using File Manager as an FTP server https://play.google.com/store/apps/details?id=com.alphainventor.filemanager&hl=en_CA&gl=US&pli=1
And on windows NetDrive to map the FTP connection as a drive letter
https://www.netdrive.net/
Great. Although you should consider incorporating the refresh into the sync routine
That's a good idea. I added it. 👍
Is there anything else that is not working at this time?
After I manually refreshed the group the sync for that specific group did not display a notification about completion. The comment got deleted anyway. Subsequent closing and reopening of the program resulted in disappearance of the sync groups again. The attached .json is FYI. Something is not right.
sync_groups.json
The sync_groups.json opens normally here.
With missing path of course, but it opens.
There is something different there.
Can you try to disable the Defender checks while doing the tests?
I will try to do some more testing tomorrow
OK. Tomorrow I'll try to upload a newer beta.. 🙏
Also, can you please send me an archive with any error_log_2024-04-xx.txt you'll find in the C:\Users\--YOUR_USER_NAME--\AppData\Roaming\KoHighlights directory?
After every group sync there should be a popup to inform you about a success or a failure.
The absence of one shows a program crash.
These error text hacks might show us what happened (although I haven't implement a normal logging into the app yet).
Also, can you please send me an archive with any error_log_2024-04-xx.txt you'll find in the C:\Users--YOUR_USER_NAME--\AppData\Roaming\KoHighlights directory?
Here we go
error logs KOHighlights.zip
Well, now we're getting somewhere..
I can see some errors there, but since I have uploaded many different betas, it is difficult to find all of them in the code.
Did find one that can be fixed, and there are some older ones that they shouldn't happen now, but the main discovery is a lot of PermissionError: [Errno 13] Permission denied errors happening while the app tries to read some metadata file.
This inability to read the data might create all the consecutive errors.
That means that someone is locking some metadata file and KOHighlights can't read it.
That someone can be WSA, Defender, Windows or some random other app.
WSA is the only one that I don't have here, so that might explain a lot..
This is the 1.9.9.9.beta that has the "auto reload groups before sync" feature that we discuss earlier.
You delete the current error_logs (when KOHighlights is closed of course), so we can start from scratch and see what errors will occur.
If anything strange happens on your side, you send me the new error_logs we can either fix the errors or at least recognize who is the culprit..
For a real-time check on what is happening, you can open the About KOHighlights dialog and switch to the log tab..
There is an irrelevant error in this version, that confuses the book extensions with the metadata ones.
Don't pay attention to that..
Because I found another error, I have uploaded a new 1.9.9.10.beta that fixes both of them...
.. and another one.. 1.9.9.11.beta with lots of changes under the hood.
Do you want me to test this with the defender on or off?
You can try it with the Defender on as you normally work.
If any problem happen, then we have to check different configurations.
And read the error_logs..
1.9.9.11 installed ok. Prior to the installation I deleted the old error logs. The program displayed the sync groups and archived books.
Proceeding to test:
Opened Koreader on D1 (Kohighlights is running in the background),
Made a comment on a book
Exited Koreader
Back to Kohighlights Sync / Groups view.
First attempt to sync the group did not display the "completed" dialog box .
Results:
The archived file doesn't show the new comment.
Re-opened the book on D1 and the comment is still there (at least this time it is not deleted)
Here is the error log
error_log_2024-04-27.txt
Closed and re-openned Kohighlights. One of the sync groups has disappeared from the Sync /Groups view (the one that was tested above is still there). Here is the .json
sync_groups.json
Can you send me the error_log after the not loading of the groups?
Edit to add:
It seems that one (or both) of the Calibre_Companion metadata could not be loaded.
Try to copy the .sdr folder to another location and sync them from there.
This should work.
Can you send me the error_log after the not loading of the groups?
here we go. I also tried syncing again - but the result was the same (no dialog)
error_log_2024-04-27.txt
Try to copy the .sdr folder to another location and sync them from there.
This should work.
Strange thing hapenned. I copied both android .sdr files to C:/ drive location and created another sync group for the same book (see the screenshot). Then I executed the sync on the second (new) group. It displayed the completion dialog box! After that I tried syncing the old group (then one that used to fail) and ... it also synced... 🤯🤯🤯
It's the same error that happens when syncing or loading the sync groups.
It reads the first metadata (from "F:\Macarius the Great\FIFTY ...") and when it tries to read the next ("G:\Calibre_Companion\FIFTY...") it gets nothing.
Did you try to copy the .sdr folder someplace else (after the edits) to use these as sync paths?
It's most probably the android paths that are getting locked...
Did you try to copy the .sdr folder someplace else (after the edits) to use these as sync paths?
Yes, I copied the .sdrs to C - see the post above (the screenshot shows that the second group has paths pointing to C drive). The strange thing is that after syncing the newly created group it also synced the .sdrs from the android devices
it reads the first metadata (from "F:\Macarius the Great\FIFTY ...") and when it tries to read the next
Actually, I have not tried syncing the book you mentioned. I synced "F:\Isaac the Syrian..." not "F:\Macarius the Great\FIFTY ..."
it seems that you are looking at the wrong place in the log. The Macarius book was the one that disappeared from the Sync / Groups
The paths are from the sync_groups.json second group and I used them as an example.
Here is yet another 1.9.9.12.beta that I re-enable the "Permission denied" errors that I had previously disabled.
Using this might tell as what file is getting locked..
BTW, the groups are always there, even if the paths are locked.
If you close the WSA sessions they will get loaded (with red, missing paths of course)
BTW, the groups are always there, even if the paths are locked. If you close the WSA sessions they will get loaded (with red, missing paths of course)
Let me try this and see if the missing book re-appears. I will then do a full uninstall with Revo removing all traces of 1.9.9.11 and earlier. Then I will do a clean install of 1.9.9.12
I will then do a full uninstall with Revo removing all traces of 1.9.9.11 and earlier. Then I will do a clean install of 1.9.9.12
This is not necessary, since the setup installs only the Program files folder.
Nothing special is written to the registry (except the uninstall info/paths etc.)
And the settings that I told were there are..
I shut down WSA and reopened Kohighlights. Both "Isaac the Syrian" groups are there but "Macarius" book is not there. I opened the .json file to see if it is there - it is not there either. So it has disappeared for good
The "Macarious" group was in the sync_groups.json you upload a few messages back.
Because there weren't loaded when you created the new one (because of the locks), they didn't get saved again.
installed 1.9.9.12
Recreated the Macarius group.
Opened the book, made a comment. Exited Koreader
Went back to Kohighlights
Tried to sync the group
same issue (no dialog informing about completion). The error log says there was a crash (see attached)
error_log_2024-04-27.txt
sync_groups.json
Unfortunately no extra info other than before.
The same missing metadata.
Copying the .sdr folders most probably will work as before..
A new 1.9.9.13.beta that's inform you if a file can't be read..
A new
1.9.9.13.betathat's inform you if a file can't be read..
The windows explorer can read all the files ok. The "file lock" you mention seems only to occur during the run time. I am clearly not an expert on this and I don't know what the refresh routine does, but I wonder if it can be further enhanced by forcing the program to perform some operation on the file such as reading its attributes, etc. .. In other words to force the android release the lock if it is there (just as it "releases" it when the file is selected in windows explorer). I speak as an amateur so pardon me if this doesn't make sense.
Unfortunately, what I'm doing is just accessing the files to read their contents.
The problem is that the files should be in a really peculiar state.
A simple non accessible file will produce a simple error and I could just inform you that I can't access it.
What this is doing is that sometimes it lets me access the file and then locks the process and doesn't let me read it.
I don't understand how this is possible, but it's enough low level for me to move it beyond the scope of this simple app.
Perhaps I could do more if there was an easy way to reproduce that, but at the moment there isn't.
version 1.9.9.13 displayed warning about inability to read one of the files. This allowed me to use the select button to re-add the file to the group (at this point it displayed a dialog saying that the file was already in this group). Following this it did sync ok! So we are getting somewhere.
Unfortunately, after the second group synced successfully the program lost the groups again. I tried closing and reopening the program , but alas...
Since the groups are stored in .json located on C:/ drive this is something not related to android locks. Once the sync group disappearance issue is addressed (or you find a way to reload the .json from the interface) we should probably be content with the build and resume testing once Koreader releases the new metadata version. Do you have any ideas why the groups keep disappearing?