Bug: Overwrite oldest file

Question

Bug: Overwrite oldest file

Closed this issue a month ago · 18 comments

Is your feature request related to a problem? Please describe.

Yes
I use my phone (android) and my Linux setup to sync with onedrive.
So, imagine this scenario

I use a phone and pc for syncing
Modify a file in phone and then sync with Onedrive
Modify the file from 2 in PC the then sync with Onedrive
bypass_data_preservation is false and I'll have two files.

Describe the solution you'd like

If I set bypass_data_preservation as true, it will use the file from 2 and disregard the file from 3
But What I want to happen is to not have two files and also the file from 3 replace the one from 2

Describe alternatives you've considered

An option like overwrite_oldest_file would be nice
What does this do? the file that has older timestamp will be replaced with the newer one.

Additional context

No sync is done in PC between 2 and 3

abraunegg commented a month ago

Great !

abraunegg commented a month ago

Thanks

❤️1

Answer 1 · 2024-10-03T23:58:01.000Z

What does this do? the file that has older timestamp will be replaced with the newer one.

The older file already gets overwritten if the file is older and the content is different.

If the older file content is the same, the older timestamp is corrected.

If the older file generates a conflict, it gets renamed.

Please review the documentation.

Answer 2 · 2024-10-04T00:13:17.000Z

If the older file generates a conflict, it gets renamed.

How can I get this?
If the older file generates a conflict, it gets replaced.
Instead of
If the older file generates a conflict, it gets renamed.

I also tried bypass_data_preservation as true but it still generated a duplicate file "to avoid potential local data loss."

[hl@DIAMOND ~]$ onedrive -s
Reading configuration file: /home/hl/.config/onedrive/config
Configuration file successfully loaded
D-Bus message bus daemon is available; GUI notifications are now enabled
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Configuring Global Azure AD Endpoints
WARNING: Application has been configured to bypass local data preservation in the event of file conflict.
WARNING: Local data loss MAY occur in this scenario.
Fetching items from the OneDrive API for Drive ID: 5a69b9216e4e0694 .. 
Processing 3 applicable changes and items received from Microsoft OneDrive . 
Number of items to download from Microsoft OneDrive: 1
The local file to replace (Notes/Brain/Repeated Mistakes.md) has been modified locally since the last download. Renaming it to avoid potential local data loss.
Downloading file: Notes/Brain/Repeated Mistakes.md ... done
Performing a database consistency and integrity check on locally stored data .. 
Scanning the local file system '~/OneDrive' for new data to upload . 
New items to upload to Microsoft OneDrive: 1
Uploading new file: ./Notes/Brain/Repeated Mistakes-DIAMOND.md ... done
Performing a last examination of the most recent online data within Microsoft OneDrive to complete the reconciliation process
Fetching items from the OneDrive API for Drive ID: 5a69b9216e4e0694 .. 
Processing 3 applicable changes and items received from Microsoft OneDrive . 

Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

Answer 3 · 2024-10-04T00:19:08.000Z

Correct the local file is renamed to avoid data loss aa per the application output. This functionality is there to ensure you do not suffer data loss and the integrity of your data is maintained.

Overwriting the local file would consitute data loss - and what you want to do is intentionally have data loss.

Answer 4 · 2024-10-04T00:25:11.000Z

Overwriting the local file would consitute data loss - and what you want to do is intentionally have data loss.

I guess.
Imagine I have an empty text file
I edit its content in my phone with this
Phone
and then sync with Onedrive
I do the same in my Linux but the content is
Linux
And now do a sync from my Linux
What I want is data loss which in this case the file containing Phone in Onedrive cloud gets replaced by the same file with Linux as its content

Basically, when there is a conflict, I want the older file to get replaced by the newer one. In a sense an option to control over conflicts. My problem is with duplication and this scenario is the cause.

Also, I'm also still confused about bypass_data_preservation. Is it not supposed to not make duplicate if it's set as ture? Why does it make duplicate for me?

Answer 5 · 2024-11-07T17:50:52.000Z

Using inotify can significantly improve performance and synchronization by enabling real-time, local monitoring of file changes. With inotify, files can be immediately uploaded when changes are detected, reducing the delay typically required to check for cloud updates. However, resource consumption should be carefully monitored to ensure efficient usage. If resource utilization remains comparable to that of OneDrive on Windows, this approach would be highly effective.

Answer 6 · 2024-11-07T19:34:12.000Z

Using inotify can significantly improve performance and synchronization by enabling real-time, local monitoring of file changes. With inotify, files can be immediately uploaded when changes are detected, reducing the delay typically required to check for cloud updates. However, resource consumption should be carefully monitored to ensure efficient usage. If resource utilization remains comparable to that of OneDrive on Windows, this approach would be highly effective.

The client already uses inotify when in --monitor mode to watch for local file changes

Answer 7 · 2024-11-07T23:56:19.000Z

@BEST8OY

Also, I'm also still confused about bypass_data_preservation. Is it not supposed to not make duplicate if it's set as ture? Why does it make duplicate for me?

Specifically for this, that is a bug looking at the code. The application , when it hits this section:

The local file to replace (path/filename) has been modified locally since the last download. Renaming it to avoid potential local data loss.

should not do this IF bypass_data_preservation is set to true.

Answer 8 · 2024-11-13T19:49:11.000Z

@BEST8OY
Please can you test PR #2967 to resolve your issue.

First install all the require platform dependencies to build the client on your respective platforms. Please read https://github.com/abraunegg/onedrive/blob/master/docs/install.md#building-from-source---high-level-requirements and then follow correctly for your platform.

Once this is done, to clone the PR to resolve your issue, you can use a script like the following:

#!/bin/bash

PR=2967

rm -rf ./onedrive-pr${PR}
git clone https://github.com/abraunegg/onedrive.git onedrive-pr${PR}
cd onedrive-pr${PR}
git fetch origin pull/${PR}/head:pr${PR}
git checkout pr${PR}

# Configure and Build
./configure --enable-debug --enable-notifications; make clean; make;
./onedrive --version

This script will create a local folder called onedrive-pr2967 with the PR version.

To run the PR, you need to run the client from the PR build directory:

./onedrive <any other options needed>

To install the PR, you will need to perform sudo make install to install the PR version to your system.

When running the PR, your version should be: onedrive v2.5.2-48-gfb7dcda or greater.

In addition to this, please read:

https://github.com/abraunegg/onedrive/blob/master/docs/client-architecture.md#file-conflict-handling---default-operational-modes

Answer 9 · 2024-11-13T21:33:51.000Z

Please can you test PR #2967 to resolve your issue.

Thanks for the effort.

Duplication with bypass_data_preservation set as true is fixed
The docs are really comprehensive and helpful. Based on the last link I figured that based on what scenario I'm in I have to switch between onedrive -s and onedrive -s --local-first
During the testing I faced a bug with --dry-run. It seems that --dry-run doesn't respect --local-first and operate with default configuration of conflict handling.

Here is the scenario

Make a file locally and onedrive -s
Edit the online file contents
Edit the local file contents
Now I want to upload the local file so I must use onedrive -s --local-first and if I do onedrive -s it will download the online file even though the online file is older.
But before doing onedrive -s --local-first I did onedrive -s --local-first --dry-run and I faced the issue of --dry-run not respecting --local-first

[hl@DIAMOND ~/OD-Test/onedrive-pr2967]$ ./onedrive -s --local-first --dry-run
Reading configuration file: /home/hl/.config/onedrive/config
Configuration file successfully loaded
D-Bus message bus daemon is available; GUI notifications are now enabled
DRY-RUN Configured. Output below shows what 'would' have occurred.
DRY-RUN: Copying items.sqlite3 to items-dryrun.sqlite3 to use for dry run operations
DRY RUN: Not creating backup config file as --dry-run has been used
DRY RUN: Not updating hash files as --dry-run has been used
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Configuring Global Azure AD Endpoints
WARNING: Application has been configured to bypass local data preservation in the event of file conflict.
WARNING: Local data loss MAY occur in this scenario.
Performing a database consistency and integrity check on locally stored data .. 
Changed local items to upload to Microsoft OneDrive: 1
Uploading modified file: Notes/Brain/test.md ... done
Scanning the local file system '~/OneDrive' for new data to upload . 
Fetching items from the OneDrive API for Drive ID: 5a69b9216e4e0694 .. 
Processing 3 applicable changes and items received from Microsoft OneDrive . 
Number of items to download from Microsoft OneDrive: 1
The local file to replace (Notes/Brain/test.md) has been modified locally since the last download. Renaming it to avoid potential local data loss.
WARNING: Local Data Protection has been disabled - not renaming local file. You may experience data loss on this file: Notes/Brain/test.md
Downloading file: Notes/Brain/test.md ... done

Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete


[hl@DIAMOND ~/OD-Test/onedrive-pr2967]$ ./onedrive -s --local-first          
Reading configuration file: /home/hl/.config/onedrive/config
Configuration file successfully loaded
D-Bus message bus daemon is available; GUI notifications are now enabled
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Configuring Global Azure AD Endpoints
WARNING: Application has been configured to bypass local data preservation in the event of file conflict.
WARNING: Local data loss MAY occur in this scenario.
Performing a database consistency and integrity check on locally stored data .. 
Changed local items to upload to Microsoft OneDrive: 1
Uploading modified file: Notes/Brain/test.md ... done
Scanning the local file system '~/OneDrive' for new data to upload .. 
Fetching items from the OneDrive API for Drive ID: 5a69b9216e4e0694 . 
Processing 3 applicable changes and items received from Microsoft OneDrive . 

Sync with Microsoft OneDrive is complete
Attempting to perform a database vacuum to optimise database
Database vacuum is complete

Answer 10 · 2024-11-13T21:37:58.000Z

When using --dry-run renames, uploads, downloads are a no-op

So the online OneDrive data is not changed - thus, when using that option it will still send you the online file in its existing state.

Answer 11 · 2024-11-13T21:45:58.000Z

When using --dry-run renames, uploads, downloads are a no-op

So the online OneDrive data is not changed - thus, when using that option it will still send you the online file in its existing state.

I know what --dry-run does!
The problem is ./onedrive -s --local-first --dry-run says Downloading file: Notes/Brain/test.md ... done and ./onedrive -s --local-first says Uploading modified file: Notes/Brain/test.md ... done which is correct.

I was about to be misled by the --dry-run report. Or is that the design of --dry-run?

Answer 12 · 2024-11-13T21:48:44.000Z

Okay I missed Uploading modified file: Notes/Brain/test.md ... done line in the --dry-run report.
It's all right!
Sorry for the inconvenience!

Answer 13 · 2024-11-13T22:04:40.000Z

Given your testing, you can confirm this issue is resolved ?

Answer 14 · 2024-11-13T22:14:18.000Z

Given your testing, you can confirm this issue is resolved ?

Yes. Resloved!

Answer 15 · 2024-11-13T22:26:09.000Z

Merged fix into master

Answer 16 · 2024-11-21T01:59:26.000Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.