abraunegg/onedrive

Bug: application crash due to database table missing when attempting to re-create table

Closed this issue · 17 comments

Describe the bug

Onedriver v.2.5.2 does not sync and does not recognize the database table structures.
I have already reported this bug in the OneDriveGUI app (bpozdena/OneDriveGUI#201)

Operating System Details

#uname -a 
Linux 6.1.0-25-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.106-3 (2024-08-26) x86_64 GNU/Linux

# lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

Client Installation Method

From 3rd Party Source (PPA, OpenSuSE Build Service etc)

OneDrive Account Type

Business | Office365

What is your OneDrive Application Version

onedrive v2.5.2-1+np1+1.1

What is your OneDrive Application Configuration

$ onedrive --display-config
D-Bus message bus daemon is available; GUI notifications are now enabled
Application version                          = onedrive v2.5.2-1+np1+1.1
Compiled with                                = LDC 2100
User Application Config path                 = /home/$USER/.config/onedrive
System Application Config path               = /etc/onedrive
Applicable Application 'config' location     = /home/$USER/.config/onedrive/config
Configuration file found in config location  = false - using application defaults
Applicable 'sync_list' location              = /home/$USER/.config/onedrive/sync_list
Applicable 'items.sqlite3' location          = /home/$USER/.config/onedrive/items.sqlite3
Config option 'drive_id'                     = 
Config option 'sync_dir'                     = ~/OneDrive
Config option 'enable_logging'               = false
Config option 'log_dir'                      = /var/log/onedrive
Config option 'disable_notifications'        = false
Config option 'skip_dir'                     = 
Config option 'skip_dir_strict_match'        = false
Config option 'skip_file'                    = ~*|.~*|*.tmp|*.swp|*.partial
Config option 'skip_dotfiles'                = false
Config option 'skip_symlinks'                = false
Config option 'monitor_interval'             = 300
Config option 'monitor_log_frequency'        = 12
Config option 'monitor_fullscan_frequency'   = 12
Config option 'read_only_auth_scope'         = false
Config option 'dry_run'                      = false
Config option 'upload_only'                  = false
Config option 'download_only'                = false
Config option 'local_first'                  = false
Config option 'check_nosync'                 = false
Config option 'check_nomount'                = false
Config option 'resync'                       = false
Config option 'resync_auth'                  = false
Config option 'cleanup_local_files'          = false
Config option 'classify_as_big_delete'       = 1000
Config option 'disable_upload_validation'    = false
Config option 'disable_download_validation'  = false
Config option 'bypass_data_preservation'     = false
Config option 'no_remote_delete'             = false
Config option 'remove_source_files'          = false
Config option 'sync_dir_permissions'         = 700
Config option 'sync_file_permissions'        = 600
Config option 'space_reservation'            = 52428800
Config option 'application_id'               = d50ca740-c83f-4d1b-b616-12c519384f0c
Config option 'azure_ad_endpoint'            = 
Config option 'azure_tenant_id'              = 
Config option 'user_agent'                   = ISV|abraunegg|OneDrive Client for Linux/v2.5.2-1+np1+1.1
Config option 'force_http_11'                = false
Config option 'debug_https'                  = false
Config option 'rate_limit'                   = 0
Config option 'operation_timeout'            = 3600
Config option 'dns_timeout'                  = 60
Config option 'connect_timeout'              = 10
Config option 'data_timeout'                 = 60
Config option 'ip_protocol_version'          = 0
Config option 'threads'                      = 8
Environment var 'XDG_RUNTIME_DIR'            = true
Environment var 'DBUS_SESSION_BUS_ADDRESS'   = true
Config option 'notify_file_actions'          = false

Selective sync 'sync_list' configured        = false

Config option 'sync_business_shared_items'   = false

Config option 'webhook_enabled'              = false

What is your 'curl' version

$ curl --version
curl 7.88.1 (x86_64-pc-linux-gnu) libcurl/7.88.1 OpenSSL/3.0.14 zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 libidn2/2.3.3 libpsl/0.21.2 (+libidn2/2.3.3) libssh2/1.10.0 nghttp2/1.52.0 librtmp/2.3 OpenLDAP/2.5.13
Release-Date: 2023-02-20, security patched: 7.88.1-10+deb12u7
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

Where is your 'sync_dir' located

Local

What are all your system 'mount points'

$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=6049368k,nr_inodes=1512342,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1215852k,mode=755,inode64)
/dev/sdb2 on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=674)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/sdb3 on /home type ext4 (rw,relatime)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
/dev/sda1 on /mnt/sda1 type ext4 (rw,nosuid,nodev,relatime,x-gvfs-show)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=1215848k,nr_inodes=303962,mode=700,uid=1000,gid=1000,inode64)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
Nextcloud-3.1.3-x86_64.AppImage on /tmp/.mount_NextclGwug5m type fuse.Nextcloud-3.1.3-x86_64.AppImage (ro,nosuid,nodev,relatime,user_id=1000,group_id=1000)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

What are all your local file system partition types

$ lsblk -f
NAME FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda                                                                         
└─sda1
     ext4   1.0   DiscoBackup
                        0e69004b-3081-44cc-a8f4-75a66f4e2413   23,7G    90% /mnt/sda1
sdb                                                                         
├─sdb1
│    swap   1           25b65447-7e7e-46cc-bfdf-a735676eb487                [SWAP]
├─sdb2
│    ext4   1.0         b58b066e-9af4-488f-8eea-6ebc5818fdb7    5,9G    79% /
└─sdb3
     ext4   1.0         bf82b6d1-ff28-4cda-9964-e572bf932b52   13,7G    92% /home
sdc                                                                         
sr0

How do you use 'onedrive'

My institution bought an Office365 licence for the staff. I use Debian 12 on my desktop and notebook and share documents between them and with other colleagues.

Steps to reproduce the behaviour

  1. Install onedriver v.2.5.2 following instructions for Debian 12 (via OpenSuse Building System) and dependencies.
  2. Lunch onedriver and configure the access.
  3. Onedriver does not recognize the database structure.

Complete Verbose Log Output

$ onedrive --confdir='/home/$USER/.config/onedrive/accounts/name.surname@email.com' --monitor -v
Reading configuration file: /home/$USER/.config/onedrive/accounts/name.surname@email.com/config
The option 'force_http_2' has been deprecated and will be ignored. Please read the updated documentation and update your client configuration to remove this option.
Configuration file successfully loaded
Using 'user' configuration path for application config and state data: /home/$USER/.config/onedrive/accounts/name.surname@email.com
D-Bus message bus daemon is available; GUI notifications are now enabled
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
The item database is incompatible, re-creating database table structures
sqlite.SqliteException@src/sqlite.d(141): no such table: item
----------------
??:? [0x559f1ddb7617]
??:? [0x559f1ddbaf08]
??:? [0x559f1de855dc]
??:? void rt.dmain2._d_run_main2(char[][], ulong, extern (C) int function(char[][])*).runAll() [0x7f185eb9db7b]
??:? _d_run_main2 [0x7f185eb9d995]
??:? _d_run_main [0x7f185eb9d7ed]
??:? [0x559f1debb7a1]
??:? [0x7f185e8fb249]
??:? __libc_start_main [0x7f185e8fb304]
??:? [0x559f1dd8f220]

Screenshots

No response

Other Log Information or Details

No response

Additional context

No response

@ilchimicopazzo
Please can you test PR #2919 to resolve your issue.

First install all the require platform dependencies to build the client on your platforms. Please read https://github.com/abraunegg/onedrive/blob/master/docs/install.md#building-from-source---high-level-requirements and then follow correctly for your platform.

Once this is done, to clone the PR to resolve your issue, you can use a script like the following:

#!/bin/bash

PR=2919 

rm -rf ./onedrive-pr${PR}
git clone https://github.com/abraunegg/onedrive.git onedrive-pr${PR}
cd onedrive-pr${PR}
git fetch origin pull/${PR}/head:pr${PR}
git checkout pr${PR}

# Configure and Build
./configure --enable-debug --enable-notifications; make clean; make;
./onedrive --version

This script will create a local folder called onedrive-pr2919 with the PR version.

To run the PR, you need to run the client from the PR build directory:

./onedrive <any other options needed>

To install the PR, you will need to perform sudo make install to install the PR version to your system.

When running the PR, your version should be: onedrive v2.5.2-16-g4691fac or greater.

Current Master

Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application config and state data: /home/alex/.config/onedrive
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
The item database is incompatible, re-creating database table structures
sqlite.SqliteException@src/sqlite.d(141): no such table: item
----------------
src/sqlite.d:29 void sqlite.Database.exec(const(char)[]) [0x7678ce]
src/itemdb.d:293 itemdb.ItemDatabase itemdb.ItemDatabase.__ctor(immutable(char)[]) [0x761293]
src/main.d:485 _Dmain [0x5dfdf2]

PR#2919

Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application config and state data: /home/alex/.config/onedrive
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
The item database is incompatible, re-creating database table structures
WARNING: Table 'item' does not exist, skipping table drop.     <--- Your Issue Fixed
Application Version:  onedrive v2.5.2-16-g4691fac
Account Type:         personal
Default Drive ID:     66d53be8a5056eca
Default Root ID:      66D53BE8A5056ECA!101
Remaining Free Space: 4.80 GB (5153960755 bytes)
Sync Engine Initialised with new Onedrive API instance
.........

@ilchimicopazzo
Any update in testing the provided PR for resolution of this issue?

@ilchimicopazzo
Please can you respond. This issue is holding up prepping a v2.5.3 release.

I have a similar issue on latest master (onedrive v2.5.2-24-g99cd975), only my error is that table already exists. I tried to upgrade from version 2.4.

Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Configuring Global Azure AD Endpoints
The item database is incompatible, re-creating database table structures
WARNING: Table 'item' does not exist, skipping table drop.
sqlite.SqliteException@src/sqlite.d(150): table item already exists
----------------
??:? void sqlite.Database.exec(const(char)[]) [0x5e2d9f28178d]
??:? void itemdb.ItemDatabase.createTable() [0x5e2d9f27b590]
??:? itemdb.ItemDatabase itemdb.ItemDatabase.__ctor(immutable(char)[]) [0x5e2d9f27b275]
??:? _Dmain [0x5e2d9f0f3b53]

EDIT:

The normal 2.5.2 release works actually with the same config.

@Abasz
Can you please provide a detailed reproduction here ...

If the table does not exist, it is skippes from being dropped .. hence the message generated.

This is a very odd error I am unable to reproduce.

Thanks for the quick response.

I build with dmd (v109) as per the docs, the latest master (v2.5.2-24-g99cd975), then run: /usr/local/bin/onedrive --sync --confdir=/root/.config/onedrive --verbose

config

skip_file = "_vti_cnf|~*|*.tmp|*RECYCLE.BIN|desktop.ini|*.adi"
monitor_interval="3600"
sync_dir="/home/onedrive/"
classify_as_big_delete="30000"

Its a onedrive business account if it may matter.

Interesting part is that when I downgraded to tag 2.5.2, everything works (with the same command). Its currently running a sync (with tag 2.5.2). Once its done I will upgrade to master again and try.

What other details of reproduction can I provide you with? There is nothing special I think in my config. I dont use monitor.

More so your entire process

You were on v2.4.x and then upgraded to . ?

I was starting on tag: v2.4.23. With a fully synced and operational database (though those syncs were made with --synchronise). Last sync was done at 1am today (completed with Sync with OneDrive is complete).

  1. I recloned the repo (head at latest master commit),
  2. activated dmd,
  3. run ./configure (no flags); make clean; make; make install;
  4. deactivated dmd
  5. run onedrive --version which returned onedrive v2.5.2-24-g99cd975
  6. run the command: /usr/local/bin/onedrive --sync --confdir=/root/.config/onedrive --verbose
  7. got the error.
  8. run the command: /usr/local/bin/onedrive --synchronise--confdir=/root/.config/onedrive --verbose
  9. got the same error

Then

  1. deleted repo,
  2. recloned the repo
  3. checked out tag 2.5.2.
  4. activated dmd,
  5. run ./configure (no flags); make clean; make; make install;
  6. deactivated dmd
  7. run onedrive --version that returned onedrive v2.5.2
  8. run the command: /usr/local/bin/onedrive --sync --confdir=/root/.config/onedrive --verbose
  9. started syncing.

@Abasz
Thanks. Will investigate this tomorrow.

@Abasz Thanks. Will investigate this tomorrow.

Thanks, I will revert back if after syncing with tag 2.5.2 and upgrading to latest master has the same issue or not.

@Abasz
I have been able to reproduce this scenario:

Old Master (v2.5.2-24-g99cd975)

Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application config and state data: /home/alex/.config/onedrive
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
The item database is incompatible, re-creating database table structures
WARNING: Table 'item' does not exist, skipping table drop.
sqlite.SqliteException@src/sqlite.d(150): table item already exists
----------------
src/sqlite.d:38 void sqlite.Database.exec(const(char)[]) [0x76831e]
src/itemdb.d:378 void itemdb.ItemDatabase.createTable() [0x762037]
src/itemdb.d:294 itemdb.ItemDatabase itemdb.ItemDatabase.__ctor(immutable(char)[]) [0x761cc4]
src/main.d:498 _Dmain [0x5e0125]

Updated Code: Use Case - Upgrade from v2.4.x DB to v2.5.x Master

[alex@onedrive-client-dev fix-issue-2920v2]$ 
[alex@onedrive-client-dev fix-issue-2920v2]$ 
[alex@onedrive-client-dev fix-issue-2920v2]$ 
[alex@onedrive-client-dev fix-issue-2920v2]$ ./onedrive -s -v   
Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application config and state data: /home/alex/.config/onedrive
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
The item database is incompatible, re-creating database table structures
Application Version:  onedrive v2.5.2-24-g99cd975
Account Type:         personal
.......

Updated Code: Use Case - Database missing table (original issue|fix)

Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application config and state data: /home/alex/.config/onedrive
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
The item database is incompatible, re-creating database table structures
WARNING: Table 'item' does not exist, skipping table drop.
Application Version:  onedrive v2.5.2-24-g99cd975
.......

Will merge this into 'master' shortly.

@Abasz
This is now fixed in 'master':

Reading configuration file: /home/alex/.config/onedrive/config
Configuration file successfully loaded
Using 'user' configuration path for application config and state data: /home/alex/.config/onedrive
Using IPv4 and IPv6 (if configured) for all network operations
Attempting to contact Microsoft OneDrive Login Service
Successfully reached Microsoft OneDrive Login Service
Checking Application Version ...
Attempting to initialise the OneDrive API ...
Configuring Global Azure AD Endpoints
The OneDrive API was initialised successfully
Opening the item database ...
The item database is incompatible, re-creating database table structures
Application Version:  onedrive v2.5.2-25-g2267a23
Account Type:         personal
...

Thanks, I was going to test this this Morning but I noticed that the sync has not completed (its been running for like 30 hours now)...

It is logging this:

This local file has a different modified time 2024-Oct-22 12:35:52Z (UTC) when compared to remote modified time 2024-Aug-24 20:31:09Z (UTC)
Local item has the same hash value as the item online - correcting the applicable file timestamp
The source of the incorrect timestamp was OneDrive online - correcting timestamp online

and the items.sqlite3-wal file is 13 GB. I mean I have a lot of files but 13 GB?

This is with tag 2.5.2, same command etc.

Anyway, I think this is unrelated issues from the update from 2.4.x to 2.5.2-master bug, just wanted to mention. I will let it finish and then update to latest master. Thanks for your patience

@Abasz

Thanks, I was going to test this this Morning but I noticed that the sync has not completed (its been running for like 30 hours now)...

Please read:

and the items.sqlite3-wal file is 13 GB. I mean I have a lot of files but 13 GB?

The items.sqlite3-wal file is a "Write-Ahead Log" (WAL) associated with your SQLite database. When SQLite is in WAL mode, it writes changes to this .sqlite3-wal file rather than directly into the main database file (items.sqlite3).

These are committed into the database at periodic intervals, and at the end of the sync process. It is nothing to be concerned about.

Thanks, I missed those release notes... should have read it. Anyway I checked and all is working fine. cache database is gone too.

Thanks for the prompt help!

@ilchimicopazzo Please can you respond. This issue is holding up prepping a v2.5.3 release.

Hi @abraunegg
I am sorry for the delay.
Please, write to me if you still need my check with PR #2919.
Otherwise, I will suppose that you don't need my check.

Thanks again for your support!
Bye bye

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.