Changed file IDs in Groupfolders
gzoritchak opened this issue · 13 comments
We experienced the same issue mentioned here:
https://help.nextcloud.com/t/changed-file-ids-in-groupfolders/91916
It's very annoying since we used a root spreadsheet to referenced a lot of internal documents. All links are now broken.
@juliushaertl @icewind1991 Are the file ids meant to stay unchanged? If yes, this is a bug. If no there should be something in the docs that informs users about this. Personally I think it is a bug? What's your point of view?
Yes, file ids should remain unchanged and I haven't encountered such a behavior until now. Please fill out the issue template and it would be especially interesting if there is any groupfolder related errors in the logs. Might also be worth to check if there was any suspicious action like a delete or move during the time this occured in the access logs of the webserver.
Hi @juliushaertl there was nothing in the logs either about group folders or anything else. The problem faced by @gzoritchak has arrived at night, with little or no activity at all.
After investigation, at the SQL dump from 3h26am (nightly backup) & production at 13h43 the same day, we could obseve an ID 2281 and an ID 6208 for the same file, exposing this ID increment, respectivelly (this is an example, all files were affected).
There was nothing unusual in the webserver, database, php or nextcloud logs. Only things related to #1381 and #1472 essentially.
@juliushaertl when we encountered this, there was a bulk file ownership change on the server. The file ownerships were adjusted to match the ones used by the Nextcloud Docker image. Some files were previously owned by the root user and group.
The change was made with chown.
Also experienced the same issue, we did a files:scan for manually imported files, and some file ids have changed in other group folders, maybe this has something to do with the ids changing
we have the same problem after moving the data directory....
I have checked that in the table oc_filecache, entries are duplicated. Example below..
Have anyone a solution for this problem?
Our current idea is that we replace the fileids of the new database with the old one... but if that will work...
would be great if someone has another idea that does not attack the DB...
example: (same path but different fileid)
MariaDB [nextcloud]> SELECT * FROM oc_filecache WHERE fileid=3485643;
+---------+---------+------------------------------------------------+----------------------------------+---------+-----------+----------+----------+----------+------------+---------------+-----------+----- -------------+---------------+-------------+----------+
| fileid | storage | path | path_hash | parent | name | mimetype | mimepart | size | mtime | storage_mtime | encrypted | unen crypted_size | etag | permissions | checksum |
+---------+---------+------------------------------------------------+----------------------------------+---------+-----------+----------+----------+----------+------------+---------------+-----------+----- -------------+---------------+-------------+----------+
| 3485643 | 280 | __groupfolders/83/2021-09-10_ISAB-02/Upload AM | e9779aced0ca83c359098cc637dfebf1 | 3485639 | Upload AM | 2 | 1 | 61154122 | 1629368109 | 1629367052 | 0 | 0 | 611e2f2d32386 | 31 | |
+---------+---------+------------------------------------------------+----------------------------------+---------+-----------+----------+----------+----------+------------+---------------+-----------+----- -------------+---------------+-------------+----------+
1 row in set (0.00 sec)
MariaDB [nextcloud]> SELECT * FROM oc_filecache WHERE fileid=1689517;
+---------+---------+------------------------------------------------+----------------------------------+---------+-----------+----------+----------+----------+------------+---------------+-----------+----- -------------+---------------+-------------+----------+
| fileid | storage | path | path_hash | parent | name | mimetype | mimepart | size | mtime | storage_mtime | encrypted | unen crypted_size | etag | permissions | checksum |
+---------+---------+------------------------------------------------+----------------------------------+---------+-----------+----------+----------+----------+------------+---------------+-----------+----- -------------+---------------+-------------+----------+
| 1689517 | 1 | __groupfolders/83/2021-09-10_ISAB-02/Upload AM | e9779aced0ca83c359098cc637dfebf1 | 1689457 | Upload AM | 2 | 1 | 48815043 | 1629124416 | 1629124416 | 0 | 0 | 611a7740865db | 31 | |
+---------+---------+------------------------------------------------+----------------------------------+---------+-----------+----------+----------+----------+------------+---------------+-----------+----- -------------+---------------+-------------+----------+
1 row in set (0.00 sec)
When moving the data directory to a different path this is expected as the existing filecache entries are not longer valid for the new location. For a possible workaround for that see https://help.nextcloud.com/t/howto-change-move-data-directory-after-installation/17170 however this is not a supported approach.
Note that execution of the scan command could cause creation of new filecache entries if Nextcloud is not able to map the existing entries to the path.
after this instruction I have actually also proceeded... I have just not updated the oc_storage table... should be no longer necessary from version 13.1 I thought...
but now our problem is that I have moved the directory 3 days ago and the operation continued normally... today we have seen that the shares are no longer there...
Is this now a problem if I adjust the oc_storage afterwards?
Has anyone ever found the reason for this issue (besides moving the data folder)?
Still happening with our nextcloud installation on the currently latest version.
When reading related code I might have a guess where this could potentially go wrong. I prepared a patch that could provide some logging in this case for the Nextcloud server itself.
index 3d5a2f098b2..2dadfe66d50 100644
--- a/lib/private/Files/Storage/Common.php
+++ b/lib/private/Files/Storage/Common.php
@@ -894,6 +894,9 @@ abstract class Common implements Storage, ILockingStorage, IWriteStreamStorage {
public function getDirectoryContent($directory): \Traversable {
$dh = $this->opendir($directory);
+ if ($dh === false) {
+ throw new \Exception('opendir failed, not yielding any results');
+ }
if (is_resource($dh)) {
$basePath = rtrim($directory, '/');
while (($file = readdir($dh)) !== false) {If anyone who encounters this regularly could apply this patch to their instance and share the log results to see if this part is logged with a trace, this would be much appreciated.
More context for further debugging:
When a scan runs for some reason this could lead to no results which then causes the scanner to diff and remove those entries, the next scan probably readds it again with new file ids:
https://github.com/nextcloud/server/blob/ea06cf2f39c5c4e7f1ea20d075c01e3ba0cc6dd0/lib/private/Files/Cache/Scanner.php#L427
https://github.com/nextcloud/server/blob/ea06cf2f39c5c4e7f1ea20d075c01e3ba0cc6dd0/lib/private/Files/Cache/Scanner.php#L494
The same might happen with scanFile where getMetaData/stat call could return false which is not considered an error case, but can lead to a remove from cache.
Ok, opendir should still log an error already unless E_WARNING is suppressed via PHP config.
I managed to reproduce the new fileids with some fairly simple steps:
- create a groupfolder, added some files, inspected the database
- move the
__groupfoldersfolder to__groupfolder - run
occ groupfolders:scanfor the folder in question - move the folder back to
__groupfolders - run the scan again
- see new file ids, lost tags and shares and so on.
If this behavior persists I suspect that scans with a missing groupfolder are causing this bug.
We can prevent this by checking if the __groupfolders folder is present before running any scan. Otherwise the scan should err out.
We can also check if the folder for the groupfolder ( __groupfolders/123 ) in question is still present before performing a scan. If it's not we should inform the admin rather than dropping all the file cache entries.
So far I reproduced this on Nextcloud 23. I will try current master next.