Limit search to certain folders

Question

Limit search to certain folders

Closed this issue 7 years ago · 7 comments

Is there a way to limit the search scope? The new version is slower for me and I think it is because it is searching my entire Google drive which has several terabytes of data.

Here is what my log file looks like:
[2017-09-05 11:20:37.699] [27607] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:20:40.360] [27607] [DEBUG] Loaded 149335 items from cache (created 5 days ago)
[2017-09-05 11:20:42.605] [27607] [DEBUG] Execution took 4.908 seconds
[2017-09-05 11:20:42.640] [27615] [DEBUG] Start background cache update
[2017-09-05 11:20:42.694] [27615] [INFO] Access token expired 5 days ago
[2017-09-05 11:20:42.998] [27621] [DEBUG] ./google-drive.rb ["--filter", "test"]
[2017-09-05 11:20:43.001] [27621] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:20:43.002] [27621] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:20:43.206] [27615] [INFO] Access token added to keychain, expires in 3600 seconds
[2017-09-05 11:20:45.217] [27621] [DEBUG] Loaded 149335 items from cache (created 5 days ago)
[2017-09-05 11:20:46.143] [27621] [DEBUG] Execution took 3.145 seconds
[2017-09-05 11:20:46.174] [27629] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:20:46.421] [27631] [DEBUG] ./google-drive.rb ["--filter", "test "]
[2017-09-05 11:20:46.424] [27631] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:20:46.424] [27631] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:20:48.420] [27631] [DEBUG] Loaded 149335 items from cache (created 5 days ago)
[2017-09-05 11:20:49.386] [27631] [DEBUG] Execution took 2.965 seconds
[2017-09-05 11:20:49.416] [27637] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:20:49.647] [27639] [DEBUG] ./google-drive.rb ["--filter", "test "]
[2017-09-05 11:20:49.648] [27639] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:20:49.649] [27639] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:20:51.517] [27639] [DEBUG] Loaded 149335 items from cache (created 5 days ago)
[2017-09-05 11:20:52.693] [27639] [DEBUG] Execution took 3.045 seconds
[2017-09-05 11:20:52.743] [27642] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:20:52.839] [27615] [INFO] Got 460 items from Google Drive

Answer 1 · 2017-09-05T18:40:33.000Z

It says “Loaded 149335 items from cache” which does indeed sound like a problem (i.e. slow) But then the last line (where it gets updated items from Google Drive) says “Got 460 items from Google Drive”. That indicates that you “only” have 460 items and after that, there should only be 460 items in the cache, which should load a lot faster than the 149335 items. So what does the log say after having refreshed the cache? And how many items (roughly) are actually on your Google Drive?

…

On 5 Sep 2017, at 20:28, Mike King wrote: Is there a way to limit the search scope? The new version is slower for me and I think it is because it is searching my entire Google drive which has several terabytes of data. Here is what my log file looks like: [2017-09-05 11:20:37.699] [27607] [DEBUG] Latest online version is v2.0, we are running 2.0 [2017-09-05 11:20:40.360] [27607] [DEBUG] Loaded 149335 items from cache (created 5 days ago) [2017-09-05 11:20:42.605] [27607] [DEBUG] Execution took 4.908 seconds [2017-09-05 11:20:42.640] [27615] [DEBUG] Start background cache update [2017-09-05 11:20:42.694] [27615] [INFO] Access token expired 5 days ago [2017-09-05 11:20:42.998] [27621] [DEBUG] ./google-drive.rb ["--filter", "test"] [2017-09-05 11:20:43.001] [27621] [DEBUG] Loaded 12 releases from cache (created 5 days ago) [2017-09-05 11:20:43.002] [27621] [DEBUG] Latest online version is v2.0, we are running 2.0 [2017-09-05 11:20:43.206] [27615] [INFO] Access token added to keychain, expires in 3600 seconds [2017-09-05 11:20:45.217] [27621] [DEBUG] Loaded 149335 items from cache (created 5 days ago) [2017-09-05 11:20:46.143] [27621] [DEBUG] Execution took 3.145 seconds [2017-09-05 11:20:46.174] [27629] [DEBUG] Skip background cache update: Another process is already running [2017-09-05 11:20:46.421] [27631] [DEBUG] ./google-drive.rb ["--filter", "test "] [2017-09-05 11:20:46.424] [27631] [DEBUG] Loaded 12 releases from cache (created 5 days ago) [2017-09-05 11:20:46.424] [27631] [DEBUG] Latest online version is v2.0, we are running 2.0 [2017-09-05 11:20:48.420] [27631] [DEBUG] Loaded 149335 items from cache (created 5 days ago) [2017-09-05 11:20:49.386] [27631] [DEBUG] Execution took 2.965 seconds [2017-09-05 11:20:49.416] [27637] [DEBUG] Skip background cache update: Another process is already running [2017-09-05 11:20:49.647] [27639] [DEBUG] ./google-drive.rb ["--filter", "test "] [2017-09-05 11:20:49.648] [27639] [DEBUG] Loaded 12 releases from cache (created 5 days ago) [2017-09-05 11:20:49.649] [27639] [DEBUG] Latest online version is v2.0, we are running 2.0 [2017-09-05 11:20:51.517] [27639] [DEBUG] Loaded 149335 items from cache (created 5 days ago) [2017-09-05 11:20:52.693] [27639] [DEBUG] Execution took 3.045 seconds [2017-09-05 11:20:52.743] [27642] [DEBUG] Skip background cache update: Another process is already running [2017-09-05 11:20:52.839] [27615] [INFO] Got 460 items from Google Drive -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #38

Answer 2 · 2017-09-05T18:52:15.000Z

I have unlimited storage and have a bunch of backup files in google drive. I think 150,000+ files sounds right. That is why I was thinking it would be nice if I could limit the search to exclude the backup folder. Here is the log with a few more searches. It looks like it is getting a new cache file each time.

[2017-09-05 11:35:09.640] [27615] [DEBUG] Finished background cache update
[2017-09-05 11:44:02.976] [29416] [DEBUG] ./google-drive.rb ["--filter", "t"]
[2017-09-05 11:44:02.979] [29416] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:02.980] [29416] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:05.587] [29416] [DEBUG] Loaded 149344 items from cache (created 23 minutes ago)
[2017-09-05 11:44:08.573] [29416] [DEBUG] Execution took 5.596 seconds
[2017-09-05 11:44:08.611] [29426] [DEBUG] Start background cache update
[2017-09-05 11:44:09.057] [29432] [DEBUG] ./google-drive.rb ["--filter", "test "]
[2017-09-05 11:44:09.059] [29432] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:09.060] [29432] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:10.646] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:12.101] [29432] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:12.450] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:13.418] [29432] [DEBUG] Execution took 4.361 seconds
[2017-09-05 11:44:13.447] [29442] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:13.680] [29444] [DEBUG] ./google-drive.rb ["--filter", "test "]
[2017-09-05 11:44:13.681] [29444] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:13.681] [29444] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:14.485] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:15.475] [29444] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:15.840] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:16.322] [29444] [DEBUG] Execution took 2.642 seconds
[2017-09-05 11:44:16.390] [29446] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:16.886] [29448] [DEBUG] ./google-drive.rb ["--filter", "n"]
[2017-09-05 11:44:16.887] [29448] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:16.888] [29448] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:18.134] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:18.895] [29448] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:19.536] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:21.026] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:21.053] [29448] [DEBUG] Execution took 4.167 seconds
[2017-09-05 11:44:21.086] [29454] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:21.380] [29456] [DEBUG] ./google-drive.rb ["--filter", "new test"]
[2017-09-05 11:44:21.381] [29456] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:21.381] [29456] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:22.399] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:23.324] [29456] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:23.766] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:24.000] [29456] [DEBUG] Execution took 2.62 seconds
[2017-09-05 11:44:24.033] [29464] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:24.274] [29466] [DEBUG] ./google-drive.rb ["--filter", "new test"]
[2017-09-05 11:44:24.275] [29466] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:24.275] [29466] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:24.920] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:26.103] [29466] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:26.354] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:26.828] [29466] [DEBUG] Execution took 2.555 seconds
[2017-09-05 11:44:26.864] [29468] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:27.113] [29470] [DEBUG] ./google-drive.rb ["--filter", "new test"]
[2017-09-05 11:44:27.115] [29470] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:27.115] [29470] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:27.433] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:29.552] [29470] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:29.841] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:30.252] [29470] [DEBUG] Execution took 3.14 seconds
[2017-09-05 11:44:30.278] [29477] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:30.495] [29479] [DEBUG] ./google-drive.rb ["--filter", "cache"]
[2017-09-05 11:44:30.496] [29479] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:30.496] [29479] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:31.119] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:32.199] [29479] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:32.828] [29479] [DEBUG] Execution took 2.333 seconds
[2017-09-05 11:44:32.859] [29481] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:33.100] [29483] [DEBUG] ./google-drive.rb ["--filter", "cache"]
[2017-09-05 11:44:33.102] [29483] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:33.102] [29483] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:33.482] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:34.926] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:34.939] [29483] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:35.564] [29483] [DEBUG] Execution took 2.463 seconds
[2017-09-05 11:44:35.598] [29491] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:35.851] [29493] [DEBUG] ./google-drive.rb ["--filter", "cache"]
[2017-09-05 11:44:35.852] [29493] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:35.852] [29493] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:36.451] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:37.560] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:37.661] [29493] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:38.229] [29493] [DEBUG] Execution took 2.378 seconds
[2017-09-05 11:44:38.256] [29496] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:38.505] [29498] [DEBUG] ./google-drive.rb ["--filter", "cache"]
[2017-09-05 11:44:38.506] [29498] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:38.507] [29498] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:38.826] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:39.902] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:40.260] [29498] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:40.890] [29498] [DEBUG] Execution took 2.385 seconds
[2017-09-05 11:44:40.916] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:40.923] [29504] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:41.149] [29506] [DEBUG] ./google-drive.rb ["--filter", "cache"]
[2017-09-05 11:44:41.150] [29506] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:41.151] [29506] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:42.163] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:43.037] [29506] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:43.400] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:43.739] [29506] [DEBUG] Execution took 2.59 seconds
[2017-09-05 11:44:43.771] [29513] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:44.041] [29515] [DEBUG] ./google-drive.rb ["--filter", "cache"]
[2017-09-05 11:44:44.042] [29515] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:44.042] [29515] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:44.570] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:45.669] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:46.093] [29515] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:46.792] [29515] [DEBUG] Execution took 2.751 seconds
[2017-09-05 11:44:46.814] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:46.834] [29519] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:47.075] [29521] [DEBUG] ./google-drive.rb ["--filter", "cache"]
[2017-09-05 11:44:47.076] [29521] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 11:44:47.077] [29521] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 11:44:47.955] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:49.136] [29521] [DEBUG] Loaded 149344 items from cache (created 24 minutes ago)
[2017-09-05 11:44:50.050] [29521] [DEBUG] Execution took 2.975 seconds
[2017-09-05 11:44:50.099] [29531] [DEBUG] Skip background cache update: Another process is already running
[2017-09-05 11:44:50.576] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:51.759] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:52.901] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:54.077] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:55.315] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:56.546] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:58.192] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:44:59.784] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:01.048] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:02.374] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:03.565] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:04.755] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:06.137] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:07.845] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:09.156] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:10.358] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:11.791] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:12.958] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:14.239] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:15.298] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:16.518] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:17.765] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:18.899] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:20.057] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:21.212] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:22.286] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:23.496] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:24.585] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:25.750] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:27.013] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:28.015] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:29.084] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:30.162] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:32.396] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:33.489] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:34.515] [29426] [INFO] Got 460 items from Google Drive
[2017-09-05 11:45:35.649] [29426] [INFO] Got 460 items from Google Drive

Answer 3 · 2017-09-05T19:11:32.000Z

I see, and I was misinterpreting the log, the “Got 460 items…” is not the total amount, because Google does paged responses, so each set of 460 items gets accumulated into, I guess, ~150,000 items.

But this is a bit puzzling, because the workflow only request Google Document types (docs, sheets, forms, and slides) plus PDF files, though also folders.

At the top of the script you can find an array with the requested mime types (called MIME_TYPE_ICONS) and you can comment out e.g. PDFs, but I am thinking that perhaps it’s folders being the majority of these ~150,000 items?

It would also be possible to exclude folders, but then the workflow would be unable to show path information. For this, search for mimeType='application/vnd.google-apps.folder' or and simply remove that part.

Can you experiment with these things yourself?

Answer 4 · 2017-09-05T19:18:57.000Z

Yes. That did the trick. Removing the folder search reduced the cache size.

Here is the latest log:

[2017-09-05 12:16:57.058] [31573] [DEBUG] ./google-drive.rb ["--filter", "mike"]
[2017-09-05 12:16:57.059] [31573] [DEBUG] Loaded 12 releases from cache (created 5 days ago)
[2017-09-05 12:16:57.059] [31573] [DEBUG] Latest online version is v2.0, we are running 2.0
[2017-09-05 12:16:57.327] [31573] [DEBUG] Loaded 21280 items from cache (created 2 minutes ago)
[2017-09-05 12:16:57.437] [31573] [DEBUG] Execution took 0.38 seconds

Answer 5 · 2017-09-05T19:29:28.000Z

Thanks for the update, I’ll leave the issue open until a potential setting is added to disable folders from the default fetch.

Out of curiosity, is the backup keeping old versions, i.e. do you have multiple versions of the same PDFs etc., and what software are you using for the backup? (i.e. is there a “standard” way to backup to Google Drive that the workflow could perhaps be made “compatible” with).

I still find it a bit crazy that you are left with 21280 items, that should only be either Google docs, sheets, forms, slides, or PDF files.

Answer 6 · 2017-09-05T19:37:37.000Z

Yes, it is creating multiple versions. I think I will change the backups to zip files which should fix the issue.

Answer 7 · 2017-09-14T18:19:19.000Z

In 2.1 it is now possible to set custom_query to restrict search to a folder.

See the Configuration section of the README for instructions.

2.1 is currently marked a pre-release so you will need to manually update.