Missing GMail labels

Question

Missing GMail labels

ktf opened this issue 7 years ago · 17 comments

Similarly to #46 I've:

pull: full synchronization (no previous synchronization state)                                                                                    [3/8741]
fetching messages: 123866it [04:10, 414.05it/s]
receiving content:   0%|                                                                             | 3/57916 [00:00<9:22:58,  1.71it/s]Traceback (most r
ecent call last):
  File "/nix/store/5zb6zlyxg2xjz7p3kqr9hahl3vdbzm63-python3.6-gmaileer-v0.3-20171115/bin/.gmi-wrapped", line 9, in <module>
    g.main ()
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 136, in main
    args.func (args)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 307, in pull
    self.full_pull ()
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 530, in full_pull
    updated = self.get_content (message_gids)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 595, in get_content
    self.remote.get_messages (need_content, _got_msg, 'raw')
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 109, in func_wrap
    return func (self, *args, **kwargs)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 282, in get_messages
    batch.execute (http = self.http)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/googleapiclient/http.py", line 1437, in execute
    self._callback(request_id, response, exception)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/remote.py", line 262, in _cb
    cb (resp)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/gmailieer.py", line 593, in _got_msg
    self.local.store (m, db)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 346, in store
    self.update_tags (m, p, db)
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 354, in update_tags
    labels = [self.gmailieer.remote.labels[l] for l in labels]
  File "/nix/store/b0rgiilc0nhszygkgqy4sj47nhrbakzx-python3-3.6.2-env/lib/python3.6/site-packages/lieer/local.py", line 354, in <listcomp>
    labels = [self.gmailieer.remote.labels[l] for l in labels]
KeyError: 'Label_3'

I did not create a label while it was running and I do not have Label_3 as label.

Answer 1 · 2017-11-15T10:53:42.000Z

Just for the record: Labels have internal id like that, they are not necessarily the names you gave them. For example, I have a label ubuntu-devel, but it's internal id is Label_44. Here it's mapping the internal ids to the names AFAICT.

Answer 2 · 2017-11-15T11:07:34.000Z

Julian Andres Klode writes on november 15, 2017 11:53:

Just for the record: Labels have internal id like that, they are not necessarily the names you gave them. For example, I have a label ubuntu-devel, but it's internal id is Label_44. Here it's mapping the internal ids to the names AFAICT.

That's correct; we fetch the label list at start-up with all the mappings. You can print them with the '-t' option for `pull` I believe.. Does the relevant label show there? If you know what it is. Does re-starting the pull fix the issue? I don't know why not all the labels are pulled, could be an API issue.

Answer 3 · 2017-11-15T15:08:41.000Z

gmi pull -t shows labels and the third one in the output order is FromMailingList, however I see no mapping output. Restarting the pull did not help. I do have somewhat strange labels though, like R&D or `Something/something - (else)" could that be a problem?

Answer 4 · 2017-11-16T10:10:28.000Z

Giulio Eulisse writes on november 15, 2017 16:18:

`gmi pull -t` shows labels and the third one in the output order is `FromMailingList`, however I see no mapping output. Restarting the pull did not help. I do have somewhat strange labels though, like `R&D` or `Something/something - (else)" could that be a problem?

I don't think so. I've added a change on master which also prints the gmail key. Can you see if you also get the problematic Label_3? - gaute

Answer 5 · 2017-11-16T10:28:45.000Z

I don't get any Label_3:

+ gmi pull -t
+ grep Label_3
CWG4                           Label_3939416323915462435
OpenSource Mailing lists       Label_3194149110520196790
CMS                            Label_394156353969411168
Nix                            Label_3817195439730139360
Simulation & Generators        Label_3946793891174421901
Infrastructure                 Label_3990267443586131481
WP3 - Tools                    Label_3023913578609897416
Daily Build Reports            Label_330431641839543684
Banking                        Label_3461285147915119731

Answer 6 · 2017-11-16T10:31:33.000Z

Giulio Eulisse writes on november 16, 2017 11:28:

I don't get any `Label_3`: ``` + gmi pull -t + grep Label_3 CWG4 Label_3939416323915462435 OpenSource Mailing lists Label_3194149110520196790 CMS Label_394156353969411168 Nix Label_3817195439730139360 Simulation & Generators Label_3946793891174421901 Infrastructure Label_3990267443586131481 WP3 - Tools Label_3023913578609897416 Daily Build Reports Label_330431641839543684 Banking Label_3461285147915119731 ```

Do you have the chance to go here: https://developers.google.com/gmail/api/v1/reference/users/labels/list Enter your email in the example API on the right and post the result somewhere? I realize this is private, what I'm looking for is anything else than just labels.. According to the docs there shouldn't be any nextPage fields, but maybe you have enough labels..

Answer 7 · 2017-11-16T10:46:11.000Z

Ciao,
I just did and AFAICT, there is no pagination entry in the output (nor `Label_3). I will try reducing the number of labels. How do you get Label_3 in the first place? Is it attached to the message?

Answer 8 · 2017-11-16T12:18:19.000Z

Giulio Eulisse writes on november 16, 2017 11:46:

Ciao, I just did and AFAICT, there is no pagination entry in the output (nor `Label_3). I will try reducing the number of labels. How do you get Label_3 in the first place? Is it attached to the message?

Yes, it comes from message.get().

Answer 9 · 2017-11-16T13:25:59.000Z

Did you identify which message has the Label_3? You could maybe add a print google-id so that you get some more output before it crashes (in Local.update_tags). Then you could try and fetch that message from the Gmail API explorer and see if the label is actually there as well. Can you test out labels.get() on Label_3 in the API explorer? Maybe it shows up if you try to fetch it directly..

Answer 10 · 2017-11-16T13:46:46.000Z

For the moment I just patched it to add an unknown label in case the Remote label is not found. That should allow me to identify the broken messages as well. I want to get to the end and see how much I like notmuch before I spend time debugging this.

Answer 11 · 2017-11-16T13:49:05.000Z

Giulio Eulisse writes on november 16, 2017 14:46:

For the moment I just patched it to add an `unknown` label in case the Remote label is not found. That should allow me to identify the broken messages as well. I want to get to the end and see how much I like notmuch before I spend time debugging this.

Good plan. You might want to add unknown to ignored labels so that its not synced back up to your account if you do push or sync. Or just avoid pushing up before you've figured out the issue.

…

-g

Answer 12 · 2017-11-16T13:55:22.000Z

Thanks for mentioning the ignored label stuff. I was actually wondering about that. I will add it. If I understand what's going on I'll let you know right away.

Answer 13 · 2017-11-16T13:56:10.000Z

Giulio Eulisse writes on november 16, 2017 14:55:

Thanks for mentioning the ignored label stuff. I was actually wondering about that. I will add it. If I understand what's going on I'll let you know right away.

Thanks, appreciate the effort.

Answer 14 · 2017-11-16T14:11:32.000Z

Thank you. If this works as I expect it to work and scales to my mailbox size, you made an happy man.. ;-)

Answer 15 · 2017-11-16T21:43:14.000Z

With #49, I am able to go past my issue and I've a full sync done. Notice that some of the messages which are reported to have "Label_3" do not actually have any label associated. I wonder if this is simply a problem with some old labelling schema on google side..

Answer 16 · 2017-11-16T22:27:51.000Z

Weird; if you put in the messages in https://developers.google.com/gmail/api/v1/reference/users/messages/get do they show Label_3?

Answer 17 · 2017-11-19T11:26:49.000Z

Fixed in #51:

If you encounter this error run:

$ gmi set --drop-non-existing-labels

and re-try the pull.

This will drop any labels that cannot be found remotely. If the message tags are modified locally (through notmuch) the label will be removed on the remote side on the next push/sync.