args-out-of-range

Question

args-out-of-range

YiLiu6240 opened this issue 8 years ago · 24 comments

When I try to fetch the bibtex info from google scholar, it returns the following in *Messages*

gscholar-bibtex--write-bibtex-to-database-impl: Args out of range: [], 0

The following is the debug infomation from toggle-debug-on-error:

Debugger entered--Lisp error: (args-out-of-range [] 0)
  gscholar-bibtex-retrieve-and-show-bibtex()
  gscholar-bibtex--write-bibtex-to-database-impl(t)
  gscholar-bibtex-append-bibtex-to-database()
  funcall-interactively(gscholar-bibtex-append-bibtex-to-database)
  call-interactively(gscholar-bibtex-append-bibtex-to-database nil nil)
  command-execute(gscholar-bibtex-append-bibtex-to-database)

I get this problem persistently on OS X. However in my Linux computer everything works fine.

Answer 1 · 2016-09-23T14:35:56.000Z

Can you provide the steps to reproduce the problem using emacs -Q? Also, is there any difference between your Emacs configs on macOS and Linux?

Answer 2 · 2016-09-23T14:45:22.000Z

And can you check whether this is the same as #10? The cookie might be causing the problem.

Answer 3 · 2016-09-23T17:40:25.000Z

I am a recent vim convert so I can't use plain emacs -Q without evil bindings, etc.
For #10 I can't find a cookies file/directory in my ~/.emacs.d.

There should be no difference for my configs between my macOS and Linux machines. It seems to me to be a platform specific problem.

Answer 4 · 2016-09-23T17:58:57.000Z

OK. Using emacs -Q is very simple. Follow these steps:

emacs -Q from the terminal emulator
M-x package-initialize
M-x gscholar-bibtex

Can you still reproduce the problem using these steps?

Answer 5 · 2016-09-23T19:10:19.000Z

Thank you.
The error message when I try to search something like foobar in Google Scholar is
Symbol's function definition is void: url-cookie-handle-set-cookie

Answer 6 · 2016-09-23T19:49:40.000Z

Oh, sorry. That seems like a different problem. I've pushed a simple fix 70075de. For now, can you use the following steps instead?

emacs -Q from the terminal emulator
M-x package-initialize
M-: (require 'url)
M-x gscholar-bibtex

Thanks!

Answer 7 · 2016-09-23T20:16:00.000Z

I toggled M-x toggle-debug-on-error also, and the error message is the same:

Debugger entered--Lisp error: (args-out-of-range [] 0)
  gscholar-bibtex-retrieve-and-show-bibtex()
  gscholar-bibtex--write-bibtex-to-database-impl(t)
  gscholar-bibtex-append-bibtex-to-database()
  funcall-interactively(gscholar-bibtex-append-bibtex-to-database)
  call-interactively(gscholar-bibtex-append-bibtex-to-database nil nil)
  command-execute(gscholar-bibtex-append-bibtex-to-database)

I also tested it in Linux, there is no problem.

Answer 8 · 2016-09-24T18:37:04.000Z

Sorry. I'm unable to help since I can't reproduce the problem. Could you please provide some information about your system and Emacs version so that someone else who see this issue might be able to help?

If you know some basic ELisp, you can try to debug the gscholar-bibtex-google-scholar-search-results. It would be easy if you use Edebug: type C-u C-M-x on the function definition and then run gscholar-bibtex. It will stop at the entry of the function. Then you can type n to step through every sexp inside the function.

Answer 9 · 2016-09-25T10:07:17.000Z

OS infomation:

OS X El Capitan version 10.11.6

emacs-version:

GNU Emacs 25.1.1 (x86_64-apple-darwin15.6.0, NS appkit-1404.47 Version 10.11.6 (Build 15G31)) of 2016-09-19

I put my debug result here:
https://gist.github.com/YiLiu6240/6954da000e63b12934d76202daee2a20

What I did is use Edebug on gscholar-bibtex-google-scholar-search-results,
then use gscholar-bibtex to search "the lemon market",
and try to append to my database by pressing A.

It seems to me that gscholar-bibtex-google-scholar-search-results finishes the
debug process. Please let me know if you need the debug information on other
functions.

Answer 10 · 2016-09-26T19:39:33.000Z

Thanks @YiLiu6240 ! It seems to me that the cookie we set in the code doesn't have any effect. So still, it is a problem of the cookie.

Since you're saying that you don't have the ~/.emacs.d/url/cookies file, have you tried the two methods metioned in #10 ?

In terminal:

curl -v -A "Mozilla/5.0" --cookie "GSP=ID=87969bbbc5530bab:CF=4" "http://scholar.google.com/scholar?q=test"

and in Emacs:

(let* ((url-request-method "GET")
    (url-request-extra-headers
     `(("User-Agent" . "Mozilla/5.0")
       ("Cookie" . "GSP=ID=87969bbbc5530bab:CF=4"))))
  (url-retrieve "https://scholar.google.com/scholar?q=test"
        (lambda (status) (switch-to-buffer (current-buffer)))))

Are the results the same?

Answer 11 · 2016-09-26T19:41:33.000Z

In case you don't know, to see the result in Emacs, paste the code into a buffer in emacs-lisp-mode, move the cursor to the end of the s-expression and type C-x C-e to run the code.

Answer 12 · 2016-09-27T09:35:59.000Z

The results are here https://gist.github.com/YiLiu6240/6954da000e63b12934d76202daee2a20
I recorded my steps in that gist page as well.

The Set-Cookie lines are missing in the output of my usual emacs session, but they are present in the plain emacs output and terminal output. The "Import into BibTeX" strings are also missing in emacs output, but present in plain emacs output and terminal output.

However I don't know whether these differences have anything to do with this issue, since I also cannot use gscholar to fetch from Google Scholar in plain emacs.

Answer 13 · 2016-09-27T19:39:44.000Z

If "Import into BibTeX" appears in the output, then gscholar-bibtex should work fine.

Based on the current information, we can see that

In your usual Emacs, neither directly setting the cookie nor using url-cookie-handle-set-cookie could set the cookie properly
In your plain Emacs, directly setting the cookie works (based on your last response), but using url-cookie-handle-set-cookie doesn't, which is our current implementation.
curl does work, so there is no problem to get the BibTeX entries from your system.

So a few more questions, can you paste the values of url-cookie-storage, url-cookie-secure-storage and url-cookie-file if it is good for you? Since cookies may have some privacy issues, you can also choose not to share them and take a look at these variables yourself to see if there is anything wrong. And thanks for helping debugging the problem!

Answer 14 · 2016-09-28T09:59:49.000Z

Thank you. I checked these variables in my linux computer and my mac computer.
Those variables were nil before gscholar-bibtex was initialized.

`url-cookie-file`

url-cookie-file is set to be "~/.emacs.d/.cache/url/cookies" ("~" refers the actual home directory locations in both computers). I was wrong about not having a cookies file there, the cookie files both exist and they do have cookie information.

In plain emacs, url-cookie-file is set to be "~/.emacs.d/url/cookies". I am using spacemacs and it resets url-cookie-file to the location above when it inits the url package with spacemacs-base/init-url (https://github.com/syl20bnr/spacemacs/blob/master/layers/+distribution/spacemacs-base/packages.el). But I don't think it is relevant.

`url-cookie-storage`

url-cookie-storage has the following information regarding "scholar.google.com",
where "{0}" are strings of 10 digits and "{1}" are strings of 16 chars (I don't know whether they are safe to show here):

linux:

(("scholar.google.com"
[url-cookie "GSP" "CF=4:LM={0}:S={1}" "23-Sep-2018 20:07:56.00 GMT" "/" "scholar.google.com" nil]))

mac:

(("scholar.google.com"
  [url-cookie "GSP" "LM={0}:S={1}" "25-Sep-2018 09:32:47.00 GMT" "/" "scholar.google.com" nil]))

In plain emacs (in mac, I did not check on linux), the code structure is the same but the actual strings are different, and expire time is also reset.

`url-cookie-secure-storage`

url-cookie-secure-storage is nil in mac and contains no information related to "scholar.google.com".

Answer 15 · 2016-09-28T17:35:13.000Z

OK. Then I think it is safe to say the problem is caused by the cookie. In your linux, there is CF=4 in the cookie while no CF=4 can be found in the cookie on your mac. Try to remove the cookie file, and run gscholar-bibtex again.

Answer 16 · 2016-09-29T09:33:10.000Z

I remove the cookie file but it doesn't help.

When gscholar starts the url-cookie-storage is updated from nil to the format in my last post (there is code for ".google.com" as well, which also exists in the code value of my last post, but I didn't know it was associated with gscholar so I didn't include that bit):

(("scholar.google.com"
  [url-cookie "GSP" "LM={0}:S={1}" "25-Sep-2018 09:32:47.00 GMT" "/" "scholar.google.com" nil])
  (".google.com"
  [url-cookie "HttpOnly" nil "31-Mar-2017 09:14:27.00 GMT" "/" ".google.com" nil]
  [url-cookie "NID" "87={3}" "31-Mar-2017 09:14:27.00 GMT" "/" ".google.com" nil]))

It seems to me the file is not the problem, because after I deleted the cookie file and initialize gscholar, the file was not recreated.

Answer 17 · 2016-09-29T18:48:28.000Z

OK. Then can you do the following the see whether the content of url-cookie-storage changes?

(let* ((url-request-method "GET")
       (random-id (format "%016x" (random (expt 16 16))))
       (expiration (format-time-string "%a, %d %b %Y %H:%M:%S.00 %Z"
                                       (time-add (current-time)
                                                 (seconds-to-time 3600)) t))
       (my-cookie (mapconcat #'identity
                             (list (format "GSP=ID=%s:CF=4; " random-id)
                                   (format "expires=%s" expiration)
                                   "path=/"
                                   "domain=scholar.google.com")
                             "; "))
       (url-current-object
        (url-generic-parse-url "http://scholar.google.com")))
  (url-cookie-handle-set-cookie my-cookie))

This code basically tries to set the cookie for scholar.google.com.

Then evaluate the url-cookie-storage to see whether the entry of scholar.google.com changes. Try it multiple times and try to set with different cookies (change the "GSP=ID=%s:CF=4; " thing to whatever you like) to see whether our current implementation in gscholar-bibtex can change the cookie in your system. The missing CF=4 in your cookie is the main reason why gscholar-bibtex fails.

Answer 18 · 2016-09-29T19:15:48.000Z

And after deleting the url-cookie-file, do you use (setq url-cookie-storage nil) to clear up the cache? If not, and you don't restart the Emacs, the content still remains the same and may cause some inconsistencies.

So you can also try to first (setq url-cookie-storage nil) and then set the cookie using the previous code to see how the value of url-cookie-storage changes.

Answer 19 · 2016-09-29T20:39:05.000Z

I will test the code above with my mac tomorrow.

Regarding your latest post, I did restart Emacs after deleting url-cookie-file, and the value of url-cookie-storage is generated (maybe by gscholar package) after gscholar initialized rather than loaded from a cookie file. This behavior is the same with my usual emacs session (url-cookie-file points to "~/.emacs.d/.cache/url/cookies") and a plain emacs session (url-cookie-file points to "~/.emacs.d/url/cookies"): there are no cookie files in both locations, and url-cookie-file generates strings like "LM=%s:S=%s" in both sessions.

Answer 20 · 2016-09-30T09:25:10.000Z

The code above can set url-cookie-storage to the following

("scholar.google.com"
  [url-cookie "GSP" "ID=2d1dd67322ec7eb8:CF=4" "Fri, 30 Sep 2016 10:14:05.00 UTC" "/" "scholar.google.com" nil])

And I can confirm that the string format can be changed to whatever format I chose.
But when I invoke gscholar-bibtex, it reverts to

("scholar.google.com"
  [url-cookie "GSP" "LM=1475226995:S=b9ezPxUfl5K64CLO" "30-Sep-2018 09:16:35.00 GMT" "/" "scholar.google.com" nil])

Update:

I tried to test the code in gscholar-bibtex-google-scholar-search-results, and maybe there is something unexpected with the (gscholar-bibtex--url-retrieve-as-string ...) part. The results are posted here https://gist.github.com/YiLiu6240/6954da000e63b12934d76202daee2a20

Answer 21 · 2016-10-03T21:38:17.000Z

Thanks for the debugging efforts so far. So the culprit is the url-retrieve-synchronously, which doesn't set the right cookie. I don't know who is to blame here, either the emacs url lib on macOS or the macOS itself. I don't have macOS myself and since no other users report the same problem, I'm sorry that you may have to debug url-retrieve-synchronously to see how it operate the cookies. If you're interested in Elisp, it may make you learn some Elisp and it would be fun. If you're not into it, leave this issue here and see if others could help.

If you try to debug further, don't hesitate to share your findings here.

Answer 22 · 2016-10-05T08:33:03.000Z

OK, I will see what I can find out.

Thanks, it's a great package!

Answer 23 · 2017-02-26T14:42:30.000Z

Update: no issue now!

I had tried gscholar-bibtex on my mac again, and it worked as expected. Maybe the original error is from some of its dependencies.

Closing this issue. Thanks for the package!

Answer 24 · 2017-02-26T19:04:48.000Z

Good to know it is now working for you. Thanks.