operatorequals/httpimport

can't access private repo from github

Opened this issue · 4 comments

ggg5269 commented

Hi, I ran the code:

import httpimport
profile = """
[github]
headers:
  Authorization: token ghp_WjewI0mwxxxxxxxxxxxxxxxxez9"""

httpimport.set_profile(profile)

with httpimport.github_repo('myid', 'myrepo', profile='github'):
    import mypkg

but it gave ModuleNotFoundError.
could you please test with this function, and maybe make more instruction for us?

janalysis commented

This is working for me on our private repos. However, I have noticed there is a 5-10 minute delay before the import command picks up changes to the repository on GitHub.com, even though I can see the change right away at https://raw.githubusercontent.com/...

Would be nice if we could figure out what is causing this delay and mitigate it, since it slows down importing from one repo to another when you are making updates in the source repo.

operatorequals commented

Hello all!
I have the feeling that this is behaviour of Github itself, treating Python urllib User-Agent differently. This could make sense, as such User-Agents are sent by scripts and automated things,
and Github does not want to spend too much processing power to them (serving cached content more).

Yet, this is my speculation, as I can't find anything in the code, that would create such delays.

Can you please test the case of setting a different Usr-Agent header here:
https://github.com/operatorequals/httpimport/blob/master/httpimport.py#L105 and report back?

janalysis commented

I have added the following to my profile headers:
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:124.0) Gecko/20100101 Firefox/124.0

The import then picks up the changes within 1-2 minutes if I start a new python kernel. This is an improvement!

However, due to module caching within an existing python kernel it will never updated. Is there any reload function for httpimport like the reload function included with importlib? This is particular useful when working in development tools like Spyder where you are re-running code in the same kernel, vs calling a new python process every time the code runs.

  • 👍1
operatorequals commented

I have added the following to my profile headers:
User-Agent: Mozilla/5.0 (X11; Linux i686; rv:124.0) Gecko/20100101 Firefox/124.0

The import then picks up the changes within 1-2 minutes if I start a new python kernel. This is an improvement!

Glad it worked 🙃

However, due to module caching within an existing python kernel it will never updated. Is there any reload function for httpimport like the reload function included with importlib? This is particular useful when working in development tools like Spyder where you are re-running code in the same kernel, vs calling a new python process every time the code runs.

I have the feeling that using importlib.reload should work. I remember handling something like that at some point.