opensanctions/yente

Allow httpx to use proxies

Closed this issue · 4 comments

hotdog809 commented

Dear team,

Related to #84, since the switch to httpx for data retrieval we cannot use our proxy anymore and therefore cannot update Yente. Could you please add the option to specify a proxy, as in: https://www.python-httpx.org/advanced/proxies/ ?

for example
with httpx.Client(proxy=YENTE_PROXY) as client: ...
where YENTE_PROXY is an env var.

Thank you very much!

hotdog809 commented

My apologies, I seem to be having a different issue. httpx does accept the environmental variables HTTP_PROXY and HTTPS_PROXY as aiohttp does (see: https://www.python-httpx.org/environment_variables/).

I will close this issue.

SimonThordal commented

Thanks for closing this, and please feel free to open a new issue if you're still having trouble with yente.

RMHogervorst commented

Good day,

This might be related. I've found that the the async client does not respect the HTTP_PROXY env var.
I'm running an data update and it fails in 3.8.4 where it worked in 3.7.x

Here is what I did:

  • set the HTTP_PROXY and HTTPS_PROXY env vars
  • run the proces
  • I get vague error messages . HTTPX.connect errors (but no details, quite frustrating!)
  • manually running a httpx.get works. so the proxy is set correctly.

When I added a proxy arg here: https://github.com/opensanctions/yente/blob/main/yente/data/util.py#L134
the update process just works.
Can we pass the proxy env if it exists?

SimonThordal commented

Thank you very much, @RMHogervorst - I've opened up for setting YENTE_HTTP_PROXY to explicitly set a proxy if it isn't automatically found by httpx. PR for it here if you want to check the implementation: #428

Does that look like it will fix it for you?