How to use Proxies while searching?

Question

How to use Proxies while searching?

Closed this issue 4 years ago · 22 comments

Hi,

Thanks for the nice tool. It's really easy to search youtube videos quickly with this tool. But can you please let me know if it supports Proxy for search?

Answer 1 · 2021-03-24T06:28:10.000Z

Thanks a lot. I'm glad you liked this library.

I don't really have an interest in dealing with proxies & is way beyond the scope of searching or getting data from YouTube.
Look forward to any pull requests that you or community offers.

Have great day.

Answer 2 · 2021-04-20T15:32:24.000Z

I can actually try to do that if you (still) want...

Answer 3 · 2021-04-20T16:26:04.000Z

Sure @mytja . I think It will be useful for others too. Thanks In advance.

Answer 4 · 2021-04-20T17:18:33.000Z

Hello!
I am finished. I personally don't have any proxy provider or any IP for testing. Can you please test it?
Currently it works only on VideosSearch.
This is sample request:

videosSearch = VideosSearch('NoCopyrightSounds', limit = 10, language = 'en', region = 'US', proxy = "your proxy IP", proxyType = "https")

proxyType must be http or https

Install package:

pip install git+https://github.com/mytja/youtube-search-python --upgrade

If it doesn't work, may I get the error thrown? For example something like this if possible:

I can't wait for your response.

Answer 5 · 2021-04-20T17:26:45.000Z

@mytja
You can make proxyType an enum class (inside constants.py) :-). always good to have intellisense instead of strings.

Should I assign this issue to you?

Answer 6 · 2021-04-20T17:35:59.000Z

@mytja, Thanks for the fix. Sure I will test and let you know if it works. Thanks.

Answer 7 · 2021-04-20T17:43:05.000Z

Should I assign this issue to you?

You can if you want to.

Another question.
Why don't you have a bunch of User-Agent headers in list, and them randomly select them?

@mytja, Thanks for the fix. Sure I will test and let you know if it works. Thanks.

No problem. Again, let me know if it works.

Answer 8 · 2021-04-20T17:56:19.000Z

Why don't you have a bunch of User-Agent headers in list, and them randomly select them?

This library and its certain features (like webp preview etc.) only work with this specific user agent.

Answer 9 · 2021-04-20T18:00:30.000Z

Hmmm. It's weird
With requests library, proxying works, but with urllib not...

Answer 10 · 2021-04-20T18:01:56.000Z

@mytja,
Looks like the proxy with authentication format is not working.

videosSearch = VideosSearch('NoCopyrightSounds', limit = 10, language = 'en', region = 'US', proxy = "209.127.191.180:9279:axvjnt-dest:bvcb2jhk5", proxyType = "https")

ERROR.

raise InvalidURL("nonnumeric port: '%s'" % host[i+1:])
http.client.InvalidURL: nonnumeric port: 'bvcb2jhk5'

Answer 11 · 2021-04-20T18:02:50.000Z

Yeah. I have to take some time to troubleshoot. Currently, I don't have much time

Answer 12 · 2021-04-20T18:04:01.000Z

Sure...take your time. Currently, I am also not using it. So you can take time.

Answer 13 · 2021-04-21T12:21:30.000Z

@alexmercerind I really don't know. I can't prepare urllib to work with proxying. It always makes an error. Meanwhile, when I use requests, which uses urllib3 works just fine.

So, I have 3 options:

Use httpx in all requests
Create a bug on urllib library bug tracker
Mark this as wontfix

What do you think?

Answer 14 · 2021-04-21T15:27:00.000Z

I think It's better to use always HTTPS instead of marking it as won't fix.
Since Proxy is optional for most of the users we can mention in documentation to use HTTPS instead of HTTP if they ever need it.
Meanwhile, we can raise a bug against urllib3 and see if they can fix it. With the first option, we no need to worry if they fix it or not.

Answer 15 · 2021-04-26T02:47:58.000Z

You can implement a proxy by using socks5. I use PIA and it works great.

Answer 16 · 2021-04-26T16:06:14.000Z

You can implement a proxy by using socks5. I use PIA and it works great.

Hello, @jacksonw765 !
Yes, I thought about this option, but httpx library doesn't support them yet, and so I'd have to add another dependency to the library, and we are trying to keep it without dependencies if possible...
I personally currently cannot proxy anything, not with urllib nor with httpx. I can currently proxy only with requests library, which itself is built on top of urllib3. I can try to implement everything using requests, but that itself is another dependency.

Answer 17 · 2021-04-26T16:35:59.000Z

I have built a working proxying release.
You can install it here: pip install git+https://github.com/mytja/youtube-search-python.
It requires requests library, which you can install using pip install requests.

@alexmercerind I tried all kinds of libraries, and requests was the only one, that worked. Do you think another dependency is too much?? I have created this issue (hoping they will answer me): encode/httpcore#309

Answer 18 · 2021-05-02T12:29:43.000Z

Hi @mytja, thanks for working on this.
I missed commenting, for proxies requests library is one of the standard libraries for python. So I think this dependency shouldn't be a problem. Because I have seen several other python projects use 'requests' library for proxies.
Here is one of the python code repo https://github.com/bstoilov/py3-pinterest which uses requests library.

I will try to use your code and test. I hope this fix will get integrated into the main branch. because In practice we do need proxies to do searches successfully without our IPs getting banned.

Answer 19 · 2021-09-23T16:51:29.000Z

Hey @mytja,

I tried to use your fork, but I got this error:

TypeError: __init__() got an unexpected keyword argument 'proxy'

I installed it using this command:

pip install git+https://github.com/mytja/youtube-search-python --upgrade

and it went to version 1.4.8

Am I doing something wrong?

Answer 20 · 2021-09-23T17:21:55.000Z

Hey @mytja,

I tried to use your fork, but I got this error:
TypeError: __init__() got an unexpected keyword argument 'proxy'
I installed it using this command:
pip install git+https://github.com/mytja/youtube-search-python --upgrade
and it went to version 1.4.8

Am I doing something wrong?

I don't know exact error, and on what line is this happening, but it seems like you are trying to pass proxy parameter to a class. Proxying isn't implemented on this fork. I have moved all proxying stuff to https://github.com/mytja-archive/youtube-search-python-proxy.
I do not recommend this repository tho, since it is quite outdated.
I am not going to implement proxying since there were quite a bit of problems with proxies for me. This is why I added wontfix label.

Answer 21 · 2021-09-23T18:29:12.000Z

Oh ok, I didn't note the "wontfix" label, sorry. Thanks for the reply.

Answer 22 · 2021-09-23T18:31:39.000Z

By the way, I checked the httpx documentation, they say that we could use HTTP_PROXY environment variable to set a simple proxy configuration.

Do you think it could work if I just set this environment variable to use with youtube-search-python? It's just what I would need.

I'm trying it but I'm still not sure how to know whether it's working or not.

Thanks anyway

EDIT: I just tested with netcat and it really doesn't work.