Install/run puppeteer module on beamup?
Closed this issue · 7 comments
I managed to successfully deploy stremio add-on that is relying on puppeteer library on beamup. Unfortunately I got errors for missing chrome/puppeteer installation I don't know how to tackle.
Is it possible to install and run puppeteer module in beamup servers? Any guidance would be much appreciated!
It's not wise to run puppeteer on any free web hosting.
That's good to know.
The only reason to do it is to login and to store credential. I tried with other libraries but they didn't work.
In addition to that I have minimized the traffic and memory/cpu usage by disabling images, and opening a single tab and single page only. So it's very optimal usage in terms of memory/cpu usage and traffic.
I agree it's not as optimal as other libraries but definitely not heavy at all.
Any advices for alternative libraries with support for cookies and login capabilities?
@svetlozardraganov u can use axios + axios-cookiejar-support + tough-cookie for cookies and login capabilities. and if u got stuck, just send me the repo and i will give u a hand.
Thanks @dexter21767-dev - I have been working on different ways to login into the website for the past 2 days. I have tried number of different ways to do it with Axios but always failed.
I managed to successfully login using:
- Python with Session library: https://github.com/svetlozardraganov/stremio-hello-world/blob/main/login_python.py
- NodeJs with SuperAgent library: https://github.com/svetlozardraganov/stremio-hello-world/blob/main/login_nodejs_superagent.js
And after I read your comment I managed to login via NodeJs with Axios library as well: https://github.com/svetlozardraganov/stremio-hello-world/blob/main/login_nodejs_axios.js
The problem now is that I cannot find a NodeJs library with good Xpath support for extracting webpage-elements.
I tried with various libraries including Cherio, xmldom but nothing gives me acceptable results. Either the element is not found or it gives validation errors of the HTML as:
- attribute missed value
- unclosed xml attribute
- entity not found
I got the same errors when loading webpage source into online XML validators, example:
https://codebeautify.org/xmlvalidator/y24897c0a
Xpath selectors in Puppeteers always worked as a charm. Can you recommend me a reliable NodeJs library for extracting web-elements via xpath selectors?
@svetlozardraganov i don't understand why you're using an XML validator for validating HTML.
I agree that Cherio isn't really intuitive to work with. i have used fast-html-parser and node-html-parser in the past (they are limited in how u can query elements iirc) but still usable.
also note that the website you're trying to scrape has a rate limit. and u need some time between requests.
Thanks @dexter21767-dev
I used the validators because the Cherio library didn't properly extract web-elements using xpath-selectors.
In the past I have tried using css-selectors but they weren't that reliable as the xpath-ones.
Nevertheless I decided to use the Chrerio library with css selectors and so far everything works great.
Can you provide more details about the website rate-limit? Where I can get this information and how should I handle it in my code? Is it as simple as using setTimeout() or I should use something else?
Thanks in advance.
@svetlozardraganov sorry, i thought i replied. iirc setTimeout should be enough, u should experiment to find the correct values to use.