Enhancing the WebsiteAgent: Browser Emulation
irfancharania opened this issue · 22 comments
(or a new agent) that will render the page to make for easier DOM parsing.
There are a number of sites like https://www.google.com that have ugly source, but a nicer looking DOM. We could look at using something like phantom.js to pull out the rendered document.
Could be implemented as an option in the existing WebsiteAgent.
Since we already have therubyracer for the JavaScriptAgent, maybe something like https://github.com/tmpvar/jsdom would be a good solution?
The title of this issue is not very specific. You should at least include the word JavaScript in it.
Or maybe "browser emulation".
I don't know much about jsdom, but capybara+capybara-webkit might also be a good option.
There will be so much amount of work needed, but it could be worthwhile?
- Running a given piece of JavaScript code onload/document ready
- Providing JavaScript API to emit an event from within the code
- Storing and extracting cookies/localStrorage in Agent's memory
- Memory management (maybe the headless browser should be a separate process)
I know there's debate about future use of Heroku, but from quick googling, it looks like capybara-webkit won't run on Heroku whereas Phantomjs does. Something to consider...
Currently, I'm running a separate instance of PhantomJS on Heroku which returns a rendered page to the calling Huginn agent.
I wonder if there are any web services available that allow you to run a script of your choice in the context of a website of your choice, and get the output as an API result.
PhantomJs Cloud looks interesting. It has a free plan that should be enough for personal use.
If someone gets this working well with Huginn it'd absolutely be worth documenting in the wiki.
I added an entry in the wiki for using PhantomJs Cloud with Website Agent.
Alternative to building an agent for it.
@irfancharania very cool, thanks you!
@dsander: The workflow actually works quite well.
It would be nice to have the ability to set API key and request options in a dedicated agent, but I wasn't sure how a separate agent + interactions with other agents would work with all the options PhantomJs Cloud has: GET/POST, output Html/Json/PDF/Image, etc
@irfancharania You should be able to use credentials and the credential liquid tag to extract the API key from the agent options like this: "https://PhantomJsCloud.com/api/browser/v2/{% credential credential-name %}/?request={{url}}" A separate agent sounds interesting, if the PhantomJs Cloud are not dynamic using FormConfigurable would be a nice option to configure the Agent.
@dsander I forgot about credentials. Thanks for pointing that out. I'll update the wiki entry
@dsander I decided to try creating a PhantomJs Cloud Agent with FormConfigurable. My knowledge of ruby/rails is very limited, so I need some help.
So far I've got it creating an event with the correct url.
What I can't figure out is:
- How to present a list of credentials for user to select their API key from
- Why receive(incoming_events) doesn't seem to fire
FYI I just published a daemon written in Node that makes this dead-simple: https://github.com/strugee/fulldom-server
You can run fulldom on any old box and point WebsiteAgent at a fulldom URL instead of the actual URL, and fulldom will load up the page, run JS and then serve you the result. See the README for details.
Nice!
Mind putting a usage example on the wiki?
Sure. Though the README has it covered quite well.
@cantino just wrote this up on the wiki: https://github.com/cantino/huginn/wiki/Browser-Emulation-using-fulldom-server
It doesn't work yet due to #1735, though. Soon!
The PhantomJs Cloud Agent has been working well for me for quite some time.
Think we could merge that in?
Yes! Sorry I lost track of this.
It got merged!