A DSL aimed at making writing web scrapers/crawlers a breeze
init {
get "https://google.com", { "Authorization": "Token asdflkjasdflj" }
headers = {
"Accept": "application/json"
}
@anchors get "https://google.com", headers
urls = ['https://google.com', 'https://chura.co.ke']
@anchors get urls, headers
}
default {
print content
}
anchors {
print headers, headers['Content-Length']
print headers['Content-Type']
anchors = jq 'a#base'
anchors.loop(() {
@anchor get it~href
print it~id
})
closure = (item, index ) {
@anchor get item~href
}
anchors.loop closure
}
- Make "post" expressions keyword expressions
- Handle quering JSON responses
- Handle GraphQL queries
- Implement Resolver
- Will check that there are no cycles
- Check return statement is used properly
- Check for missing but referenced tagged closures
- Design how to handle terminating conditions for tagged closures that might lead into an infinite loop