This is a simple web crawler that takes a URI and returns a list of all internal URIs along with and associated asset list for each URI. Usage Crawler.new(uri, options = {exclude: '', allow_redirections: :none, output: false})