crawler Web Crawler in Java In SpiderTest.java, the input to the function is the url which we want to crawl. In Spider.java, MAX_PAGES_TO_SEARCH would determine the maximum number of pages we want to crawl. The response of each web-page is stored in files.