/ajax-crawl-simulator

A simulator for the ajax crawling protocol

Primary LanguageJavaOtherNOASSERTION

Introduction

This tool is aimed at helping web developers to make ajax crawlable applications.

It is intended to be a simulator of crawlers that rely on the [ajax crawling] (https://developers.google.com/webmasters/ajax-crawling/) protocol.

It comes with a web application that enables to launch crawl simulations and explore the result.

By default it stores crawl simulations in memory, but it is able to use a Mongo DB database as backend to support data persistence.

Build

This tool is written in Java 8. It build relies on Maven 3. To compile it and get a standalone executable jar, just launch the following command at the repository root:

mvn package

The executable jar is web-launcher/target/web-launcher-1.0-SNAPSHOT-jar-with-dependencies.jar

Run

Once the project is built, it can be simply run thanks to the following command.

java -jar web-launcher/target/web-launcher-1.0-SNAPSHOT-jar-with-dependencies.jar

Use the -h option to discover the launcher options.

java -jar web-launcher/target/web-launcher-1.0-SNAPSHOT-jar-with-dependencies.jar -h

Once the tool is launched, open this URL with your browser:

http://localhost:8080/

This URL changes if the HTTP port is not the default one or if the app does not run in the local system.

Technologies

Here is a non exhaustive list of the technologies used in this tool.

Crawl

  • Apache HTTP Client A powerful HTTP client
  • jsoup A Java library that is able to parse and manipulate real-word HTML documents

Storage

  • MongoDB A NoSQL document-oriented database
  • Jongo A high-level MongoDB driver

Web Server

  • Fluent HTTP A simple and yet powerful web server
  • Guice A dependency injection library
  • Jackson A serialization/deserialization library that supports object mapping and many formats such as JSON

Web Client

  • AngularJS A very powerful Javascript framework
  • Bootstrap 3 A very famous web presentation framework

Testing

  • JUnit A very famous Java testing tool
  • Hamcrest An library that provides matchers to write concise and readable tests
  • Mockito A famous Java mock factory
  • Fongo A fake Mongo Java implementation that runs in memory
  • RestAssured A web service testing tool
  • Wiremock A web service mocking tool
  • Awaitility A tool that makes asynchronous testing simple