This project aims to implement the robots.txt parser and matcher in Java. It is based on the C++ implementation.
The Robots Exclusion Protocol (REP) is a standard that enables website owners to control which URLs may be accessed by automated clients (i.e. crawlers) through a simple text file with a specific syntax. It's one of the basic building blocks of the internet as we know it and what allows search engines to operate.
Because the REP was only a de-facto standard for the past 25 years, different implementers implement parsing of robots.txt slightly differently, leading to confusion. This project aims to fix that by releasing the parser that Google uses.
The library is a Java port of C++ parser and matcher which is a slightly modified production code used by Googlebot, Google's crawler. The library is released open-source to help developers build tools that better reflect Google's robots.txt parsing and matching.
For webmasters, we included a runnable class RobotsParserApp which is a small
application that allows testing a single URL and several user-agents against a
robots.txt.
You need Maven to build this project. Download and install it from the official website.
You can also install it like this if your Linux supports it:
$ sudo apt-get install maven
Standard maven commands work here.
$ mvn install
Or if you want a build from scratch:
$ mvn clean install
Alternatively, you can compile the entire project into a single JAR using the following command:
$ mvn clean compile assembly:single
You can find the result in target directory.
Following commands will run an application that parses given robots.txt file
and print a matching verdict: ALLOWED or DISALLOWED (exit codes are 0
and 1 respectively).
You should provide a target URL using -u (--url) flag. At least one agent
must be specified using -a (--agent) flag (verdict DISALLOWED is printed
iff none of the user-agents are allowed to crawl given URL).
When flag -f (--file) is omitted, robots.txt contents are expected to be
received via standard input:
$ mvn exec:java -Dexec.mainClass=com.google.search.robotstxt.RobotsParserApp -Dexec.args="--agent FooBot --url http://foo.com/bar"
If you want the application to read an existing robots.txt file, use flag -f
(--file):
$ mvn exec:java -Dexec.mainClass=com.google.search.robotstxt.RobotsParserApp -Dexec.args="--agent FooBot --url http://foo.com/bar --file path/to/robots.txt"
If you have built the project into JAR, you can run it from there (reading robots.txt from standard input):
$ java -jar target/robotstxt-java-1.0-SNAPSHOT-jar-with-dependencies.jar --agent FooBot --url http://foo.com/bar
Or (reading from file):
$ java -jar target/robotstxt-java-1.0-SNAPSHOT-jar-with-dependencies.jar --agent FooBot --url http://foo.com/bar --file path/to/robots.txt
Parsing of robots.txt files themselves is done exactly as in the production
version of Googlebot, including how percent codes and unicode characters in
patterns are handled. The user must ensure however that the URI passed to the
Matcher methods, or to the --url parameter of the application, follows the
format specified by RFC3986, since this library will not perform full
normalization of those URI parameters. Only if the URI is in this format, the
matching will be done according to the REP specification.
The robots.txt parser and matcher Java library is licensed under the terms of the Apache license. See LICENSE for more information.
Every file containing source code must include copyright and license information. This includes any JS/CSS files that you might be serving out to browsers. (This is to help well-intentioned people avoid accidental copying that doesn't comply with the license.)
Apache header:
Copyright 2020 Google LLC
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
It can be done easily by using the addlicense tool.
Install it:
$ go get -u github.com/google/addlicense
Use it like this to make sure all files have the licence:
$ ~/go/bin/addlicense -c "Google LLC" -l apache .