rbtxt
is a tool that allows you to check the robots.txt of each site from the CLI.
- Check existence of robots.txt
- Display the contents of robots.txt
- Save the content of robots.txt to local file
This program requires Go to be installed. If not, please install Go 1.21.0 or higher from the official website.
To install the latest version of rbtxt, execute the following command.
go install github.com/aqyuki/rbtxt@latest
You can install any version of rbtxt by specifying the tag.
go install github.com/aqyuki/rbtxt@<tag>
All contributions are welcome. All contributors may participate in this project.
- Bug report
- Bug fix
- Propose new functions and functional improvements
This program is released under the MIT License.