/eleventy-plugin-robotstxt

Generate a robots.txt file for your Eleventy site

Primary LanguageJavaScriptMIT LicenseMIT

eleventy-plugin-robotstxt

Note

This plugin is only compatible with Eleventy versions >= 3.0.0-alpha.6.

Automatically generate a robots.txt file for your Eleventy site using front matter.

Getting Started

Install the plugin in your project using your preferred package manager:

npm install --save-dev eleventy-plugin-robotstxt

And update your Eleventy config to import and use the plugin:

const EleventyPluginRobotsTxt = require("eleventy-plugin-robotstxt");

module.exports = (eleventyConfig) => {
  /** @type {import("eleventy-plugin-robotstxt/typedefs.js").EleventyPluginRobotsTxtOptions} */
  const eleventyPluginRobotsTxtOptions = {};
  eleventyConfig.addPlugin(
    EleventyPluginRobotsTxt,
    eleventyPluginRobotsTxtOptions,
  );
};

See the examples for how you might configure the plugin for different use cases.

API

The following plugin options are available for use in your .eleventy.js configuration:

Option Type Description Example
sitemapURL string | undefined (Optional) The absolute location of a sitemap for this site. The sitemap URL must be a fully-qualified URL; Google doesn't assume or check http/https/www.non-www alternates. Sitemaps are a good way to indicate which content Google should crawl, as opposed to which content it can or cannot crawl. See also: Google Search Central - How to write robots.txt rules. https://www.example.com/sitemap.xml
rules Map<string | string[], ({ allow: string } | { disallow: string })[]> | undefined (Optional) A map of robots.txt rules, grouped by user agent. Each key is an array of user agents in the group; the value is an array of allowed or disallowed paths. See examples.
shouldBlockAIRobots string | undefined (Optional) Whether to soft-block a list of known AI robots (see the ai-robots GitHub repository for context). true
frontMatterOverrides Record<string, unknown> | undefined Front matter overrides to apply to the robots.txt template. By default, the plugin automatically applies eleventyExcludeFromCollections: true and permalink: /robots.txt, so you do not need to set these yourself. { "frontMatterKey": "value" }

Examples

The following examples are direct translations of Google's guide on how to write and submit a robots.txt file.

Disallow crawling of the entire site

Input:

const eleventyPluginRobotsTxtOptions = {
  rules: new Map([["*", [{ disallow: "/" }]]]),
};

Output:

User-agent: *
Disallow: /

Disallow crawling of a directory and its contents

Input:

const eleventyPluginRobotsTxtOptions = {
  rules: new Map([
    [
      "*",
      [
        { disallow: "/calendar/" },
        { disallow: "/junk/" },
        { disallow: "/books/fiction/contemporary/" },
      ],
    ],
  ]),
};

Output:

User-agent: *
Disallow: /calendar/
Disallow: /junk/
Disallow: /books/fiction/contemporary/

Allow access to a single crawler

Input:

const eleventyPluginRobotsTxtOptions = {
  rules: new Map([
    ["Googlebot-news", [{ allow: "/" }]],
    ["*", [{ disallow: "/" }]],
  ]),
};

Output:

User-agent: Googlebot-news
Allow: /

User-agent: *
Disallow: /

Allow access to all but a single crawler

Input:

const eleventyPluginRobotsTxtOptions = {
  rules: new Map([
    ["Unnecessarybot", [{ disallow: "/" }]],
    ["*", [{ allow: "/" }]],
  ]),
};

Output:

User-agent: Unnecessarybot
Disallow: /

User-agent: *
Allow: /

Disallow crawling of a single web page

For example, disallow the useless_file.html page located at https://example.com/useless_file.html, and other_useless_file.html in the junk directory.

Input:

const eleventyPluginRobotsTxtOptions = {
  rules: new Map([
    [
      "*",
      [
        { disallow: "/useless_file.html" },
        { disallow: "/junk/other_useless_file.html" },
      ],
    ],
  ]),
};

Output:

User-agent: *
Disallow: /useless_file.html
Disallow: /junk/other_useless_file.html

Disallow crawling of the whole site except a subdirectory

Input:

const eleventyPluginRobotsTxtOptions = {
  rules: new Map([["*", [{ disallow: "/" }, { allow: "/public/" }]]]),
};

Output:

User-agent: *
Disallow: /
Allow: /public/

Block a specific image from Google Images

Input:

const eleventyPluginRobotsTxtOptions = {
  rules: new Map([["Googlebot-Image", [{ disallow: "/images/dogs.jpg" }]]]),
};

Output:

User-agent: Googlebot-Image
Disallow: /images/dogs.jpg

Block all images on your site from Google Images

Input:

const eleventyPluginRobotsTxtOptions = {
  rules: new Map([["Googlebot-Image", [{ disallow: "/" }]]]),
};

Output:

User-agent: Googlebot-Image
Disallow: /

Disallow crawling of files of a specific file type

Input:

const eleventyPluginRobotsTxtOptions = {
  rules: new Map([["Googlebot", [{ disallow: "/*.gif$" }]]]),
};

Output:

User-agent: Googlebot
Disallow: /*.gif$

Disallow crawling of an entire site, but allow Mediapartners-Google

Input:

const eleventyPluginRobotsTxtOptions = {
  rules: new Map([
    ["*", [{ disallow: "*" }]],
    ["Mediapartners-Google", [{ allow: "/" }]],
  ]),
};

Output:

User-agent: *
Disallow: /

User-agent: Mediapartners-Google
Allow: /

Disallow crawling for multiple user agents in one go

Input:

const eleventyPluginRobotsTxtOptions = {
  rules: new Map([[["agent1", "agent2", "agent3"], [{ disallow: "/" }]]]),
};

Output:

User-agent: agent1
User-agent: agent2
User-agent: agent3
Disallow: /

Notes