/plug_robots

An Elixir plug serving robots.txt.

Primary LanguageElixir

PlugRobots

An Elixir plug serving robots.txt.

Installation

Add plug_robots to your list of dependencies in mix.exs:

def deps do
  [{:plug_robots, "~> 0.1.0"}]
end

Usage

  • Add it to your endpoint before the router

endpoint.ex

plug PlugRobots, rules: :allow_all

plug MyApp.Router

Examples

  • Deny all crawler
plug PlugRobots, rules: :deny_all

plug MyApp.Router
  • Returns different rules by environment
case Mix.env do
    :prod -> plug PlugRobots, rules: :allow_all
    _     -> plug PlugRobots, rules: :deny_all
end

plug MyApp.Router
  • Returns custom rules
@googlebot_rule """
User-agent: GoogleBot
Disallow: /images/dogs.jpg
"""

plug PlugRobots, rules: @googlebot_rule

plug MyApp.Router

Options

  • rules
    • :allow_all: Allow all crawlers to all path
    • :deny_all: Disallow all crawlers to all path
    • string: any other custom rules