Dynamic Sitemaps is a plugin for Ruby on Rails that enables you to easily create flexible, dynamic sitemaps. It creates sitemaps in the sitemaps.org standard which is supported by several crawlers including Google, Bing, and Yahoo.
Dynamic Sitemaps is designed to be very (very) simple so there's a lot you cannot do, but possibly don't need (I didn't). If you need an advanced sitemap generator, please see Karl Varga's SitemapGenerator.
Version 2.0 makes it possible to make very large sitemaps (up to 2.5 billion URLs) in a fast and memory efficient way; it is built for large amounts of data, i.e. millions of URLs without pushing your server to the limit, memory and CPU wise.
Version 2.0 is not compatible with version 1.0 (although the configuration DSL looks somewhat the same) as version 2.0 generates static sitemap XML files whereas 1.0 generated them dynamically on each request (slow for large sitemaps).
DynamicSitemaps is tested in Rails 3.2.13 using Ruby 1.9.3 and 2.0.0, but should work in other versions of Rails 3 and above and Ruby 1.9 and above. Please create an issue if you encounter any problems.
Add this line to your application's Gemfile:
gem "dynamic_sitemaps", "2.0.0.beta2"
And then execute:
$ bundle
Or install it yourself as:
$ gem install dynamic_sitemaps
To generate a simple example config file in config/sitemap.rb
:
$ rails generate dynamic_sitemaps:install
If you want to use version 1.0 (v1.0.8) of DynamicSitemaps, please see v1.0.8 of the project. Please note that this version isn't good for large sitemaps as it generates them dynamically on each request.
The configuration file in config/sitemap.rb
goes like this (also see the production example below for more advance usage like multiple sites / hosts, etc.):
host "www.example.com"
# Basic sitemap – you can change the name :site as you wish
sitemap :site do
url root_url, last_mod: Time.now, change_freq: "daily", priority: 1.0
end
# Pings search engines after generation has finished
ping_with "http://#{host}/sitemap.xml"
The host is needed to generate the URLs because the rake task doesn't know anything about the host being used.
Then, to generate the sitemap:
$ rake sitemap:generate
This will, by default, generate a sitemap.xml
file in <project root>/public/sitemaps
that will look like this:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://www.example.com/</loc>
<lastmod>2013-07-08T17:02:45+02:00</lastmod>
<changefreq>daily</changefreq>
<priority>1.0</priority>
</url>
</urlset>
You then need to symlink from public/sitemap.xml
(or whatever you choose) to public/sitemaps/sitemap.xml
:
$ ln -s /path/to/project/public/sitemaps/sitemap.xml /path/to/project/public/sitemap.xml
See the below production example for inspiration on how to do this with Capistrano, and other things like multiple sites / hosts, etc.
If a sitemap contains over 50,000 URLs, then by default, as specified by the sitemaps.org standard, DynamicSitemaps will split it into multiple sitemaps and generate an index file that will also be named public/sitemaps/sitemap.xml
by default.
The sitemap files will then be named site.xml
, site2.xml
, site3.xml
, and so on, and the index file will link to these files using the host set with host
.
DynamicSitemaps can automatically generate sitemaps for ActiveRecord models with the built-in Rails resourceful routes (the ones you create using routes :model_name
).
Example:
host "www.example.com"
# Basic sitemap
sitemap :site do
url root_url, last_mod: Time.now, change_freq: "daily", priority: 1.0
end
# Automatically link to all pages using the routes specified
# using "resources :pages" in config/routes.rb. This will also
# automatically set <lastmod> to the date and time in page.updated_at.
sitemap_for Page.scoped
# For products with special sitemap name and priority, and link to comments
sitemap_for Product.published, name: :published_products do |product|
url product, last_mod: product.updated_at, priority: (product.featured? ? 1.0 : 0.7)
url product_comments_url(product)
end
This generates the sitemap files site.xml
, pages.xml
, and published_products.xml
and links them together in the sitemap.xml
index file, splitting them into multiple sitemap files if the number of URLs exceeds 50,000.
The argument passed to sitemap_for
needs to respond to #find_each
, like an ActiveRecord Relation.
This is to ensure that the records from the database are lazy loaded 1,000 at a time, so that it doesn't accidentally load millions of records in one call when the configuration file is read.
Therefore we use Page.scoped
instead of the normal Page.all
.
You can configure different options of how DynamicSitemaps behaves, including the sitemap path and index file name.
In an initializer, e.g. config/initializers/dynamic_sitemaps.rb
:
# These are the built-in defaults, so you don't need to specify them.
DynamicSitemaps.configure do |config|
config.path = Rails.root.join("public")
config.folder = "sitemaps" # This folder is emptied on each sitemap generation
config.index_file_name = "sitemap.xml"
config.always_generate_index = false # Makes sitemap.xml contain the sitemap
# (e.g. site.xml) when only one sitemap
# file has been generated
config.config_path = Rails.root.join("config", "sitemap.rb")
config.per_page = 50000
end
DynamicSitemaps can automatically ping Google and Bing (and other search engines you specify) with the sitemap when the generation finishes.
In config/sitemap.rb
:
host "www.example.com"
sitemap :site do
url root_url
end
ping_with "http://#{host}/sitemap.xml"
To customize it, in e.g. config/initializers/dynamic_sitemaps.rb
:
DynamicSitemaps.configure do |config|
# Default is Google and Bing
config.search_engine_ping_urls << "http://customsearchengine.com/ping?url=%s"
# Default is pinging only in production
config.ping_environments << "staging"
end
DynamicSitemaps generates to a temporary directory (<rails root>/tmp/dynamic_sitemaps
) first and, when finished, moves the files into the destination (by default public/sitemaps
).
So in case you have generated a sitemap succesfully and the next sitemap generation fails, your sitemap files will remain untouched and available.
This is an example of a real production app that uses DynamicSitemaps with multiple sites and domains in one app, Capistrano for deployment, and Whenever for crontab scheduling.
In config/sitemap.rb
:
Site.all.each do |site|
folder "sitemaps/#{site.key}"
host site.domain
sitemap :site do
url root_url, priority: 1.0, change_freq: "daily"
url blog_posts_url
url tags_url
end
sitemap_for site.pages.where("slug != 'home'")
sitemap_for site.blog_posts.published
sitemap_for site.tags.scoped
sitemap_for site.products.where("type_id != ?", ProductType.find_by_key("unknown").id) do |product|
url product, last_mod: product.updated_at, priority: (product.featured? ? 1.0 : 0.7)
end
ping_with "http://#{host}/sitemap.xml"
end
In config/routes.rb
:
get "sitemap.xml" => "home#sitemap", format: :xml, as: :sitemap
get "robots.txt" => "home#robots", format: :text, as: :robots
In app/controllers/home_controller.rb
:
class HomeController < ApplicationController
# ...
def sitemap
path = Rails.root.join("public", "sitemaps", current_site.key, "sitemap.xml")
if File.exists?(path)
render xml: open(path).read
else
render text: "Sitemap not found.", status: :not_found
end
end
def robots
end
end
In app/views/home/robots.text.erb
:
Sitemap: <%= sitemap_url %>
Capistrano deployment configuration in config/deploy.rb
:
after "deploy:update_code", "sitemaps:create_symlink"
namespace :sitemaps do
task :create_symlink, roles: :app do
run "mkdir -p #{shared_path}/sitemaps"
run "rm -rf #{release_path}/public/sitemaps"
run "ln -s #{shared_path}/sitemaps #{release_path}/public/sitemaps"
end
end
For automatic crontab scheduling with Whenever, in config/schedule.rb
:
every 1.day, at: "6am" do
rake "sitemap:generate"
end
This will automatically generate the sitemaps and ping Google and Bing every day at 6am using the sitemap URLs configured above.
If you encounter any problems with DynamicSitemaps, please create an issue. If you want to fix the problem (please do 😄), please see below.
Help is always appreciated whether it be improvement of the code, testing, or adding new relevant features. Please create an issue before implementing a new feature, so we can discuss it in advance. Thanks.
- Fork the repo
- Create your feature branch (
git checkout -b my-new-feature
) - Commit your changes (
git commit -am 'Add feature'
) - Push to the branch (
git push origin my-new-feature
) - Create new Pull Request