Robotify - robots.txt middleware for .NET core (.netstandard2.0)
Install package via NuGet:
Install-Package Robotify.AspNetCoreAdd Robotify to the middleware pipeline like any other:
Add to the ConfigureServices method to ensure the middleware and configuration are added to the services collection
public void ConfigureServices(IServiceCollection services)
{
services.AddOptions();
services.AddRobotify();
services.AddMvc();
}To configure what robots directives you want to appear in your robots file you can create and assign a IRobotifyRobotGroupProvider. Multiple providers can be created and registered in the standard DI container, each provider will then be iterated when generating the contents of the robots file.
A IRobotifyRobotGroupProvider contains a single method that returns a list of RobotGroup, below is an example of the RobotifyDisallowAllRobotGroupProvider:
public class YourOwnProvider : IRobotifyRobotGroupProvider
{
public IEnumerable<RobotGroup> Get()
{
yield return new RobotGroup()
{
UserAgent = "something",
Disallow = new[] { "/" }
}
}
}To register a provider, use the .AddRobotGroupProvider<TProvider>() extension method when adding robotify:
public void ConfigureServices(IServiceCollection services)
{
services.AddRobotify(c => c
.AddDisallowAllRobotGroupProvider()
.AddRobotGroupsFromAppSettings()
.AddRobotGroupProvider<YourOwnProvider>()
);
}Then use the middleware as part of the request pipeline, I would suggest adding it at least after the .UseStaticFiles() middleware, then if you have a static file robots.txt this will be used instead.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
app.UseStaticFiles();
app.UseRobotify();
app.UseMvc(routes =>
{
routes.MapRoute(
name: "default",
template: "{controller=Home}/{action=Index}/{id?}");
});
}Robotify can be configured within the .UseRobotify() method or via json configuration:
{
"Robotify": {
"Enabled": true,
"SitemapUrl": "https://www.example.com/sitemap.xml",
"CrawlDelay": 10,
"Groups": [
{
"UserAgent": "*",
"Disallow": "/"
},
{
"UserAgent": "Googlebot",
"Disallow": ["/admin"]
},
{
"UserAgent": "AnotherBot",
"Allow": ["/search"]
}
]
}
}Json configuration and code configuration will be merged at boot time. If no groups are specified a default deny all ill be added.