This is an admin extension for Optimizely CMS 12+ for managing robots.txt on a per site and host definition basis. The Stott.Optimizely.RobotsHandler project is a Razor Class Library which contains Razor Files, static JS files and relevent business logic specific to this functionality.
Stott Robots Handler is a free to use module, however if you want to show your support, buy me a coffee on ko-fi:
After pulling in a reference to the Stott.Optimizely.RobotsHandler project, you need to ensure the following lines are added to the startup class of your solution:
public void ConfigureServices(IServiceCollection services)
{
services.AddRobotsHandler();
}
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
app.UseEndpoints(endpoints =>
{
endpoints.MapContent();
endpoints.MapControllers();
});
}
The call to services.AddRobotsHandler() sets up the dependency injection requirements for the RobotsHandler solution and is required to ensure the solution works as intended. This works by following the Services Extensions pattern defined by Microsoft.
The call to endpoints.MapControllers(); ensures that the routing for the administration page, assets and robots.txt are correctly mapped.
As this package includes static files such as JS and CSS files within the Razor Class Library, your solution must be configured to use Static Web Assets. This is done by adding webBuilder.UseStaticWebAssets(); to your Program.cs as follows:
Host.CreateDefaultBuilder(args)
.ConfigureCmsDefaults()
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
webBuilder.UseStaticWebAssets();
});
You can read more about shared assets in Razor Class Libraries here: Create reusable UI using the Razor class library project in ASP.NET Core
This solution also includes an implementation of IMenuProvider which ensures that the Robots Handler administration pages are included in the CMS Admin menu under the title of "Robots". You do not have to do anything to make this work as Optimizely CMS will scan and action all implementations of IMenuProvider.
The configuration of the module has some scope for modification by providing configuration in the service extension methods. The provision of authorizationOptions is optional in the following example.
Example:
services.AddRobotsHandler(authorizationOptions =>
{
authorizationOptions.AddPolicy(RobotsConstants.AuthorizationPolicy, policy =>
{
policy.RequireRole("WebAdmins");
});
});If the authorizationOptions is not provided, then any of the following roles will be required by default:
- CmsAdmins
- Administrator
- WebAdmins
If you are using the new Optimizely Opti ID package for authentication into Optimizely CMS and the rest of the Optimizely One suite, then you will need to define the authorizationOptions for this module as part of your application start up. This should be a simple case of adding policy.AddAuthenticationSchemes(OptimizelyIdentityDefaults.SchemeName); to the authorizationOptions as per the example below.
services.AddRobotsHandler(authorizationOptions =>
{
authorizationOptions.AddPolicy(RobotsConstants.AuthorizationPolicy, policy =>
{
policy.AddAuthenticationSchemes(OptimizelyIdentityDefaults.SchemeName);
policy.RequireRole("WebAdmins");
});
});I am open to contributions to the code base. The following rules should be followed:
- Contributions should be made by Pull Requests.
- All commits should have a meaningful message.
- All commits should have a reference to your GitHub user.
- Ideally, all new changes should include appropriate unit test coverage.
Thank you for your feedback and contributions go to the following members of the community:
| Contributor | Bug Reports | Pull Requests |
|---|---|---|
| Paul Mcgann | 1 | 1 |
| Ellinge | 1 | 1 |
| Tomas Hensrud Gulla | - | 1 |
| Anish Peethambaran | 1 | - |
| Deepa V Puranik | 1 | - |
| Mahdi Shahbazi | 1 | - |
| Praveen Soni | 1 | - |