Elasticsearch Azure Marketplace offering

This repository consists of:

src/mainTemplate.json - The main Azure Resource Management (ARM) template. The template itself is composed of many nested linked templates with the main template acting as the entry point.
src/createUiDefinition - UI definition file for our Azure Marketplace offering. This file produces an output JSON that the ARM template can accept as input parameters.

Building

After pulling the source, call npm install once to pull in all devDependencies.

You may edit build/allowedValues.json, which the build will use to patch the ARM template and Marketplace UI definition.

Run npm run build; this will validate EditorConfig settings, JSON files, patch the allowedValues and create a zip in the dist folder.

For more details around developing the template, take a look at the Development README

Azure Marketplace

The Azure Marketplace Elasticsearch offering offers a simplified UI over the full power of the ARM template.

It will always bootstrap a cluster complete with a trial license of Elastic's commercial X-Pack plugins.

Did you know that you can apply for a free basic license? Go check out our subscription options

Deploying through the Marketplace is great and easy way to get your feet wet for a first time with Elasticsearch (on Azure) but in the long run, you'll want to deploy the templates directly though the Azure CLI or PowerShell SDKs. Check out the examples.

VERY IMPORTANT

This template does not configure SSL/TLS for communication with Elasticsearch through an external load balancer. It is strongly recommended that you secure communication before using in production.

You can secure external access to the cluster with TLS by using gateway as the loadBalancerType and supplying a PFX certificate with the appGatewayCertBlob parameter. This sets the cluster up to use Application Gateway for load balancing and SSL offload.

You can secure external access from the browser to Kibana with TLS by supplying a certificate and private key with kibanaCertBlob and kibanaKeyBlob, respectively.

You can view the UI in developer mode by clicking here. If you feel something is cached improperly use this client unoptimized link instead

Reporting bugs

Have a look at this screenshot to see how you can navigate to the deployment error status message. Please create an issue with that message and in which resource it occured on our github issues

ARM template

The output from the Azure Marketplace UI is fed directly to the ARM deployment template. You can use the ARM template on its own without going through the MarketPlace. In fact, there are many features in the ARM template that are not exposed within the Marketplace such as configuring

Azure Storage account to use for Snapshot/Restore
Application Gateway to use for TLS and SSL offload
The number and size of disks to attach to each data node VM

Check out our examples repository for examples of common scenarios and also take a look at the following blog posts for further information

Parameters

Parameter	Type	Description	Default Value
artifactsBaseUrl	string	The base url of the Elastic ARM template.	Raw content of the current branch
esVersion	string	A valid supported Elasticsearch version. See this list for supported versions	The latest version of Elasticsearch supported by the ARM template version
esClusterName	string	The name of the Elasticsearch cluster. Required	`""`
loadBalancerType	string	The load balancer to set up to access the cluster. Can be `internal`, `external` or `gateway`. By choosing `internal`, only an internal load balancer is deployed. Useful when connecting to the cluster happens from inside the Virtual Network By choosing `external`, both internal and external load balancers will be deployed. Kibana communicates with the cluster through the internal load balancer. By choosing `gateway`, Application Gateway will be deployed for load balancing, allowing a PFX certificate to be supplied for transport layer security to and from Application Gateway, and providing SSL offload. An internal load balancer will also deployed. Application Gateway and Kibana communicate with the cluster through the internal load balancer. If you are setting up Elasticsearch or Kibana on a publicly available IP address, it is highly recommended to secure access to the cluster with a product like Elastic's Security, in addition to configuring transport layer security.	`internal`
azureCloudPlugin	string	Either `Yes` or `No` to install the Azure Cloud plugin for snapshot/restore. When set to `Yes`, both `azureCloudeStorageAccountName` and `azureCloudStorageAccountKey` should be specified to configure the plugin correctly.	`No`
azureCloudStorageAccountName	string	The name of an existing storage account to use for snapshots with Azure Cloud plugin. Must be a valid Azure Storage Account name.	`""`
azureCloudStorageAccountKey	securestring	The access key of an existing storage account to use for snapshots with Azure Cloud plugin.	`""`
xpackPlugins	string	Either `Yes` or `No` to install a trial license of the commercial X-Pack plugins: Monitoring, Security, Alerting and Graph (Elasticsearch 2.3.0+).	`Yes`
esAdditionalPlugins	string	Additional elasticsearch plugins to install. Each plugin must be separated by a semicolon. e.g. `analysis-icu;mapper-attachments`	`""`
esAdditionalYaml	string	Additional configuration for Elasticsearch yaml configuration file. Each line must be separated by a newline character `\n` e.g. `"action.auto_create_index: .security\nindices.queries.cache.size: 5%"`. This is an expert level feature - It is recommended that you run your additional yaml through a linter before starting a deployment.	`""`
kibana	string	Either `Yes` or `No` to provision a machine with a public IP that has Kibana installed on it. If you have opted to also install the Elasticsearch plugins using `xpackPlugins` then a trial license of the commercial X-Pack Kibana plugins as well as Sense Editor are also installed.	`Yes`
vmSizeKibana	string	Azure VM size of the Kibana instance. See this list for supported sizes. Check that the size you choose is available in the region you choose.	`Standard_A1`
kibanaCertBlob	string	A Base-64 encoded form of the certificate (.crt) to secure HTTPS communication between the browser and Kibana.	`""`
kibanaKeyBlob	securestring	A Base-64 encoded form of the private key (.key) to secure HTTPS communication between the browser and Kibana.	`""`
kibanaKeyPassphrase	securestring	The passphrase to decrypt the private key. Optional as the key may not be encrypted. Supported only in 5.3.0+	`""`
jumpbox	string	Either `Yes` or `No` to optionally add a virtual machine to the deployment which you can use to connect and manage virtual machines on the internal network.	`No`
vmHostNamePrefix	string	The prefix to use for hostnames when naming virtual machines in the cluster. Hostnames are used for resolution of master nodes so if you are deploying a cluster into an existing virtual network containing an existing Elasticsearch cluster, be sure to set this to a unique prefix, to differentiate the hostnames of this cluster from an existing cluster. Can be up to 5 characters in length, must begin with an alphanumeric character and can contain alphanumeric and hyphen characters.	`""`
vmSizeDataNodes	string	Azure VM size of the data nodes. See this list for supported sizes. Check that the size you choose is available in the region you choose.	`Standard_D1`
vmDataDiskCount	int	Number of disks to attach to each data node in RAID 0 setup. Must be one of `0`, `1`, `2`, `4`, `8`, `16`, `32`, `40`. If the number of disks selected is more than can be attached to the data node VM size, the maximum number of disks that can be attached for the data node VM size will be used. Equivalent to taking `min(vmDataDiskCount, max supported disks for data node VM size)` When 1 disk is selected, the disk is not RAIDed. When 0 disks are selected, no disks will be attached to each data node; instead, the temporary disk will be used to store Elasticsearch data. The temporary disk is ephemeral in nature and not persistent. Consult Microsoft Azure documentation on temporary disks to understand the trade-offs in using it for storage.	`40` i.e. the max supported disks for data node VM size
vmDataDiskSize	string	The disk size of each attached disk. Choose `Large` (1024Gb), `Medium` (512Gb) or `Small` (128Gb). For Premium Storage, disk sizes equate to P30, P20 and P10 storage disk types, respectively.	`Large`
vmDataNodeCount	int	The number of data nodes you wish to deploy. Must be greater than 0.	`3`
storageAccountType	string	The storage account type of the attached disks. Choose either `Default` or `Standard`. The `Default` storage account type will be Premium Storage for VMs that support Premium Storage and Standard Storage for those that do not.	`Default`
dataNodesAreMasterEligible	string	Either `Yes` or `No` to make all data nodes master eligible. This can be useful for small Elasticsearch clusters however, for larger clusters it is recommended to have dedicated master nodes. When `Yes` no dedicated master nodes will be provisioned.	`No`
vmSizeMasterNodes	string	Azure VM size of dedicated master nodes. See this list for supported sizes. By default the template deploys 3 dedicated master nodes, unless `dataNodesAreMasterEligible` is set to `Yes`. Check that the size you choose is available in the region you choose.	`Standard_D1`
vmClientNodeCount	int	The number of client nodes to provision. Must be a positive integer. By default, the data nodes are added to the backendpool of the loadbalancer but if you provision client nodes, these will be added to the loadbalancer instead. Client nodes can be useful in offloading the gather process from data nodes and are necessary to scale an Elasticsearch cluster deployed with this template beyond 100 data nodes (the maximum number of VMs that can be added to a load balancer backendpool).	`0`
vmSizeClientNodes	string	Azure VM size of the client nodes see this list for supported sizes. Check that the size you choose is available in the region you choose.	`Standard_D1`
adminUsername	string	Admin username used when provisioning virtual machines. Must be a valid Linux username i.e. avoid any of the following usernames for Ubuntu	`""`
authenticationType	string	The authentication type for the Admin user. Either `password` or `sshPublicKey`	`password`
adminPassword	securestring	When `authenticationType` is `password` this sets the OS level user's password	`""`
sshPublicKey	securestring	When `authenticationType` is `sshPublicKey` this sets the OS level sshKey that can be used to login.	`""`
securityAdminPassword	securestring	The password for 5.x's superuser `elastic` or, in 2.x the `es_admin` user, with admin role.	`""`
securityReadPassword	securestring	Security password for the `es_read` user with user (read-only) role, must be > 6 characters	`""`
securityKibanaPassword	securestring	Security password for the `es_kibana` user with kibana4 role, must be > 6 characters	`""`
location	string	The location where to provision all the items in this template. Defaults to the special `ResourceGroup` value which means it will inherit the location from the resource group. Any other value must be a valid Azure region.	`ResourceGroup`
vNetNewOrExisting	string	Whether the Virtual Network is `new` or `existing`. An `existing` Virtual Network in another Resource Group in the same Location can be used.	`new`
vNetName	string	The name of the Virtual Network.	`es-net`
vNetExistingResourceGroup	string	The name of the Resource Group in which the Virtual Network resides when using an existing Virtual Network. Required when using an existing Virtual Network	`""`
vNetNewAddressPrefix	string	The address prefix when creating a new Virtual Network. Required when creating a new Virtual Network	`10.0.0.0/24`
vNetLoadBalancerIp	string	The internal static IP address to use when configuring the internal load balancer. Must be an available IP address on the provided `vNetClusterSubnetName`.	`10.0.0.4`
vNetClusterSubnetName	string	The name of the subnet to which Elasticsearch nodes will be attached.	`es-subnet`
vNetNewClusterSubnetAddressPrefix	string	The address space of the subnet. Required when creating a new Virtual Network	`10.0.0.0/25`
vNetAppGatewaySubnetName	string	Subnet name to use for the Application Gateway. Required when selecting `gateway` for load balancing.	`es-gateway-subnet`
vNetNewAppGatewaySubnetAddressPrefix	string	The address space of the Application Gateway subnet. Required when creating a new Virtual Network and selecting `gateway` for load balancing.	`10.0.0.128/28`
appGatewayTier	string	The tier of the Application Gateway, either `Standard` or `WAF`. Required when selecting `gateway` for load balancing.	`Standard`
appGatewaySku	string	The size of the Application Gateway. Choose `Small`, `Medium` or `Large`. When choosing appGatewayTier `WAF`, the size must be at least `Medium`. Required when selecting `gateway` for load balancing.	`Medium`
appGatewayCount	int	The number instances of the Application Gateway. Can be a value between `1` and `10`. A minimum of `2` is recommended for production. Required when selecting `gateway` for load balancing.	`2`
appGatewayCertBlob	string	A Base-64 encoded form of the PFX certificate for the Application Gateway. This certificate is used to secure HTTPS connections to and from the Application Gateway. Required when selecting `gateway` for load balancing.	`""`
appGatewayCertPassword	securestring	The password for the PFX certificate for the Application Gateway. Defaults to `""`. Required when selecting `gateway` for load balancing.	`""`
appGatewayWafStatus	string	The firewall status of the Application Gateway, either `Enabled` or `Disabled`. Required when selecting `gateway` for load balancing and using appGatewayTier `WAF.`	`Enabled`
appGatewayWafMode	string	The firewall mode of the Application Gateway, either `Detection` or `Prevention`. Required when selecting `gateway` for load balancing and using appGatewayTier `WAF.`	`Detection`
userCompany	string	The name of your company.	`""`
userEmail	string	Your email address	`""`
userFirstName	string	Your first name	`""`
userLastName	string	Your last name	`""`
userJobTitle	string	Your job title. Pick the nearest one that matches from the list of job titles	`Other`
userCountry	string	The country in which you are based.	`""`

Command line deploy

You can deploy using the template directly from Github using the Azure CLI or Azure PowerShell

Azure CLI

Log into Azure

azure login

Ensure you are in arm mode

azure config mode arm

Create a resource group <name> in a <location> (e.g westeurope) where we can deploy too

azure group create <name> <location>

Use our published template directly using --template-uri

azure group deployment create --template-uri https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json --parameters-file parameters/password.parameters.json -g name

or if your are executing commands from a clone of this repo using --template-file

azure group deployment create --template-file src/mainTemplate.json --parameters-file parameters/password.parameters.json -g <name>

where <name> refers to the resource group you just created.

NOTE

The --parameters-file can specify a different location for the items that get provisioned inside of the resource group. Make sure these are the same prior to deploying if you need them to be. Omitting location from the parameters file is another way to make sure the resources get deployed in the same location as the resource group.

Azure PowerShell

Log into Azure

Login-AzureRmAccount

Select a Subscription Id

Select-AzureRmSubscription -SubscriptionId "<subscriptionId>"

Define the parameters object for your deployment

$clusterParameters = @{
    "artifactsBaseUrl"="https://raw.githubusercontent.com/elastic/azure-marketplace/master/src"
    "esVersion" = "5.1.2"
    "esClusterName" = "elasticsearch"
    "location" = "ResourceGroup"
    "loadBalancerType" = "internal"
    "vmDataDiskCount" = 1
    "adminUsername" = "russ"
    "adminPassword" = "Password1234"
    "securityAdminPassword" = "Password123"
    "securityReadPassword" = "Password123"
    "securityKibanaPassword" = "Password123"
}

Create a resource group <name> in a <location> (e.g westeurope) where we can deploy too

New-AzureRmResourceGroup -Name "<name>" -Location "<location>"

Use our template directly from Github

New-AzureRmResourceGroupDeployment -Name "<deployment name>" -ResourceGroupName "<name>" -TemplateUri "https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json" -TemplateParameterObject $clusterParameters

Targeting a specific template version

You can target a specific version of the template by modifying the URI of the template and the artifactsBaseUrl parameter of the template.

For example, to target the 5.0.1 tag release with PowerShell

$templateVersion = "5.0.1"
$templateBaseUrl = "https://raw.githubusercontent.com/elastic/azure-marketplace/$templateVersion/src"

$clusterParameters = @{
    "artifactsBaseUrl"= $templateBaseUrl
    "esVersion" = "5.0.0"
    "adminUsername" = "russ"
    "adminPassword" = "Password1234"
    "securityAdminPassword" = "Password123"
    "securityReadPassword" = "Password123"
    "securityKibanaPassword" = "Password123"
}

New-AzureRmResourceGroup -Name "<name>" -Location "<location>"
New-AzureRmResourceGroupDeployment -Name "<deployment name>" -ResourceGroupName "<name>" -TemplateUri "$templateBaseUrl/mainTemplate.json" -TemplateParameterObject $clusterParameters

Web based deploy

The above button will take you to the autogenerated web based UI based on the parameters from the ARM template.

License

This project is MIT Licensed and is based heavily on the Elasticsearch azure quick start arm template

makhdumi/azure-marketplace