logstash-output-documentdb is a logstash plugin to output to Azure DocumentDB. Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite destinations. Azure DocumentDB is a managed NoSQL database service provided by Microsoft Azure. It’s schemaless, natively support JSON, very easy-to-use, very fast, highly reliable, and enables rapid deployment, you name it.
You can install this plugin using the Logstash "plugin" or "logstash-plugin" (for newer versions of Logstash) command:
bin/plugin install logstash-output-documentdb
# or
bin/logstash-plugin install logstash-output-documentdb (Newer versions of Logstash)
Please see Logstash reference for more information.
output {
documentdb {
docdb_endpoint => "https://<YOUR ACCOUNT>.documents.azure.com:443/"
docdb_account_key => "<ACCOUNT KEY>"
docdb_database => "<DATABASE NAME>"
docdb_collection => "<COLLECTION NAME>"
auto_create_database => true|false
auto_create_collection => true|false
partitioned_collection => true|false
partition_key => "<PARTITIONED KEY NAME>"
offer_throughput => <THROUGHPUT NUM>
}
}
- docdb_endpoint (required) - Azure DocumentDB Account endpoint URI
- docdb_account_key (required) - Azure DocumentDB Account key (master key). You must NOT set a read-only key
- docdb_database (required) - DocumentDB database nameb
- docdb_collection (required) - DocumentDB collection name
- auto_create_database (optional) - Default:true. By default, DocumentDB database named docdb_database will be automatically created if it does not exist
- auto_create_collection (optional) - Default:true. By default, DocumentDB collection named docdb_collection will be automatically created if it does not exist
- partitioned_collection (optional) - Default:false. Set true if you want to create and/or store records to partitioned collection. Set false for single-partition collection
- partition_key (optional) - Default:nil. Partition key must be specified for paritioned collection (partitioned_collection set to be true)
- offer_throughput (optional) - Default:10100. Throughput for the collection expressed in units of 100 request units per second. This is only effective when you newly create a partitioned collection (ie. Both auto_create_collection and partitioned_collection are set to be true )
logstash-output-documentdb adds id attribute (UUID format) to in-coming events automatically and send them to DocumentDB. Here is an example configuration where Logstash's event source and destination are configured as Apache2 access log and DocumentDB respectively.
input {
file {
path => "/var/log/apache2/access.log"
start_position => "beginning"
}
}
filter {
if [path] =~ "access" {
mutate { replace => { "type" => "apache_access" } }
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
documentdb {
docdb_endpoint => "https://yoichikademo.documents.azure.com:443/"
docdb_account_key => "EMwUa3EzsAtJ1qYfzwo9nQxxydofsXNm3xLh1SLffKkUHMFl80OZRZIVu4lxdKRKxkgVAj0c2mv9BZSyMN7tdg==(dummy)"
docdb_database => "testdb"
docdb_collection => "apache_access"
auto_create_database => true
auto_create_collection => true
}
# for debug
stdout { codec => rubydebug }
}
You can find example configuration files in logstash-output-documentdb/examples.
Now you run logstash with the the example configuration like this:
# Test your logstash configuration before actually running the logstash
bin/logstash -f logstash-apache2-to-documentdb.conf --configtest
# run
bin/logstash -f logstash-apache2-to-documentdb.conf
Here is an expected output for sample input (Apache2 access log):
Apache2 access log
124.211.152.166 - - [27/Dec/2016:02:12:28 +0000] "GET /test.html HTTP/1.1" 200 316 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
Output (rubydebug)
{
"message" => "124.211.152.166 - - [27/Dec/2016:02:12:28 +0000] \"GET /test.html HTTP/1.1\" 200 316 \"-\" \"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36\"",
"@version" => "1",
"@timestamp" => "2016-12-27T02:12:28.000Z",
"path" => "/var/log/apache2/access.log",
"host" => "yoichitest01",
"type" => "apache_access",
"clientip" => "124.211.152.166",
"ident" => "-",
"auth" => "-",
"timestamp" => "27/Dec/2016:02:12:28 +0000",
"verb" => "GET",
"request" => "/test.html",
"httpversion" => "1.1",
"response" => "200",
"bytes" => "316",
"referrer" => "\"-\"",
"agent" => "\"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36\"",
"id" => "0cae1966-b7ab-4f32-8893-b4fabc7800ae"
}
Bug reports and pull requests are welcome on GitHub at https://github.com/yokawasa/logstash-output-documentdb.