garyrusso/spreadsheets

Search oriented spreadsheet database app

NoSQL Spreadsheet Demo

Purpose

This app will show the following.

1. Best Approaches for storing and retrieving spreadsheet data.
2. Best way to do AngularJS with NoSQL.
3. Ways to extend the REST API.
4. Configure and test leading wildcard searches.
5. Best Approaches for Schema-agnostic data.
6. Multi-Statement ACID Transactions.
7. Bi-Temporal Data Structure and Query used for Point-In-Time Audits.
8. Triplestore Document Linking.
9. Triplestore Queries for Forward and Backward Chain Inferencing.

RESTful APIs

Request Mappings
Method	URL Template	Parameters	Description
Search APIs
GET	/search	rs:q query string leading and trailing wildcards constraints: dname row col cell rs:start starting record rs:pageLength number of records to return	Searches entire data hub. Returns search results using custom snippets with highlighting.
GET	/spreadsheet/search	rs:q query string leading and trailing wildcards constraints: dname row col cell user rs:start starting record rs:pageLength number of records to return	Search across all spreadsheet data of all users. Returns search results using custom snippets with highlighting.
GET	/spreadsheet/{user}/search	rs:q query string leading and trailing wildcards constraints: dname row col cell rs:start starting record rs:pageLength number of records to return	Search across all spreadsheets of a specific user. Returns search results using custom snippets with highlighting.
GET	/spreadsheet/{user}/{docid}/search	rs:q query string leading and trailing wildcards constraints: dname row col cell rs:start starting record rs:pageLength number of records to return	Search within specific spreadsheet data. Returns search results using custom snippets with highlighting.
Spreadsheet Aggregate API
GET	/import	rs:q query string Uses rich search grammar. Examples: dname:ThisYr OR dname:AMOUNT*4 rnValue:?77 rs:txid txid (optional)	Search all spreadsheet data for items using query string. Returns aggregate results that can be imported into an Excel spreadsheet.
GET	/aggregate	rs:field field name rs:user user (optional) rs:txid txid (optional)	Search all spreadsheet data for items using the specifed "named field". Returns values associated with the "named field". It also returns the sum and average of the values. If no user parameter is specfied then the system will widen the search scope to search all spreadsheet data.
GET	/fields /fields/{$user} /fields/{$user}/{$uri}	rs:uri spreadsheet uri (optional) rs:user user (optional) rs:txid txid (optional)	Returns the list of "named fields" being used within spreadsheets. If no user or uri parameter is provided then search scope widens to search all spreadsheets for all users. If no uri parameter is specfied then the search scope widens to search across multiple spreadsheets for the specific user.
Spreadsheet Management APIs
GET	/spreadsheet/{uri}	uri Is tail of the request URL. rs:txid transaction-id is optional.	Get the Excel data and file associated with the specific URI.
GET	/spreadsheet/{uri}/file	uri uri is embedded in request URL rs:txid transaction-id is optional.	Get the Excel file associatated with the specific URI.
GET	/spreadsheet/{uri}/data	uri uri is embedded in request URL rs:txid transaction-id is optional.	Get the Excel data associatated with the specific URI.
PUT	/spreadsheet/{uri}	uri Is tail of request URL rs:txid transaction-id is optional. payload format is tbd	Updates data and file of a specfic spreadsheet.
PUT	/spreadsheet/{uri}/file	uri Is tail of request URL rs:txid transaction-id is optional. payload format is tbd	Updates file of a specfic spreadsheet.
PUT	/spreadsheet/{uri}/data	uri Is tail of request URL rs:txid transaction-id is optional. payload format is tbd	Updates data of a specfic spreadsheet.
POST	/spreadsheet	rs:txid transaction-id is optional. payload format is tbd	Creates new spreadsheet set (file with data). Data is stored separately for richer search capability. The xlsx or xls file is also stored for version history/audit purposes.
DELETE	/spreadsheet/{uri}	uri Is tail of request URL. rs:txid transaction-id is optional.	Deletes spreadsheet file and data.
DELETE	/spreadsheet/{uri}/file	uri Is tail of request URL. rs:txid transaction-id is optional.	Deletes spreadsheet file.
DELETE	/spreadsheet/{uri}/data	uri Is tail of request URL. rs:txid transaction-id is optional.	Deletes spreadsheet data.
Document APIs
GET	/document	rs:uri Is needed to retrieve specific document. rs:txid transaction-id is optional	Returns document
PUT	/document	rs:uri Is needed to target specific document. payload Requires XML payload. JSON payload to be added later. rs:txid transaction-id is optional	Updates document using payload
POST	/document	rs:txid transaction-id is optional payload Requires XML payload. JSON payload to be added later.	Creates new document using XML payload and returns success or fail status.
DELETE	/document	rs:uri Is needed to target specific document. rs:txid transaction-id is optional	Deletes document
Transaction APIs
GET	/transactions	No uri parameter or payload.	Returns transaction status for all transactions in the system.
POST	/transactions	rs:txid (optional) rs:result=(commit|rollback) (optional)	If no rs:txid is given then a new transactional context is created with new txid. If an rs:txid and rs:result is provided then transaction is either committed or rolled back. It will also discard the transactional context so that the txid is not longer valid.
Audit APIs
GET	/audit	tbd	tbd
PUT	/audit	tbd	tbd
POST	/audit	tbd	tbd
DELETE	/audit	tbd	tbd

Deploy

• $ ml local bootstrap
• $ ml local modules
• $ ml local deploy content

Content deploy copies zip file to database. Need to run the following snippet in qconsole to inflate data.

declare namespace zip="xdmp:zip";

let $_ :=
  for $x in xdmp:zip-manifest(fn:doc("/files.zip"))//zip:part/text()
    let $doc :=
      xdmp:zip-get(fn:doc("/files.zip"), $x,
        
          xml
        
      )
  
  let $uri := fn:replace($x, "test/", "/glm/")
    return
      xdmp:document-insert($uri, $doc)

let $_ := xdmp:document-delete("/files.zip")

return "done"

Roxy

Table of Contents

• Overview
• Features
• Getting Help
• Requirements
• Quick Start

Overview

Roxy is a utility for configuring and deploying MarkLogic applications. Using Roxy you can define your app servers, databases, forests, groups, tasks, etc in local configuration files. Roxy can then remotely create, update, and remove those settings from the command line.

Features

Cross Platform

Roxy runs on any platform that runs Ruby. We currently test on Mac, Linux, and Windows.

Multiple Environments

Roxy supports multiple deployment environments. You can define your own or use the default environments: local, dev, and prod. Each environment can have different settings which are specified in properties files or xml config files.

Easily Create and Deploy REST Extensions

Roxy provides scaffolding for creating REST extensions, transforms, etc. Once you have writtern your REST extension Roxy makes deploying to the server a breeze.

Capture Existing MarkLogic Settings

Whether it's a legacy application or you just prefer to configure your application using the Admin UI, Roxy can capture existing MarkLogic settings so that you can use them in your application. This feature is great for backing up Legacy Servers. Once the configurations are in Roxy you can then deploy to other servers.

Backwards Compatible

Roxy works with all supported versions of MarkLogic server out of the box.

Customization

Roxy is written in Ruby. Simply by editing the app_specific.rb file you can enhance, override, or replace the default functionality.

Run as a Java Jar

If you work in an environment where installing Ruby is not an option you can run Roxy as a self contained jar file which embeds JRuby.

Getting Help

To get help with Roxy,

• Subscribe to the Roxy mailing list
• Read up on the wiki
• Check out the Tutorials page
• For Command line usage run:
  $ ml -h

Requirements

• A supported version of MarkLogic
• Ruby 1.9.3 or greater
• Java (jdk) Only if you wish to run the mlcp, [XQSync](http://developer.marklogic.com/code/xqsync, XQSync), or RecordLoader commands.
• Git - Required to create a new project using "ml new".

Quick Start

This section describes the quickest way to get started using Roxy.

Assumptions

• You already have one or more MarkLogic Servers running somewhere that you can access from your computer. If not, get it here.*
• You know the admin logon to your MarkLogic Server(s)

Get Roxy

Use one of these three options to get started.

Using git

You can download Roxy using git $ git clone git://github.com/marklogic/roxy.git

Grab an zipped version

If you prefer to grab the archive simply download the latest release from our Releases Page

Install the Shell script or Batch File

Roxy comes with a script that you can put in your path. This file will create new projects for you by by issuing the $ ml new command. Grab one of these files and put it in a folder in your PATH. Note: In order for $ ml new to work you need to have git installed.

Windows

Download the ml.bat file

Mac/Linux

Download the ml file

Configure your application

1. Open a command prompt in the root folder of Roxy.
2. Run ml init to create sample configuration files.
   You must specify the --server-version option with a value of 4, 5, 6 or 7.
   You must specify the --app-type with a value or bare, rest, hybrid, or mvc.

$ ml init app-name --server-version=7 --app-type=rest 3. Modify deploy/build.properties with your application's settings.

# Username to authenticate to ML
user=your-ml-admin-username

# password for ML authentication
#
# leave this blank to be prompted for your password
#
password=

# the authentication type for the appserver (digest|basic|application-level)
authentication-method=application-level

# the default user to authenticate with. defaults to nobody
default-user=${app-name}-user

# Specify the server(s) you wish to deploy to here. This tutorial assumes you are using localhost.
local-server=localhost
#dev-server=
#prod-server=

Configure MarkLogic Server

This step is only needed when database configurations have changed or on a fresh install. In most cases you will not need to restart your server.

1. Open a command prompt in the root folder of Roxy.
   If your server is not configured as local-server in build.properties then substitute your environment here ( local | dev | prod )
2. $ ml local bootstrap
3. Depending on what changed you may need to restart MarkLogic in order to proceed. If you see output telling you to restart...
   $ ml local restart

Deploying Code

This step describes how to deploy your Roxy application into your MarkLogic Server modules database. If you have elected to run your code locally out of the filesystem you do not need to do this.

1. Open a command prompt in the root folder of Roxy
2. $ ml local deploy modules

Congratulations

Congratulations! You have Roxy running on your server. Now you need to start customizing it.