cl-mongo

Intro

mongo is a scalable, high-performance, open source, schema-free, document-oriented database..

This is a common lisp interface that allows you to work with a mongo document database. This is not a driver (yet) as specified in the mongo documents; There is still some functionality I need to implement.

Except for gridfs most features have been implemented.

cl-mongo provides the ability to insert, update and delete documents, indexing, searching using regexs etc. In addition it supports javascript in the repl (using a commonly available lisp to javascript compiler). This allows you to do use mongodb's map-reduce from the lisp repl.

I developed this primarily using sbcl and to some extend clozure common lisp on the linux and mac osx platforms.

This should also work on clisp and allegro common lisp and windows.

multi-threading

The do-query function uses multithreading to process the results of a batch as a query is in progress. This obviously requires that multi-threading is supported. I've found that as of now (2010-10-23) sbcl on mac osx doesn't support multithreading out of the box. So you'd need to compile a new sbcl with multithreading enabled, which is easy to do.

Version

Version 0.9.0

I've added regression testing for all features currently supported by cl-mongo.

Installation

Use asdf to install cl-mongo. cl-mongo depends on many other lisp libraries (which in turn have their own dependecies). asdf is not a dependency manager, so you would need to install all dependencies as they come up as unavailable when you try to install cl-mongo.

On the other hand, cl-mongo is included in quicklisp, which should make life easier.

Testing

The cl-mongo-test package contains regression tests for all the features currently supported. cl-mongo-test is asdf-installable, but does not export it's tests. To run the quick test, do this : (use-package :cl-mongo-test) (quick-test)

Quick test will connected to a locally running mongodb instance. It uses the "foo" collection in the "test" database.

A sample session

This connects to the test database on the local mongo server listening on its default port.

(use-package :cl-mongo)
(db.use "test")

Insert a key-value pair as the first document into collection "foo".

(db.insert "foo" (kv "document" "one") )

Pretty-print the documents returned by the find command. iter will ensure that the cursor is fully iterated over.

(pp (iter (db.find "foo" :all)))`

 {
    "_id" -> objectid(4B5CF28970DFF196A75FE1F0)
    "document"  ->  one
 }

Create a document. A document is collection of key-value pairs and a unique identifier called "_id".

(defvar *DOC* (make-document))

Add various elements to the document.

   (add-element "tag" "key" *DOC*)`


   {  DOCUMENT 

   {
   tag  -> key 
   }
   }


(add-element "array" (list 1 2 3 "hello") *DOC*)

{  DOCUMENT 

{
tag  -> key 
	 1 
	 2 
	 3 
	 hello 
 array  -> NIL 
 }
 }

Insert document into the database.

(db.insert "foo" *DOC*)

Print the current contents.

   (pp (iter (db.find "foo" 'all)))

   {
   "_id" -> objectid(4B5CF28970DFF196A75FE1F0)
   "document"  ->  one
   }

   {
   "_id" -> objectid(8B508D5CBB5D451D961F046D)
   "array"  -> [ 1, 2, 3, hello,]
   "tag"  ->  key
   }

Bind variable *DOC* to the second document returned by the find command, add an other element and save back to the collection.

  (defvar *DOC* (cadr (docs (db.find "foo" :all))))`
  (add-element "tags" (list 'good 'bad 'ugly) *DOC*)
  (db.save "foo" *DOC*)
  (pp (db.find "foo" :all))

  {
"_id" -> objectid(4B5CF28970DFF196A75FE1F0)
  	"document"  ->  one
  }

  {
    "_id" -> objectid(8B508D5CBB5D451D961F046D)
  	"tags"  -> [ GOOD, BAD, UGLY,]
 	"tag"  ->  key
 	"array"  -> [ 1, 2, 3, hello,]
  }

Check the status of the server.

  (db.use "admin")
  (nd (db.run-command :serverstatus))

    "ok"  ->  1.0d0
  "mem"  -> 
    "mapped"  ->  80
    "virtual"  ->  177
        "resident"  ->  3
    "globalLock"  -> 
      "ratio"  ->  4.644753171959394d-5
      "lockTime"  ->  6493354.0d0
    "totalTime"  ->  1.39799764586d11
    "uptime"  ->  139799.0d0

$ syntax

I've added various $.. macros which allow for a more declarative programming style. In stead of doing something like :

(db.find "foo" (kv "k" (kv "$lte" 3)))

you can use :

(db.find "foo" ($<= "k" 3))

To declare a unique ascending index on "k" in collection "foo" , you can say :

($index "foo" :unique :asc "k")

What's missing

At least the following is missing :

Request id/ Response id are left 0 in the header.
GridFS
......

API documentation

I use Edi Weitz's documentation template to generate an api description based on embedded comments.

News

2010-10-23 I've improved the read performance several orders of magnitude. This also reduces the memory foot pront quite a bit.

In addition I've added a multi-threaded reader called do-query.

CL-MONGO - api reference

Abstract

The code comes with a MIT-style license so you can basically do with it whatever you want.

Download shortcut: http://github.com/fons/cl-mongo.

Download
The CL-MONGO dictionary
Acknowledgements

Download

CL-MONGO together with this documentation can be downloaded from http://github.com/fons/cl-mongo. The current version is 0.1.0.

The CL-MONGO dictionary

[Macro]
$ &rest args => result

[Macro]
$!= &rest args => result

[Macro]
$!in &rest args => result

[Macro]
$+ arg &rest args => result

[Macro]
$- arg &rest args => result

[Macro]
$/ regex options => result

[Macro]
$< &rest args => result

[Macro]
$<= &rest args => result

[Macro]
$> &rest args => result

[Macro]
$>= &rest args => result

[Macro]
$add-to-set &rest args => result

[Macro]
$all &rest args => result

[Macro]
$em array &rest args => result

[Macro]
$exists &rest args => result

[Macro]
$in &rest args => result

[Macro]
$inc &rest args => result

[Macro]
$index collection &rest args => result

[Macro]
$map-reduce collection map reduce &key query limit out keeptemp finalize verbose => result

Run map reduce on the mongo server. map and reduce are either the names of the javascript functions, created with defjs or defsrvjs or are function definitions in javascript. The keywords refer to option available for map reduce in mongo. This returns a result summary document. When using :keeptemp t without specificing :out the collection is mr.<collection>

[Macro]
$mod &rest args => result

[Macro]
$not &rest args => result

[Macro]
$pop-back &rest args => result

[Macro]
$pop-front &rest args => result

[Macro]
$pull &rest args => result

[Macro]
$pull-all &rest args => result

[Macro]
$push &rest args => result

[Macro]
$push-all &rest args => result

[Macro]
$set &rest args => result

[Macro]
$unset &rest args => result

[Macro]
$where &rest args => result

[Special variable]
*mongo-default-db*

database opened by the default connection

[Special variable]
*mongo-default-host*

host for the default connection.

[Special variable]
*mongo-default-port*

port for the default connection.

[Special variable]
*repo-root*

root of the repository; used for documentation generation

[Generic function]
add-element key value document => result

add element with key and value to a document

[Function]
cwd &key mongo => result

Show the current database.

[Function]
date-time second minute hour day month year &optional time-zone => result

Generate a time stamp the mongo/bson protocol understands.

[Generic function]
db.add-user username password &key mongo readonly => result

Add a user to the database.

[Generic function]
db.auth username password &key mongo => result

authenticate a user with a password

[Generic function]
db.collections &key mongo => result

[Function]
db.collections &key mongo => result

Show all the collections in the current database.

[Generic function]
db.count collection selector &key mongo => result

Count all the collections satifying the criterion set by the selector. :all can be used to return a count of all the documents in the collection.

[Generic function]
db.create-collection collection &key => result

create a collection

[Generic function]
db.delete collection object &key mongo => result

Delete a document from a collection. The document field is used to identify the document to be deleted.
You can enter a list of documents. In that the server will be contacted to delete each one of these. It may be more efficient to run a delete script on the server side.

[Generic function]
db.distinct collection key &key mongo => result

Return all the distinct values of this key in the collection

[Generic function]
db.ensure-index collection keys &key drop-duplicates unique mongo asc => result

Create an index specified by the keys in a collection

[Generic function]
db.eval code &rest rest => result

run javascript code server side

[Generic function]
db.find collection kv &key selector limit skip options mongo => result

Find documents in the collection using the selector specified by kv.
Methods take two keywords. ':limit' sets the maximum number of documents returned. The default is 1. ':skip' sets the number of documents to skip in this query. It's default is 0. Since the default value of the limit is one, db.find by default is the equivalant of findOne in the mongo documentation.

[Generic function]
db.indexes &key mongo => result

[Function]
db.indexes &key mongo => result

Return all indexes in the database.

[Generic function]
db.insert collection document &key mongo => result

Insert a document in a collection. A document is typically generated by (make-document), but it can also be a hash table, a key-value pair or kv list (see the kv functions).

[Generic function]
db.iter result &key limit mongo => result

next document iteration

[Generic function]
db.next collection cursor-id &key limit mongo => result

Executes the next call on the iterator identified by cursor-id.

[Generic function]
db.run-command cmd &key arg mongo collection index => result

Run a database command on the server. See the mongo documentation for a list of commands.
For most commands you can just uses the key-value shown in the mongo documentation.

[Generic function]
db.save collection document &key mongo => result

Save a document to the collection. If the document has a unique _id value (i.e. if it's generated by (make-document) ) it will be 'upserted' (that is: it will be inserted if the document doesn't exist). If the document a hash table or a kv set, it will be inserted.
In other words this a a helper-function build around db.insert and db.update.

[Macro]
db.sort collection query &rest args => result

sort macro : Takes the same arguments and keywords as db.find but converts the query so it works as a sort. use the :field keyword to select the field to sort on. Set :asc to nil to reverse the sort order

[Generic function]
db.stop cursor &key mongo => result

Stop iterating and clean up the iterator on the server by making a server call.

[Generic function]
db.update collection selector new-document &key mongo upsert multi => result

In a collection update the document(s) identified by the selector statement.
This method has two keywords. ':upsert' : If t insert the document if the document cannot be found in the collection. ':multi' : Update all documents identified by the selector.

[Generic function]
db.use db &key mongo => result

Use a database on the mongo server. Opens a connection if one isn't already established. (db.use -) can be used to go to a previosuly visited database, similar to cd -.

[Macro]
defjs name args declaration* statement* => result

Define client side javascript. Works like defun; body is in lisp, with parenscript additions, like return. So (defjs hello (x y) (return (+ x y))) defines an adder. macro creates a lisp function which sends the javascript function over to the mongo server to be evaluated. Result is processed and returned to the reader. This will execute 10 times on the server : (mapcar (lambda (x) (hello 10 x)) (list 1 2 3 4 5 6 7 8 9 10))

[Macro]
defsrvjs name args declaration* statement* => result

Creates a function which stores and executes javascript on the server. The first time the function is called the javascript function is stored on the server. Subsequent calls will call out to the server. Works like defun; the function body is defined in lisp, with parenscript additions. Since the body of the function already resides on the server this should have less impact on the network. Use :install t to reinstall.

[Function]
do-query coll &key map-fn reduce-fn initial-value query mongo limit selector => result

Performs a multi-threaded query on a mongo database. coll is the collection name. The reduce-fn keyword is used to specify a function which will be called for each member of a batch of data returned from mongodb.
The reduce-fn function is executed while the query for the next batch is in progress. The default for reduce-fn is the identity function. The reduction keyword is used to specify a function which is executed when the database queries have finished. It's default implementation is to return a hash table, with the mongodb id as the hash key. The query, mongo, limit and selector keywords are used in the same way as for db.find.

[Function]
doc-id doc => result

return the unique document id

[Function]
docs result => result

Stop the iterator (if any) and return the list of documents returned by the query. Typical ue would be in conjunction with db.find like so (docs (iter (db.find 'foo' 'll)))

[Standard class]
document

document Document class. A document consists of key/value pairs stored in a internal hash table plus an internally generated unique id.
Accessors are : 'elements' which returns the internal hash table; '_id' which returns the unique id and '_local_id' which if true means that the document was generated by the client (as opposed to having been read from the server).

[Function]
generate-readme &key path => result

This function generates a README.md file with the latest api description. The :path keyword specifies the location. It expects a sub-directory <path>/doc. Api documentation is generated on the fly, by extracting comments from the classes, generics and fuctions exported in the packages file. The resulting file is <path>/doc/index.html. <path>/README.md is generated by
appending the api documentation to <path>/doc/readme-base.md. :path or REPO-ROOT are typically set to the root of the repository.

[Generic function]
get-element key document => result

Get an element identified by key from the document.

[Function]
ht->document ht => result

Convert a hash-table to a document.

[Macro]
install-js name => result

Allows server based javascripts the be installed without being run.

[Function]
iter result &key mongo max-per-call => result

Exhaustively iterate through a query. The maximum number of responses per query can be specified using the max-per-call keyword.

[Macro]
jsdef name => result

Return the body of the javascript function; otherwise nill.

[Generic function]
kv a &rest rest => result

This a helper function for key-value pairs and sets of key-value pairs.
In a key-value pair like (kv key value) the key has to be a string and the value something which is serializable. key-value pairs can be combined using kv as well : (kv (kv key1 val1) (kv key2 val2)). This combination of key-value pairs is equivalent to a document without a unique id. The server will assign a unique is if a list of key-value pairs is saved.

[Function]
make-document &key oid size => result

Constructor. key ':oid' is a user supplied unique id. An internal id will be generated if none is supplied.

[Function]
mapdoc fn document => result

[Standard class]
mongo

Encapsulates the connection to the mongo database. Each connection is a added to a global registry.

[Generic function]
mongo &key host port db name host port db name => result

This method returns the connection referred to by the name identifier from the connection registry. The connection name is unique. If no connection with that name exists, a new connection with the supplied or default host, port and db parameters will be created. The default host is localhost; the default port is 27017; the default db is admin.

[Generic function]
mongo-close name => result

Close the connection to the mongo database. The name should uniquely identify the connection to close. This is either a mongo object or the name the object is bound to in the connection registry. To close all open connections use the special symbol 'all

[Generic function]
mongo-registered name => result

Return a conection registered by this name or nil..

[Function]
mongo-show => result

Show all registered connections and their session id

[Generic function]
mongo-swap left right => result

Swap the names of the left and right connections. Typical use would be (mongo-swap :default :alt). After the function call :default will refer to the connection previously referred to as :alt. A connection named :default is returned by (mongo) and is the default used in the api. The connections are returned in the order they were passed in (but with the names swapped between them). To re-open a connection you can say (mongo-close (mongo-swap :default (mongo :host <newhost> :portid <portid> :name :temp))) and a new default connection is registered.

[Function]
mr.gc &key mongo => result

remove the temporary collections created by map-reduce

[Function]
mr.gc.all &key mongo => result

remove the all collections created by map-reduce, temporary as well as permanent

[Function]
mr.p results => result

show the contents of the results collection map-reduce created

[Function]
nd result &key stream => result

Pretty-print for non-document responses, like the response to a database command.

[Function]
now => result

Return the current date and time in bson format.

[Function]
nwd => result

Show the database set by the (db.use -) command

[Generic function]
pp result &key nd stream => result

Pretty-print the results returned from a query. To be used on the repl. This will format each server reply as if it were a document. This is obviously ok in mosty cases. See nd for an alternative.

[Macro]
remove-js name => result

[Function]
ret result => result

return value bound to retval in a return document. Used for db.count, db.distinct, functions etc..

[Function]
rm collection query &key mongo => result

Delete all the documents returned by a query. This is not an efficient way of deleting documents as it invloves multiple trips to the server. Mongo allows for execution of java-script on the server side, which provides an alternative. Typical use would be (rm (iter (db.find 'foo' (kv 'key' 1)))), which deletes all documents in foo, with field key equal to 1.

[Generic function]
rm-element key document => result

Remove element identified by key from a document

[Generic function]
show things &key msg nl order => result

Print a list of things. Things can be users, databases, collections in the current database, the profile and more. Things is a keyword so (show :users) will show all users.

[Function]
time-zone => result

Set the time zone appropriate for the current environment.

[Macro]
with-mongo-connection args &rest body => result

Creates a connection to a mongodb, makes it the default connection and evaluates the body form. args uses the same keyword set as mongo (:db :localhost :port) args is passed on to make-mongo when the connection is created.

Acknowledgements

This documentation was prepared with DOCUMENTATION-TEMPLATE.

$Header: /usr/local/cvsrep/documentation-template/output.lisp,v 1.14 2008/05/29 08:23:37 edi Exp $

BACK TO MY HOMEPAGE

microbial/cl-mongo