Remembrance Agent, version 2.12 Copyright (C) 1996-2003 Massachusetts Institute of Technology and Bradley Rhodes Released under the GNU General Public License (GPL) ---------------- Thank you for using the RA. If you have questions or comments, please send mail to ra-bugs@remem.org (or ra-maintainer@remem.org). If you would like to be added to an announcement mailing list for new versions (VERY low traffic) send mail to ra-announce-request@remem.org. Contents -------- 0. Introduction 1. Building the Remembrance Agent 2. Creating a database from a collection of text 3. Using the Remembrance Agent in Emacs 4. Commands (using the RA) 6. Savant's document template system and other customizations 7. What's New? 8. License information 0. Introduction ---------------- The Remembrance Agent is one of the projects being developed by the MIT Media Lab's software agents group. Given a collection of the user's accumulated email, usenet news articles, papers, saved HTML files and other text notes, it attempts to find those documents which are most relevant to the user's current context. That is, it searches this collection of text for the documents which bear the highest word-for-word similarity to the text the user is currently editing, in the hope that they will also bear high conceptual similarity and thus be useful to the user's current work. These suggestions are continuously displayed in a small buffer at the bottom of the user's emacs buffer. If a suggestion looks useful, the full text can be retrieved with a single command. The Remembrance Agent works in two stages. First, the user's collection of text documents is indexed into a database saved in a vector format. After the database is created, the other stage of the Remembrance Agent is run from emacs, where it periodically takes a sample of text from the working buffer and finds those documents from the collection that are most similar. It summarizes the top documents in a small emacs window and allows you to retrieve the entire text of any one with a keystroke. The Remembrance Agent is the PhD work of Bradley Rhodes, and includes long hours of coding by an great team of undergraduates. They can be reached with questions and comments about the Remembrance Agent via email at ra-bugs@remem.org, and new versions of the software can be found at http://www.remem.org/ Thanks for using the Remembrance Agent, and I hope you find it useful. 1. Building the Remembrance Agent ---------------------------------- The RA back-end has two executables, "ra-index" and "ra-retrieve", which together make the system called Savant. Since you are reading this file, I assume you've already uncompressed and untarred the RA distribution. To build Savant, cd to the RA directory type: ./configure; make This should analyze your system, and then make appropriate binaries of ra-index and ra-retrieve. If you have trouble, make sure you are using the GNU version of make ("make --version" should produce something sensible). It might also be called "gmake" on your system. Once the compilation is finished, the code you will need is: main/ra-index: Binary file for indexing documents main/ra-retrieve: Binary file for retrieving documents other/remem.el: Emacs interface to Savant Move ra-index and ra-retrieve to wherever you normally keep executables. Copy remem.el to wherever you keep emacs-lisp files. If you don't have a place for elisp files, you can just create a directory called "elisp" off of your main directory and put them there. You can then delete the source files if you like. 2. Creating a database from a collection of text ------------------------------------------------- Savant has two executables, one which indexes documents into databases, and one which performs interactive retrievals from these databases. To index, you must have a set of source text-files, and a directory Savant can put database files into. Usage: ra-index [-v] [-d] [-s] <base-dir> <source1> [<source2>] ... [-e <excludee1> [<excludee2>] ...] The <source> arguments may be files or directories. If a directory is in the list, Savant will use all its contents, recursing into all subdirectories. Non-text files and backup files (those appended with ~ or prepended with #) are ignored. It also ignores dot-files (those starting with .) and unless you specify the "-s" flag it will also ignore symbolic links. Any files or directories specified after the optional -e flag will be excluded. Savant will use any files it finds to create a database in the specified base directory, which must already exist. The optional -v argument (verbose) will direct Savant to keep you updated on its progress. So for example, ra-index -v ~/RA-indexes/mail ~/RMAIL ~/Rmail-files -e ~/Rmail-files/Old-files will build a database in the ~/RA-indexes/mail directory, made up of emails from my RMAIL file plus all files and subdirectories of ~/Rmail-files, excluding files and directories in ~/Rmail-files/Old-files. The optional -d flag turns on debugging information. ***IMPORTANT***: Savant can build databases in any directory you like, but the emacs interface for the Remembrance Agent expects a particular structure. For each database you want to make, you should create a directory, and all these directories should live in the same parent directory. For example, for my own use I have a directory ~/RA-indexes/, and within that are the directories ~/RA-indexes/mail/, ~/RA-indexes/papers/, etc. which actually contain the database files. To see how Savant interacts with emacs while the remembrance agent is running, try running ra-retrieve with the command 'ra-retrieve -v <base-dir>' after creating a database using index. You can use ra-retrieve as a simple search engine, but usually ra-retrieve is only called by the emacs front-end. 3. Using the Remembrance Agent in Emacs ---------------------------------------- You can load the Remembrance Agent automatically every time you run emacs by adding this line to your .emacs file in your homedir: (load "remem.el") This assumes that you put remem.el into a directory that is in your emacs load-path. To set that loadpath to include the directory ~/elisp/, you can use the line: (setq load-path (append (list (expand-file-name "~/elisp")) load-path)) Before the Remembrance Agent can be used, several variables must be configured. These are listed at the top of remem.el, and are documented there as well. You will either want to set these in your .emacs file or create a file called "remem-custom.el" and load that file after loading remem.el. You probably don't want to set them in remem.el, unless you are the only person using this copy of the remembrance agent. (setq remem-prog-dir <prog-dir-string>) (setq remem-database-dir <database-dir-string>) (setq remem-scopes-list <scope1, scope2, ...>) <prog-dir-string> should be the full name of the directory where you put the ra-retrieve executable, enclosed in double quotes. This defaults to "/usr/local/bin". On some systems, you need to make sure not to end the directory with a slash ("/"). <remem-database-dir> should be the full name of the directory which holds your database directories, enclosed in double quotes. Note that this is the name of a directory containing directories, not the directory containing the database files themselves. This defaults to "~/RA-indexes". On some systems, you need to make sure not to end the directory with a slash ("/"). remem-scopes-list: you can display several different "scopes," all with separate processes performing retrievals simultaneously. Scopes can have different databases, update frequencies, and ranges of words it uses for a "query." Each scope looks like this: (DIRN NUM-LINES UPDATE-TIME QUERY-RANGE) DIRN is the subdirectory of remem-database-dir with the desired database (This is the name of a sub-directory in remem-database-dir) NUM-LINES is the number of lines that you want the scope to show (initially) UPDATE-TIME is the time between scope updates (in seconds) QUERY-RANGE number of words around your cursor that you want the scope to query on For example, my own customization shows this: (setq remem-scopes-list '(("my-email" 6 5 500) ("my-notes" 2 10 500))) This sets up two scopes. The first displays six suggestions from the index directory "/home/rhodes/RA-indexes/my-email/", updated every five seconds, based on the last 500 words around my cursor (where a word is considered 5 characters, so that's really just the last 500 X 5 = 2500 characters). The second scope displays two lines from "/home/rhodes/RA-indexes/my-email/", updated every ten seconds, also based on 2500 characters. The rest of the customizations set colors, logging, display of suggestions, display thresholds and the prefix for RA commands. The default colors are set for a light background, but shift if you set hilit-background-mode to 'dark. They're all documented in remem.el -- play around with them. Okay! After these customizations are made, you can start the Remembrance Agent by typing C-c r t. It will create its window and after a moment or two begin to display suggestions like: 1 + rhodes RA presence paper stuff 04/14/97 papers suggestions, ra 2 tara Questions digest for IA class 09/15/97 agents-class remembrance, agent 3 ++ rhodes RA Projects Internal 03/09/99 ra-bugs-archive window, display This can be summarized as <Line#> <rating> <author or file owner> <subject> <date> <filename> <keywords> The rating is a measure from zero to two plus signs (0 to 1 if you set remem-print-exact-relevance-p to t) and indicate how relevant the document is to a sample of your current buffer. If you have remem-print-even-bad-relevance-p set to t then suggestions below the minimum threshold will print with a "-" sign, otherwise they'll show up as "No Sugestion." To see a suggested document, type C-c r <ID# of document>. The keywords are the top five keywords that contributed to making this suggestion, in order. 4. Commands (using the RA) --------------------------- C-c r t (Control-c r t): Toggle Remem Toggle the remembrance agent on and off. You can also do this by evaluating the function (remem-toggle). C-c r v (Control-c r v): View Updates the suggestion list by running queries in all scopes now. Bypasses the timer. C-c r # (Control-c r <number>): Show Suggestion Bring up the full text of a suggested document. If you set remem-load-original-suggestion to t (the default), then the full file is loaded. Otherwise a copy is displayed. If your files are big and contain lots of documents (e.g. large mail archives), or if you don't want the buffers lying around then you might want to set this to nil. If you set remem-non-r-number-keys to t, the keys C-c # will also work, thus saving you a keystroke. C-c r r # (Control-c r r <number>): Rate Rate the last suggestion. This allows you to rate the last suggestion followed with a number from 1 (bad suggestion) to 5 (great suggestion). The rating information is currently used by me to evaluate the software and figure out how to improve it, though eventually it may be used to give feedback for a machine learning system that will improve future suggestions. C-c r f (Control-c r f): Field Search Perform a search on a particular field (Body, Location, Date, Subject, or Person). C-c r q (Control-c r q): Query Bring up a query buffer to do a manual search. This allows you to fill in several fields. When the data is entered, hit C-c r v to view the results. C-c r d (Control-c r d): Database change Prompt for a new database, from the indexes in remem-database-dir. If more than one scope is active, it will prompt for which scope to replace as well. Remem will restart with the new database(s) at the original size. Left mouse-click on lineno: Show Suggestion Clicking the middle mouse-button on a line number is the same as doing C-c r # for that number. Left mouse-click on a field: Search field Clicking the middle mouse button on a field does a search on that field. For example, if I middle-click on the subject field "RA presence paper stuff", a special query will be launched just on those words in other subject fields. This is field dependent -- if I middle click on the username "tara", it does a query for other documents associated with her username. Clicking on a date will work as soon as we start indexing dates, but it doesn't yet. Middle or Right mouse-click: Keywords Clicking with the middle or right mouse button pops up keywords associated with this particular suggestion. These are the top five words that led to the document being suggested, in order that they contributed. These words will also be on your display line if you set remem-terminal-mode to t. Resize window: Resize If you resize the *remem-display* window it will fill to the new size on the next update. If you resize it too small, it will stop the RA (same as performing a remem-toggle). 5. Logging ----------- Because this is a research project, the emacs front-end logs generic data such as how often queries are generated, how many suggestions are followed, and how you rate the usefulness of followed suggestions. Logging is on by default, to turn off logging add the following to your .emacs or remem-custom.el file: (setq remem-log-p nil) All logs are kept in the file ~/.remem-log-file, or wherever specified by the remem-logfile variable. Logs are kept local, though I will be sending out email asking for a copy of log-files so I can evaluate how the RA gets used. Such participation is voluntary, and I encourage you to use the RA even if you don't keep logging info. Here's the complete list of information that gets logged: RA Version number timestamp Number of documents indexed per scope Scope name (e.g. "notes") Number of unique suggestions proposed Number of lines per scope displayed Ratings of followed suggestions Number of suggestions followed, their relevance scores (broken down by field type), docnums, and whether it was an automatic, mouse, or manual query No information about the content of documents or suggestions is kept. 6. Savant's document template system and other customizations -------------------------------------------------------------- As of Savant v.2.00, the template structure is compiled instead of set in a file. It is much more powerful than previous versions and allows for Perl-style regular expressions to pull out fields, do filtering, etc. This still has to be documented. It's pretty cool though :). Feel free to check the comments in the source-code -- they're pretty good. All the template stuff happens in RA/templates/conftemplates.c. 7. What's New? --------------- This is release v2.0, which is a huge rewrite. Here are some changes: Back-end (Savant): Redesigned from the ground up and is much cleaner now. Now uses the Okapi algorithm for text Template-defined filters for different data types (e.g. simple heuristics to remove sigs from email) Templates to recognize different queries Setable biases (weights) for fields of different document and query types More robust Front-end (Emacs RA): Redesigned from the ground up and is much cleaner now. Pretty Colors! (And useful too -- they separate fields) Mouseable queries for different fields View keywords feedback More robust See savant.h for a full changelog (yes, I really should break that out into a "changelog" file). 8. License ----------- Some files and libraries in this distribution carry their own copyright (notably the parsedate and zlib directories and libraries in savutil). In the future some plugins might carry different copyrights and licenses as well as people write their own. All other files and code is under the following copyright and license: All code included in versions up to and including 2.09: Copyright (C) 2001 Massachusetts Institute of Technology. All modifications subsequent to version 2.09 are copyright Bradley Rhodes or their respective authors. Developed by Bradley Rhodes at the Media Laboratory, MIT, Cambridge, Massachusetts, with support from British Telecom and Merrill Lynch. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. For commercial licensing under other terms, please consult the MIT Technology Licensing Office. This program may be subject to the following US and/or foreign patents (pending): "Method and Apparatus for Automated, Context-Dependent Retrieval of Information," MIT Case No. 7870TS. If any of these patents are granted, royalty-free license to use this and derivative programs under the GNU General Public License are hereby granted. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.