l3kn/org-zettelkasten

An opinionated setup for managing large collections of interlinked org files.

Org Zettelkasten

This package should be considered work in progress. I use it on a daily basis but the API regularly changes in breaking ways.

I still need to write proper setup instructions. For now, feel free to look around for pieces of code that might be useful to you.

Motivation

When managing large collections of org-files (> 1000), many of the default functions / modes (i.e. org-agenda) become very slow.

This package is an attempt at alleviating this problem by introducing a caching layer that is persisted to disk, and a set of command for interacting with / viewing this data.

The code for this cache can be found at l3kn/org-el-cache.

(Planned) features include:

• A command for finding files by their titles
• Commands for renaming files updating all links to them
• Commands for extracting a headline into a separate file (or doing the inverse)
• A hydra with a bunch of useful hotkeys
• A project view for listing files marked as GTD projects
• A graph view (prototype stage)
• A sidebar showing backlinks for the current file (prototype stage)
• A TODO list that can show tasks collected from hundreds of files (not working at the moment)
• Full-text search (not working at the moment)

Inspiration

David Allen - Getting Things Done

Lion Kimbro - How to Make a Complete Map of Every Thought You Think

Niklas Luhman - Zettelkasten

Similar Projects

• deft, Emacs mode for browsing, filtering & editing directories of files
• EFLS/zetteldeft, Zettelkasten system for deft
• jethrokuan/org-roam, Extensions to deft, graph view
• Kungsgeten/org-brain, concept mapping tool for org-mode, graph view
• alphapapa/org-graph-view, Link-graph view for org-mode
• alphapapa/org-ql, Query-language & views for org-mode

If you are not working with > 1000 org-mode files or if you have no yet run into any performance problems, one (or a combination of multiple) of the packages listed above might fit your needs better.

Limitations

org-zettelkasten uses a cache to enable fast searching of files and headlines.

This cache is updated each time a file is saved, so to ensure the information displayed in the views is up to date, files should be saved often.

I’m using bbatsov/super-save for that.

A big limitation when working with cached headlines is that we can’t use markers (like org-agenda does) to jump to a headline in a file, as these can’t be persisted to disk.

Instead, headline IDs are used for that.

This means that every headline that should be accessible through one of the views must have an ID.

There are two functions to make this easier:

org-zk-add-ids-to-buffer adds IDs to all task-headlines in the current buffer.

I’m using a hook to do this each time a file is saved.

(add-hook
 'org-mode-hook
 (lambda ()
   (add-hook 'before-save-hook 'org-zk-add-ids-to-buffer nil 'local)))

If you want to use org-zettelkasten for an existing collection of files, you can use M-x org-zk-refactor-add-ids to add IDs to all files managed by org-zettelkasten.

Dependencies

• abo-abo/swiper For searching / selecting files
• abo-abo/ace-link For following links
• abo-abo/hydra

Collections

org-zk-collections is used to customize org-zk settings per directory. Entries should be of the form (:path ... :other-settings ...).

This is most useful for configuring the :setup-fn for a file.

Due to the way this list is searched when determining the category of a file, sub-directories should be listed before their parent directory.

Example

Here is the configuration I’m using for my personal setup:

(defun org-zk-website-setup-fn (title)
  (insert "#+SETUPFILE: ~/org/website/setup.org\n")
  (org-zk-default-setup-fn title))

(setq org-zk-collections
      '((:path "~/org/flashcards/" :name "fc")
        (:path "~/org/sandbox/" :name "sandbox")
        (:path "~/org/archives/" :name "archives" :ignore t)
        (:path "~/org/website/pages/" :name "pages" :setup-fn org-zk-website-setup-fn)
        (:path "~/org/website/" :name "website" :setup-fn org-zk-website-setup-fn)
        (:path "~/org/deft/" :name "zk")
        (:path "~/org/paper/" :name "paper")
        (:path "~/org/" :name "main")))

Alternative Implementations

This could be implemented using Directory Variables, however in that case, there is no one way to see the configuration for all categories and each variable that is set has to be declared non-risky.

Backlink View

A sidebar displaying all files linking to the file currently being viewed.

File View

The file view shows a list of all files included in org-zk. It can be used to quickly edit file metadata or find all files matching some query.

M-x org-zk-file-view or F from the org-zk-hydra.

Hotkeys

• RET to open a file
• k to edit a files keywords

Screenshot

Queries

• keyword:my-keyword or k:my-keyword
• category:my-category or c:my-keyword (uses org-zk categories, not the org-mode #+CATEGORY keyword)

Project View

Variant of the file view for managing GTD projects using the #+GTD_STATE keyword.

Based on this keyword, a view of all (active) projects can be created and it’s easy to mark a whole project as on_hold or someday to remove it’s tasks from the task view.

Task View

Simplified analog to the org-agenda that uses cached headlines.

Calendar View

Variant of the task view that takes into account repeated tasks and limits the display to tasks scheduled or due in the next n days.

git

I’m willing to sacrifice a bit of memory and meaningful commit messages to make sure no information / history is lost.

Files are committed to version control every hour.

Cache

There are a few other packages that provide more advanced / performant queries on org-mode files:

• org-ql
• org-sql

This package implements a cache for files and headlines using a hash table directly in Emacs, no external database is needed.

To avoid recreating this table on each startup, it can be written to disk at regular intervals and when Emacs is closed.

Each file entry has a hash value that is checked against the files hash on startup. With this, updating the cache for my collection of files takes around 5s.

Cache entries are updated when a .org file is saved, created, moved or deleted.

Derived Tasks

Some tasks can be fully derived from the current state of the rest of the system.

Examples:

• Reviewing due flashcards
• Processing the Inbox
• Reading mail

This could be implemented by skipping the detour through .org files on disk and adding derived tasks to the list when opening the task view.

The downside of this approach is that these tasks would not show up in the default org-agenda an keeping track of their time-tracking information would require an additional database.

Instead, derived tasks are implemented by checking for some condition in regular intervals, then adding a headline to a predefined file.

To avoid cluttering this file, if it already includes a headline with the same title that is not marked as “DONE”, no new entry is added.

Derived tasks are stored in a hash-table to simplify changing the predicate of an existing derived task.

Therefore, each derived task should have a unique task title.

Derived tasks can be registered using the (def-org-zk-derived-task title priority tags predicate) macro.

The example below adds a new task “Process Inbox” if there are at least five entries in the inbox.

(def-org-zk-derived-task "Process Inbox" "A" '("gtd")
  (>= (org-zk-inbox-count) 5))

Components

org-zk-keywords

File-level attributes are stored as #+KEY: value org mode keywords. These should be placed at the start of the file, not containing any newlines before or between keywords.

• (org-zk-keywords-set-or-add (key value))
• (org-zk-keywords-add (key value))
• (org-zk-keywords-delete (key))

The macro org-zk-def-keyword can be used to create commands to set keywords to one of a list of predefined values using ivy-read. When generating the functions name, the keyword is converted to lowercase and “_” are replaced by “-“.

Example

(org-zk-def-keyword
 "GTD_STATE"
 '("active"
   "someday"
   "planning"
   "cancelled"
   "done"))

Generated function: org-zk-set-gtd-state

org-zk-cache

By default, emacs is not fast enough to efficiently search large collections (>1k files) for TODO keywords, tags, dates etc.

To get around this limitation, a cache is introduced.

This cache works by running org-element-process-buffer each time a file is saved or the buffer moves out of focus (e.g. when switching to another window or buffer).

Sub-modules can register on element types to compute data on a file or headline level that can then be used to implement fast views on all the data in the zettelkasten, e.g. for listing open projects, tasks or calendar entries, generating clocking reports and statistics or for use with the integrated spaced repetition system.

A query language is implemented on this cache for building custom views on the data.

org-zk-titlecase

When creating a note using the org-zk commands, the title that is entered is automatically converted to title-case.

This only works for English text and not all rules are implemented. Multi-word conjunctions are not supported yet.

Testing

Integration testing is done by providing an input file, a set of operations and an file with the expected output.

The output is written to a third _got file, which can be diffed with the expected output or used to replace the _expected file if the output generated was valid.

Credits

• The file view uses the same font faces as skeeto/elfeed.

License

Copyright © Leon Rische and contributors. Distributed under the GNU General Public License, Version 3