Automatic Diary

Automatic Diary is a script that creates one timeline from various digital sources describing your life.

On the screenshot above, you can see: calendar events (blue), completed todo list items (yellow), software development work (gray), sport activity (green), and films watched (red).

Why

Automatic Diary can be useful to those who:

(a) have bad memory,
(b) want to remember what they did,
(c) but don't have the time to write a real diary.

What is collected

Calendar events (via CalDAV service and iCalendar file reading)
Emails sent and received (via Maildir reading)
Sport activity (via spreadsheet parsing)
Facebook and Twitter posts (via Facebook/Twitter data archive parsing)
Software development work (via Git repository log)
Completed todo list items (via todo.txt parsing)
Films watched (via ČSFD website parsing and Trakt.tv)
Custom diary notes (via plain-text and Org-mode files parsing)

What is not collected

Automatic Diary could be extended to collect also the following data but right now it is considered low priority:

Installation

Mac

$ brew install python
$ pip install poetry
$ make setup

Arch Linux

# pacman -S python-poetry libsecret
$ make setup

Other systems

Install these dependencies manually:

Python >= 3.7
poetry
libsecret (Linux keyring) -- required by the caldav provider

Then run:

$ make setup

Configuration

Before you run Automatic Diary, you need to configure all the providers (sources from which the data for your timeline will be read). The following providers are supported:

caldav
csfd
csv
facebook
git
icalendar
maildir
orgmode
orgmodelist
todotxt
trakt
twitter
txt

All providers are configured using a single config.json file. Use config-sample.json as a template for your own configuration.

caldav

Input: CalDAV server
Output: Names and locations of calendar events

Configuration:

{
    "url": "<server url>",
    "username": "<server authentication username>",
    "password_key": "<server authentication password -- libsecret key>",
    "password_val": "<server authentication password -- libsecret value>",
    "cache_dir": "<cache directory path>"
}

csfd

Input: User profile on ČSFD (film database website, something like IMDB)
Output: Titles of films rated

Configuration:

{
    "profile_url": "<csfd.cz profile url>",
    "cache_dir": "<cache directory path>"
}

csv

Input: CSV spreadsheet (.csv) file
Output: Rows formatted using a template

Configuration

{
    "path": "<csv file path>",
    "date_source": "{{<column name>}}",
    "date_format": "<strptime date format>",
    "text_source": "<template string in the Mustache format>"
}

facebook

Input: Downloaded Facebook archive
Output: Texts of statuses

Configuration

{
    "path": "<path to wall.htm or timeline.htm>",
    "username": "<facebook username>"
}

git

Input: Directory with checked-out Git repositories and an author name
Output: Commit messages by the author from all repositories

Configuration

"config": {
    "base_path": "<path to directory - will be searched recursively for git repos>",
    "author": "<author name>"
}

icalendar

Input: Calendar events stored offline in iCalendar (.ics) files
Output: Names and locations of calendar events
Configuration:
```
{
    "paths": [
        "<path to an .ics file>",
        ...
    ]
}
```
Not that events from all the listed .ics files will be merged -- duplicate events removed.

maildir

Input: Emails stored offline in the Maildir format
Output: Subjects of emails

Configuration:

{
    "received_pathname": "<glob pathname of directories with received emails>",
    "sent_pathname": "<glob pathname of directories with sent emails>"
}

orgmode

Input: Emacs Org-mode (.org) file in format:

  * <2019-01-17 Thu>

  Lorem ipsum
  foo.

  bar

  * <2019-01-18 Fri>

  spam spam
  ...

Output: Example:

2019-01-17,Lorem ipsum foo.
2019-01-17,bar
2019-01-18,spam spam

Configuration:

{
    "path": "<path to the .org file>"
}

orgmodelist

Input: Emacs Org-mode (.org) file in format:

  - Lorem ipsum foo. <2019-01-17 Thu>
  - bar <2019-01-18 Fri 11:30>
  ...

Output: Example:

2019-01-17,Lorem ipsum foo.
2019-01-18T11:00:00+02:00,bar

Configuration:

{
    "path": "<path to the .org file>"
}

todotxt

Input: Todo.txt completed tasks file (done.txt)
Output: Texts of completed tasks
Configuration
```
{
    "path": "<done.txt file path>"
}
```

trakt

Input: Manually create an application within Trakt.tv. Manual login with OAuth when running this provider. A recommended Redirect URI is urn:ietf:wg:oauth:2.0:oob
Output: Texts of watched films and shows

Configuration

{
    "key_id": "<Trakt.tv app Key ID>",
    "key_secret": "<Trakt.tv app Secret>",
    "app_id": "<Trakt.tv app ID (numeric)>"
}

twitter

Input: Downloaded Twitter archive
Output: Texts of tweets

Configuration

{
    "path": "<path to twitter archive directory>"
}

txt

Input: Plain text (.txt) file in format:

2015-12-02 St
    Some
    Text
    Lorem
        Ipsum
            Hierarchy
            Spam
2015-12-03 Čt
    Foobar
    ...

Output: Example:

2015-12-02,Some
2015-12-02,Text
2015-12-02,Lorem: Ipsum: Hierarchy
2015-12-02,Lorem: Ipsum: Spam
2015-12-03,Foobar

Configuration

{
    "path": "<path to the .txt file>"
}

Usage

Generating CSV

The basic output of Automatic Diary is a CSV file. Generate it by running:

$ ./automatic-diary <config path> <output csv path>

Example:

$ ./automatic-diary ~/.config/automatic-diary/config.json ~/Desktop/automatic_diary.csv

The CSV output is in format:

<datetime>,<provider>,<subprovider>,<text>

Example CSV output:

2019-01-23T15:31:54-08:00,git,human-activities,model: Log exceptions while scanning
2019-01-24T09:00:00+01:00,caldav,https://dav.mailbox.org/caldav/da39a3ee5e6b,DHL Packstation
2019-01-25,todotxt,done.txt,Opravit Ondrovi kolo

See the help for all command line options:

$ ./automatic-diary --help

Visualization

The output CSV file can also be rendered as an HTML document which looks kind of like a calendar. See the screenshot above. Generate this HTML document by running:

$ ./automatic-diary-visualize <csv path> <output html path>

Example:

$ ./automatic-diary-visualize ~/Desktop/automatic_diary.csv ~/Desktop/automatic_diary.html

See the help for all command line options:

$ ./automatic-diary-visualize --help

Development

Testing and linting

$ make test
$ make lint

Help

$ make help

Contributing

Feel free to remix this project under the terms of the GNU General Public License version 3 or later. See COPYING and NOTICE.

jakubvalenta/automatic-diary

Automatic Diary

Why

What is collected

What is not collected

Installation

Mac

Arch Linux

Other systems

Configuration

caldav

csfd

csv

facebook

git

icalendar

maildir

orgmode

orgmodelist

todotxt

trakt

twitter

txt

Usage

Generating CSV

Visualization

Development

Testing and linting

Help

Contributing