/paw

Emacs Annotation and Language Learning tool.

Primary LanguageEmacs Lisp

paw (point-and-write)

images/logo.jpg

Study for a life in Emacs.

paw.el was created in 2021 as an annotation tool, but I was not used it a lot at that time. Recently I picked it up and dramatically changed its usage. Now it is a complete solution of making annotations and language learning tools, right inside Emacs.

paw relies on sqlite. All annotations are saved inside a database, which means it can be easily backup and you can read and make annotation right inside Emacs (PC/mobile) using same database.

paw is not just a dictionary tool. paw was created in mind that:

  1. Query word as easy as possible, even no need to click.
  2. Save and trace all known/unknown words and wordlist support, online or offline.
  3. Integrate with other great open source tools, libraries, and dictionaries.
  4. Integrate with external dictionary tools.
  5. Support many languages.
  6. Multi-Platform Support, including Mobile/e-link devices.
  7. Use AI as assistant.

The standout feature of Paw is the ability to scan an entire buffer, tokenize all words, batch-query them, and filter them out according to user-provided tags, ultimately applying overlays alongside meanings to all discovered words. In essence, Paw proactively queries the dictionary on your behalf, making the highlighted word’s meaning immediately available.

This means that you can simply hover your mouse over a word to view its definition, without having to click on it.

This package is mainly used along with:

  1. nov-mode
  2. org-mode
  3. eaf-mode
  4. eww-mode
  5. wallabag-entry-mode
  6. pdf-view-mode

But it is able to work with all modes (defined in paw-annotation-mode-supported-modes) in Emacs.

I’ll update the doc in the future, there are too much to cover. But I’ll show you the most useful cases first:

Reading and annotation

  1. Open an epub file using nov.el or open an org file or browse web with eww
  2. Enable paw-annotation-mode (buffer will be turned into read-only if paw-annotation-read-only-enable is t to avoid accidental alternation)
  3. Use paw add word/highlight/bookmark/todo/, for example, paw-add-word, paw-add-highlight… Please check M-x paw-add-* related commands for all type of supported annotations.
  4. Annotations will be highlighted via overlay. Under the annotation, you can do various of operations. For example, to edit the annotation, under the annotation, press i, paw-find-note. It will create an temporary org file dedicated for this annotation, then save by C-c C-c, changes will be saved in the database respectively.
  5. Call paw, paw-list-all-annotations or paw-list-all-links to manage or jump to annotation location (One useful case is that make highlight or create bookmark on eww, and jump back to the exact same annotation location simply pressing Enter)
  6. Under the annotation, M-x paw-view-notes, all notes under the same path will be shown, giving you an overview on all notes you’ve made for the same file. Or M-x paw-find-notes to update/save all notes under the same path into an org file under paw-note-dir, and jump to the annotation location inside the org file.
  7. paw-add-bookmark works great on eww-mode/eaf-mode (browser or pdf-viewer)

Language Learning

  1. Copy a text you want to learn to an org file or browse a page with eww.
  2. Enable paw-annotation-mode
  3. Click/Select word/sentence with paw-view-note, it will show you an informative buffer of the word/sentence. It has dictionary, translation, notes and also able to search by external browser.
  4. Use paw-add-offline-word or paw-add-online-word to add a new word. For Online words, please check
Anki Integration Eudic Integration

All offline/online words appeared on any buffers that were enabled paw-annoation-mode will be highlighted, useful for language learners using the important method: Repetition.

  1. For offline words, please setup paw-offline-studylist before using it. By default we have two offline studylists: English Studylist and Japanese Studylist.
  2. You can also use paw-add-word. Unlike offline/online word, the word added by paw-add-word is linked with an unique id, it is only highlighted at the location where we added it.
  3. Under any annotation, click or press v to invoke paw-view-note, a buffer with saved meanings, dictionary, translation, notes is shown.
  4. Under any annotation, click or press c to invoke paw-change-word-learning-level, select from level 1~5. 1~4 are controlled by paw-level-1-word-face to paw-level-4-word-face for online words, paw-level-1-offline-word-face to paw-level-4-offline-word-face for offline words, while 5 is considered as Known word, skipping adding overlay face (just hiding). You can change the word to level 5 while keep it hiding, but if you don’t need it anymore, simply run paw-delete-word to delete it.
  5. Click/Select word/sentence with paw-view-note-current-thing, paw-view-note-next-thing, paw-view-note-prev-thing, it will do segmentation using kagome (Japanese), and ECDICT (English).
  6. Check https://www.lingq.com/en/, you know more ideas how to learn a foreign language with paw.el.
  7. Run paw-export-notes-to-csv, check the csv file under paw-csv-file, import it to Anki for further studying.
  8. Both paw-add-word and paw-add-online-word will capture the context sentence and save it into Notes field.

Tip: Find some interesting contents on LingQ, and browse using eww, enable paw-annotation-mode, enjoy language studying!

Search a word/sentence

  1. Select/Under a word/sentence
  2. Run paw-view-note (no segmentation, good for word) or paw-view-note-current-thing (segmentation, good for sentence)
  3. Optimized for eaf-mode, nov-mode, eww-mode, focus-mode

Installation

Please install sdcv, kagome (For Japanese segmentation), ecdict(for English learning), focus, edge-tts (pronunciation), gptel, go-translate, goldendict, If you want to make use all power of it.

You may also need to install shrface, wallabag, eaf, popweb, pdf-tools, and more (Please check the source code what are missing, I’ll organize them better in the future.)

Install paw

(package! paw :recipe (:host github :repo "chenyanming/paw" :files ("*")))

Language Detection

Set paw-detect-language-p to t, and run pip install gcld3, paw will use gcld3 (may use others tools in the future, check paw-detect-language-program) to detect the language for more accurate tts pronunciation and translation.

For android install:

pkg install tur-repo # https://github.com/termux-user-repository/tur
pkg install python3.10 # install python 3.10 (newer versions may not work at the time of writing)
apt update
apt install protobuf
pip3.10 install gcld3

If you don’t want to use language detection program, paw use simple ascii rate: paw-ascii-rate to detect the language, if the rate is greater than paw-ascii-rate, then it is considered as English, otherwise use paw-detect-language-program to detect the language of the TEXT, if paw-detect-language-p is t, or return as paw-non-ascii-language if paw-detect-language-p is nil.

Setup paw-python-program if necessary, if the pip module is installed with different python version, for android, set it to python3.10

Supported edge-tts voice:

  • paw-tts-english-voice
  • paw-tts-zh-cn-voice
  • paw-tts-zh-tw-voice
  • paw-tts-japanese-voice
  • paw-tts-korean-voice
  • Other languages: paw-tts-multilingual-voice

Welcome PRs to add more languages :)

Anki Integration

  1. PC: Install Anki.
  2. PC: Install AnkiConnect, add-on code: 2055492159.
  3. Android: Install AnkiDroid from F-Driod (We need it has full media access right, so that Emacs can copy audio files to it. Instead of using AnkiConect to download the audio, paw will download and cache the audio in paw-tts-cache-dir after the voice is pronounced.).
  4. Android: Install AnkiconnectAndroid
  5. Install Anki Editor if not installed.
  6. If you just want to try or use the default settings. Please download the default template Memrise Templates (Lτ) v3.32.apkg and import it into anki then you are all done.

If you want to use different template,

  1. Run paw-anki-configure-card-format to choose from default templates paw-anki-templates, or configure deck, note type, filed-name, and filed-values one by one temporarily. Currently Supported field-value:
    • word: the word to learn
    • exp: the explanation of the word
    • sound: the sound file of the word
    • note: the note of the word
    • cloze_note: the note of the word, word is clozed
    • cloze_note_exp_hint: the note of the word, word is clozed, use exp as hint
    • choices: the choices of the word
    • nil: empty field
    • Other values: the value of the field, it must be a string
  2. If you want to make it permanent, set paw-anki-deck, paw-anki-note-type, paw-anki-field-names and paw-anki-field-values manually in your config,
  3. Configure paw-online-word-servers, (setq paw-online-word-servers '(anki)) to enable anki server, or (setq paw-online-word-servers '(eudic anki)) to enable both eudic and anki servers.

PS: All types of annotations (not all are tested) could be added into Anki. Either using paw-add-online-word (anywhere), or paw-anki-editor-push-note(s) (dashboard) paw-anki-editor-delete-note(s) (dashboard).

paw-anki-editor-delete-note: Delete note at point to Anki.

paw-anki-editor-push-note: Push note at point to Anki.

paw-anki-editor-push-notes: Push notes of marked-entries in dashboard to anki, or push all anki notes in the same origin path (same file or same buffer). Same file name under paw-annotation-search-paths is also considerred same origin path.

paw-anki-editor-delete-notes: Delete anki notes of marked-entries in dashboard, or delete all anki notes in the same origin path (same file or same buffer), Same file name under paw-annotation-search-paths is also considerred same origin path.

Other templates: MemCloze.apkg

Eudic Integration

  1. Apply Authorization key on https://my.eudic.net/OpenAPI/Authorization, and fill it into paw-authorization-keys before adding online words.
  2. Configure paw-online-word-servers, (setq paw-online-word-servers '(eudic)) to enable Eudic server, or (setq paw-online-word-servers '(eudic anki)) to enable both eudic and anki servers.

PS: Only online words can be added into Eudic. Mainly via command paw-add-online-word

EAF Integration

Use my fork: https://github.com/chenyanming/eaf-browser which adds paw support.

English Configuration

Japanese Configuration

Database Synchronization

I personally use Syncting to share the database between PC and Android. The drawback is that, if the database is in used in any party, the synchronization will stop. You can run paw-db-sync to close the connection with the database or close Emacs directly before Synchronization.

My Setup

(use-package paw
  :init
  (setq paw-db-file (expand-file-name "paw.sqlite" org-directory))
  ;; ecdict dictionary
  (setq paw-ecdict-db (expand-file-name "stardict.db" org-directory))
  ;; setup ECDICT before using it, and create the files manually if not exist
  (setq paw-ecdict-wordlist-files `(
                                    ,(expand-file-name "美国当代英语语料库.csv" org-directory) ;; https://www.eapfoundation.com/vocab/academic/other/mawl/
                                    ,(expand-file-name "mawl.csv" org-directory) ;; https://www.eapfoundation.com/vocab/academic/other/mawl/
                                    ,(expand-file-name "opal.csv" org-directory) ;; https://www.oxfordlearnersdictionaries.com/wordlists/
                                    ,(expand-file-name "5000.csv" org-directory) ;; https://www.oxfordlearnersdictionaries.com/wordlists/
                                    ,(expand-file-name "极品GRE红宝书.csv" org-directory)
                                    ,(expand-file-name "gre.txt" org-directory)
                                    ,(expand-file-name "托福绿宝书.csv" org-directory)
                                    ,(expand-file-name "2021_Teachers_AcademicCollocationList.csv" org-directory) ;; https://www.pearsonpte.com/teachers/academic-collocation
                                    ,(expand-file-name "The Unofficial Harry Potter Vocabulary Builder.csv" org-directory)
                                    ,(expand-file-name "Illustrated Everyday Expressions with Stories.csv" org-directory)
                                    ,(expand-file-name "Essential Idioms in English.csv" org-directory)
                                    ,(expand-file-name "idioms.txt" org-directory)
                                    ,(expand-file-name "phrase-list.csv" org-directory) ;; https://www.oxfordlearnersdictionaries.com/wordlists/
                                    ,(expand-file-name "英语生词本.csv" org-directory)
                                    ))
  ;; setup ECDICT before using it, and create the files manually if not exist
  (setq paw-ecdict-known-words-files `(,(expand-file-name "eudic.csv" org-directory)
                                       ,(expand-file-name "english.txt" org-directory)))
  ;; setup ECDICT before using it, and create the file manually if not exists
  (setq paw-ecdict-default-known-words-file (expand-file-name "english.txt" org-directory))

  ;; jlpt dictionary
  (setq paw-jlpt-db (expand-file-name "japanese.db" org-directory))
  ;; setup jlpt before using it, and create the files manually if not exist
  (setq paw-jlpt-wordlist-files `(,(expand-file-name "日语生词本.csv" org-directory)
                                  ,(expand-file-name "日本语红宝书.csv" org-directory)
                                  ,(expand-file-name "蓝宝书日语文法.csv" org-directory)
                                  ,(expand-file-name "NEW-JLPT.csv" org-directory)
                                    ))
  ;; setup jlpt before using it, and create the files manually if not exist
  (setq paw-jlpt-known-words-files `(,(expand-file-name "japanese.txt" org-directory)))
  ;; setup jlpt before using it, and create the file manually if not exists
  (setq paw-jlpt-default-known-words-file (expand-file-name "japanese.txt" org-directory))
  :custom
  ;; (paw-svg-enable t)
  ;; Use pbm buttons on android
  (paw-pbm-enable (if (eq system-type 'android) t))
  ;; Use all the icons icon on dashboard
  (paw-all-the-icons-icon-enable t)
  ;; Use all the icons button on non-android system
  (paw-all-the-icons-button-enable (unless (eq system-type 'android) t))
  (paw-detect-language-p t)
  (paw-python-program (if (string-equal system-type "android") "python3.10" "python3"))
  (paw-detect-language-program 'gcld3) ;; android can only install cld3
  (paw-click-overlay-enable t)
  (paw-annotation-read-only-enable t)
  (paw-annotation-show-wordlists-words-p t) ;; setup ECDICT before using it
  (paw-annotation-show-unknown-words-p nil) ;; setup ECDICT before using it
  (paw-ecdict-frq 3000) ;; all possible words, 0 no frq data
  (paw-ecdict-bnc -1) ;; all possible words, 0 no bnc data
  (paw-ecdict-tags "cet4 cet6 ielts toefl gre empty") ;; no easy words
  (paw-ecdict-oxford 0) ;; no easy words
  (paw-ecdict-collins-max-level 3) ;; no easy words
  ;; (paw-posframe-p (if (string-equal system-type "android") t))
  ;; For online words, you have to apply api on
  ;; https://my.eudic.net/OpenAPI/Authorization
  (paw-authorization-keys (auth-source-pick-first-password :host "eudic-api-key"))
  ;; limit the other languages web buttons number
  (paw-english-web-button-number (if (eq system-type 'android) 4 4))
  ;; limit the japanese web buttons number
  (paw-japanese-web-button-number (if (eq system-type 'android) 3 4))
  ;; limit the general web buttons number
  (paw-general-web-button-number (if (eq system-type 'android) 2 3))
  ;; (paw-default-say-word-function (if (eq system-type 'android) 'paw-android-say-word 'paw-say-word))
  (paw-tts-zh-cn-voice "zh-CN-YunjianNeural") ; zh-CN-XiaoxiaoNeural, zh-CN-YunyangNeural
  ;; (paw-sdcv-dictionary-list '("简明英汉字典增强版"))
  ;; add online word by default for add button
  (paw-add-button-online-p t)
  ;; show the note both in minibuffer or/and *paw-view-note*
  ;; To use this, you need to setup ECDICT (English) or JLPT (Japanese) before
  ;; use this, otherwise, use the 'buffer instead
  (paw-view-note-show-type (if (eq system-type 'android) 'buffer 'all))
  ;; must be one of the studylist name in `paw-studylist', please run `paw-get-all-studylist' to get the available studylists.
  (paw-default-online-studylist "ID: 133584289117482484, Language: en, Name: Business")
  ;; (paw-default-online-studylist "ID: 133616431737936696, Language: en, Name: 00 Japanese Mini Stories")
  (paw-offline-studylist '(("English Studylist" ;; studylist name when choosing offline studylist
                            (id . "1") ;; random id for internal use, but it could not be the same as any id in online study list defined in `paw-studylist'
                            (language . "en") ;; language of the studylist
                            (name . "English")) ;; name of the studylist
                           ("Japanese Studylist"
                            (id . "2")
                            (language . "ja")
                            (name . "Japanese"))))
  ;; must be one of the studylist name in `paw-offline-studylist'
  (paw-default-offline-studylist "English Studylist")
  (paw-search-page-max-rows (if (eq system-type 'android) 31 41))
  (paw-add-offline-word-without-asking t)
  (paw-add-online-word-without-asking t)
  :config
  (setq paw-note-dir (expand-file-name "Dict_Notes" org-directory))
  ;; if the file was moved to other places after adding annotations, we can add
  ;; the parent path of the file for paw to search. This is necessary for
  ;; multiple clients (PC/Mobile/Pad) to use the same database but file location
  ;; is different.
  (setq paw-annotation-search-paths '("~/Data/Books/"
                                       "/storage/emulated/0/Books/"
                                       "/storage/emulated/0/Download/"
                                       "/storage/emulated/0/Download/Telegram/"
                                       "/storage/emulated/0/org/web/"
                                       "~/org/web/"
                                       "~/org/web/"
                                       ))

  ;; show image annotation in *paw-view-note*
  (add-hook 'paw-view-note-after-render-hook #'org-display-inline-images)
  (add-hook 'context-menu-functions #'paw-annotation-context-menu)

  (unless (string-equal system-type "android")
      (setq paw-dictionary-browse-function 'eaf-browse)
      (setq paw-mdict-dictionary-function 'eaf-browse))

  ;; my personal configs
  (add-hook 'paw-view-note-mode-hook #'paw-view-note-setup)
  (add-hook 'paw-annotation-mode-hook #'paw-annotation-setup)
  )

(defun paw-view-note-setup ()
  (org-writeroom-setup)
  (org-modern-mode)
  (pangu-spacing-mode))

(defun paw-annotation-setup()
  ;; TODO need manual enable later
  (flyspell-mode -1))

Demos

https://emacs-china.org/uploads/default/original/3X/2/b/2bc2d9fd996827097b13f751c327ad7141376f88.gif

https://emacs-china.org/uploads/default/original/3X/3/5/3544a2bf376d1f3b8f1fc86063af2975e4da42b4.gif

https://emacs-china.org/uploads/default/optimized/3X/b/8/b8d7ae2d68baae4fe7dcb6477998cb761e28165f_2_1234x1000.png

https://emacs-china.org/uploads/default/optimized/3X/b/b/bb28af2a398f8d33861002facc62a6f7782be3b7_2_1232x1000.png

https://emacs-china.org/uploads/default/original/3X/5/5/55c6991c0521c6a70dbbce844ce1fb650119dc1e.png

https://emacs-china.org/uploads/default/original/3X/9/7/971b92c62a837e0a2e053e0e01f02916b8ae465d.png

User Discussions

https://emacs-china.org/t/paw-el-emacs-lingq/27331

https://t.me/emacs_paw

Thanks the following great tools

  1. LingQ: Learning a language by reading
  2. Kindle Vocabulary Builder
  3. org noter
  4. Chatgpt
  5. SDCV
  6. go-translate
  7. Eudic
  8. 蒙哥阅读器
  9. Anki
  10. Yomitan