/YANOM-Note-O-Matic

Yet Another Note-O-Matic - Convert files from closed file system note taking systems to various output formats

Primary LanguagePythonMIT LicenseMIT

coverage 91% open issues License version tag

YANOM - Yet Another Note-O-Matic

YANOM - stands for Yet Another Note-O-Matic.

YANOM is a file converter to support the use of non-proprietary open file formats for note-taking systems.

It achieves this by converting proprietary note system formats into Markdown or HTML files.

Additionally, YANOM has the capability to add support for modified Markdown formats used by specific Markdown note-taking system, for example Obsidian image tags are formatted to support image sizing.

YANOM's Goals

  • To be a user-friendly tool anybody can use, with documentation that is detailed enough for any user to install and use.
  • Incrementally support additional proprietary formats and open file formats when possible.

Sections in this read me are

YANOM Note-O-Matic wiki

The YANOM wiki explains features, functionality, installation and use in greater detail than this read me.

Current capabilities

YANOM only exports to HTML or Markdown.

Future versions will support conversion from additional proprietary file formats.

Current functionality

  • Convert Note-Station .nsx export files to Markdown or HTML
  • Convert Nimbus Notes .zip export files to Markdown or HTML
  • Convert HTML to Markdown
  • Convert Markdown to HTML
  • Convert Markdown to a different format of Markdown
  • Report details of and issues identified during conversion
    • Details such as
      • number of notes converted
      • attachment files and images processed,
      • links between notes recreated
    • Issues identified such as
      • Links between note station notes that could not be recreated
      • Links in notes that are for files that can not be located
      • Orphan Files that exist but are not linked to a note
  • User selectable filename and directory name cleaning options (file name cleaning only for nsx files)
  • List of available Markdown formats that can be used as inputs or outputs
    • CommonMark (Used by Joplin)
    • GFM - Git Flavoured Markdown (Typora, Git Hub, Haroopad)
    • Obsidian formatted markdown
    • MultiMarkdown (MultiMarkdown Composer)
    • Pandoc markdown (not for nimbus notes)
    • Pandoc markdown-strict
    • QOwnNotes optimised markdown-strict
  • Note content that will be converted successfully
    • Headers
    • Bulleted lists
    • Numbered lists
    • Checklists
      • Checklist items that have hyperlinks will maintain the links in exported markdown
    • Tables
    • Images
    • Image width where supported in Markdown
    • IFrames
    • Metadata - support JSON, TOML or YAML front matter, and meta tags in HTML.
      • It is also possible include the metadata as plain text at the top of the exported file
      • Option to discard metadata
    • Tags
      • included in front matter, html header or as plain text with an optional prefix character
      • option to split grouped tags photography/landscape/winter becomes #photography, #landscape, #winter
      • option to remove spaces from tag names, spaces are replaced with hyphen -
    • File attachments are maintained
    • Option to
    • Note-Station specific features
      • Charts are recreated. Options to place an image, data table of the chart data and, a link to a csv file on the exported page.
      • If duplicate files are attached to different notes within a notebook only a single copy of the attachment is stored in the attachment folder and links will be to that file.
      • Links between note pages.
      • Note-Station audio notes - are attached as an attachment
      • Option to include creation time in file name
      • Allow user to select the format for creation and modified time, and the name used for these metadata fields.
      • Add file extension for filenames, for common file types, where extension is missing. This may occur when there are sync issues between desktop and server note station versions.
      • File name cleaning options - in manual mode user can choose to keep/remove spaces, unicode characters, uppercase characters, choose what character to replace spaces with.
      • File name lengths - User can set a maximum file name length. Long note titles can cause issues on Windows where long paths are not enabled, on these systems YANOM restricts file names to 64 characters.
      • Retrieve meta data
      • Some ability to handle nsx files from systems with synchronisation issues, such as attachments with no names or extensions, missing data fields describing the data, however final exported content will be variable quality depending on the synchronisation issue.
    • Nimbus Note specific features
      • Missing data in converted notes may be caused by Nimbus failing to export the notes correctly. Multiple intermittent failures to export a complete note have been seen. Unzip the nimbus export file and open the html in a browser, typically the note will be blank.
      • The Nimbus conversion does not use pandoc.
      • Links between note pages.
        • For nimbus most of the time this will be successful if the linked to note is in the export set. A wiki entry will be created for the issues. They are very similar to those in Note-Station
        • File name cleaning options - in manual mode user can choose to keep/remove spaces, unicode characters, uppercase characters, choose what character to replace spaces with.
          • File name lengths - User can set a maximum file name length. Long note titles can cause issues on Windows where long paths are not enabled, on these systems YANOM restricts file names to 64 characters.
        • Retrieve meta data
          • Nimbus does not store tag data in the notes. If the tags are copied from the top of the nimbus note web page or app and pasted at the top of the note itself they will be identified and added as meta tags to html or a front matter section in markdown

The formatting for QOwnNotes and Obsidian are variations on common Markdown formats optimised for those note systems.

Getting Started

Packaged versions of YANOM have been created, please check the WIKI for details on how to install those.

The wiki also includes more detailed instructions on installing and using the source code than can be easily documented here.

Prerequisites

Prerequisites source code

YANOM relies on a python environment when run form source code.

  1. You will need to have a working installation of Python 3.6 or higher. Details of how to install python, and the installation files, can be found on the python website
  2. Once Python is installed, install pipenv using pip install pipenv.
  3. If using PyCharm - Edit your yanom.py Run time settings from the menu 'Run' -> 'Edit configuration..' and tick the "Emulate terminal in output console" option in the 'Execution' section. This is required for the interactive command line to run.

YANOM also requires Pandoc. Pandoc v1.16 or higher should be used and ideally 2.11 or higher as that is where most testing has been done.

Prerequisite for the Debian packaged version

Pandoc is required to be installed.

No prerequisites for the Windows and Mac OSX packaged versions, or the docker image

The packaged versions include the required python environment and pandoc.

Using the pre-built packaged versions for Debian linux, windows and Mac OSX

Packaged versions of YANOM are available. Please see the wiki for how to install and use them. Linux package is stuck at version 1.5.0 for the latest version use source code.

Also, please note the Mac OSX package is slow to start and can take 20 or more seconds to launch. The windows versoin takes around 10 seconds to start.

The Mac OSX version is code signed and has been notarised by apple, and should not give any security warnings.

Using the docker image

There is no docker version for 1.6.0

A docker image has been created for version 1.5.0 and is available on docker hub.

For further information on the duse of the YANOM docker image please check the [wiki page](https://github.com/kevindurston21/YANOM-Note-O-Matic/wiki/Installing YANOM using Docker.md)

Installing from source code

Download the source code from git hub

Un-zip the downloaded file.

Use pip install -r requirements.txt to install the required dependencies to run the code. If you are going to develop the code use pip install -r requirements_dev.txt to install additional development dependencies.

NOTE it has been seen in some linux versions that a dependency for the package toml was not installed automatically. The symptoms seen were that when a file was converted using toml as the front matter format YANOM would crash. The workaround was to use pipenv install toml and then YANOM would run OK.

At the command line type python yanom.py if you wish to add command line arguments you just add them like this python yanom.py --source notes

For details of the command line options and how to use YANOM please refer to the Using YANOM wiki page

Running the tests

Unit Tests with pytest

Some tests were created to run with unittest, but these also run with pytest.

Please refer to the test folder for the test files.

Install pytest for testing and pytest-cov for coverage reports.

You can run all tests and create html coverage report if you use this command in the root of the project

pytest --cov-report html:cov_html test/ --cov=src

Current coverage is currently 90%

End-to-end tests

Only a simple manual end-to-end test process exists.

Conversion is made from the test.nsx file to Markdown, using each combination of settings.

One set tof the markdown files generated form the test.nsx file is converted to html.

The generated html files are then converted back into a Markdown format.

Deployment

Deploying source code to a live environment is addressed in the wiki

Deployment of packages is achieved using pyinstaller. For details see the wiki

Contributing

Please read CODE_OF_CONDUCT.md for details on our code of conduct.

Please read CONTRIBUTING.md for the process for submitting pull requests to us.

Versioning

We use Semantic Versioning for versioning. For the versions available, see the tags on this repository.

Change log

Please see CHANGELOG.md

Authors

  • Kevin Durston

See also the list of contributors who participated in this project.

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Acknowledgments

  • Maboroshy for showing that dissecting an NSX file was possible