audiobookfinder (adbf)

An example program to learn Rust and meet its concepts by: find audio books on different clients/devices.

Why Rust?

Rust is an awesome but difficult to learn programming language using different approaches and concepts to solve the current main software development issues for system programming:

Secure Programming Concepts: let the computer/compiler do what it can do better than a programmer: safe threading, error-concepts, forbid everything non-safe by default
Quality: high level language concepts, easy to embed and include high quality external packages, which lets you implement more functionality in less code
Embedded: easy cross compiling, interfaces to C, becoming better to be stripped down to core system functions for the sake of minimum code footprint. async/await for non OS programs, no_std and the async executors (even own ones) will be important.
Parallelism and Concurrency: what else to do with multi-core cpu, we are not getting much faster anymore, and often cpus are idling due to blocking code. With async / await and futures Rust offers with its security features a very good way of dealing with it.
Testing and Documentation: really nice build-in concepts, even combined !!:wink:
Crossplatform: many good Rust libraries are crossplatform, and building on top of that just works

Especially for IoT: I want secure and thereby safe products at home which cannot be turned into zombie devices by buffer overflow and injection, always think of what can go wrong, and let the compiler tell you when you do a common mistake.

As a C++ developer, I know some C++11/14/17 enhancements, and some don't convince me at all, just look here about the "costs" you have and what it looks like in Simon Brand's "How Rust gets polymorphism right".

My first program in Rust
Features
Releases
Documentation
Architecture
Changes
Screenshots
ToDo
CI Continuous Integration
Goals
Dependencies
Tools
Useful links

My first program in Rust

I planned to do something useful for myself. The program collects information about audio books on different devices/clients, stores it and then processes it, e.g. showing stats, finding duplicates, aggregating everything at one place by a button click.

The task of collecting audio book data can be exchanged with any other task, this basically leads to a local network agent approach with a libp2p swarm.

There is the Architectural Design in UML.

It's crossplatform now!

Features

no unsafe Rust / no lib dependancies
crossplatform (Linux/Windows)
http-server with websocket communication to act as web ui client
multi-client via libp2p: mDNS, kademlia communication over noise protocol
build-in documentation
appveyor and travis CI
build-in unit-testing the right way
BK-tree data structure for approximate string matching
simple mobile app based on ffi interface Dart / Flutter / flutterust flutterustbridge

Releases

There are Ubuntu release install packages, (.deb files for LTS 20.04 focal, LTS 18.04 bionic, LTS 16.04 xenial, a MacOS build zipped, a Windows build 7-zipped, and an Android apk ready to be downloaded.

Documentation

The documentation is fully generated.

It is an inline documentation from CI generated documentation - Rust does a nice job here as well!

Architecture

Changes

only more tools' links (gitui, drawio-desktop, mermaid)
fixed warnings according to latest Rust release
keep and fixed minor version from libp2p (very troublesome, because of mix of new libp2p which seperates features into different crates and the old which uses only libp2p crate)
added licenses tab to flutter app and webui
added linux debian .deb package deployment
fixed and beautified CI process
flutter app:
- network view ready and functioning
github action, (it's a good idea but too many marketplace) as CI brought up, the app for android is downloadable and fast 😁
since marrying flutterust and old adbflib, package name now is adbflib, new for readers: flutterustbridge
added and fixed first shot mobile app mdns feature ... it just works 😮 😊
produced apk can even be design tested in android studio
had to introduce platform dependent crate compilation, here only webbrowser crate
added mobile app build by using flutterust by Shady Khalifa 👍 great
- Dart/Flutter coding required as frontend on mobile app side
- add github actions as 3rd CI (quite new to me)
- distinguished lib and binary builds in CI
- will lead to architectural changes -> parts to: lib, binary, shared
webui path input dialog working, more results presented
changed a range based implementation with a regex based implementation, also for the interest of the regex crate in Rust
moved direct connection to net via ipc out, net is the long running slow part, (local) collection is the fast part

click for older changes

* added `ADBF_LOG` env variable
* webui now starts with a selection dialog where you can add/change preselected folders/dirs
* defined a trait for tag information, now: id3, mp4, flac but no awesomeness ...
* debugging messages for collection, now collection is a bktree
* state machine was added, wrapped in a layer of a custom net behaviour in the libp2p swarm ([architecture design](#architecture) updated)
* my unfortunately now unused [observer pattern](../../wiki/Observer-pattern) was added to wiki - a better version of what
  could be found in internet (Rust is very restrictive, some patterns don't work that well).
* webui shows search ongoing on other peers, and then result (number of audio files)
* webui changes, option added for with or without automatic browser opening, bump
* boostrap, jquery upgraded, some webui animation, better net communication struct, multiaddr on peer per webui tooltip
* peers in webui can now deregister because of e.g. timeout
* using libp2p network swarm, replacing single-on mdsn with it, but having same functionality
* releases for ubuntu, windows, raspberry (20LTS had a upx packing problem due to changed compiler flags, I suppose)
* fixed webui behavior, now crossplatform (after cursive backend change, taglib replaced by id3)
* pretty webui design, net messages as good as tui now, fixed thread termination issues to be mostly graceful
* webui is in sync now, prepare net messages for webui to maybe replace tui
* back to many threads, but synced and working just fine - webui must be able to replace tui at some time
* fixed up many older problems, yet ready for libp2p migration for communication over net
* cleaned up yet inactivated parts: former ssh connection, state machine replacement
* introducing a nice way to sync threads on startup by creating a channel, send its sender to main thread and block own thread until sender is sent back to self controlled receiver.
* trying upx in CI builds again
* migrated first too many native threads to async green threads, as also most dependant external crates use more general futures approach. It's yet a bit confusing and inconsistant but problem were rayon thread iterator and cursive as thread dependant. But I am about to like and understand async/await quite well, also the consequences for embedded development :grin:
* bumped to Rust 2018 features async/await in net module using futures in few occasion, but will continue with that
* version changes of different used crates
* changed this README, to add version changes to *Changes*, re-ordered, and made [Goals](#goals) as a check list, bumped version to v
* updated various dependency packages from Rust libraries [actix](https://actix.rs/) (there was a dependency lock for a longer time) to [bootstrap](https://getbootstrap.com/), and [jquery](https://jquery.com).
* added small key generation documentation in `README-release.md`
* preparing cross compile for banana pi (Dockerfile)
* added and will later moving from [travis](https://travis-ci.org/) to [AppVeyor](https://www.appveyor.com/) which doesn't lack bionic 18.04 builds and possibility for Windows builds (AppVeyor was primarily used for Windows builds, Travis for Linux in Rust, so documentation for Linux and Rust builds in AppVeyor is quite bad ... testing)
* cleaned up changes list, switched to [sublime text](https://www.sublimetext.com) instead of [atom](https://atom.io/) as editor
* trying xargo instead of cargo to compile (possible problems with std in cross compiling, and optimizations), but only works with nightly.
* cleaned up deployment, added a release readme, licence to deploy as well
* repaired deployment of binaries, including stripped and packed binaries (upx optimization doesn't work somehow)
* included (should work fully offline later, all MIT licensed) 3rdparty css, js-scripts ([jquery](https://jquery.com),[bootstrap](https://getbootstrap.com/)) and all pages hard-included in to webserver (no loading of files, yet for development still possible), added state for server, connected websocket, designed a favicon plus logo
* added basic webui support: http-server with websockets ([actix](https://actix.rs)), a single page application, the page and websockets are already there.
* added architecture graphics using [draw.io](https://draw.io), which is awesome. Also connectable by [github support](https://about.draw.io/github-support/) directly via [this](https://www.draw.io/?mode=github) ([howTo](https://github.com/jgraph/drawio-github)).
* state machine not yet used (need to think more about "futures" architecture and understand futures and how to combine)
* the client ssh connector (com_client) is behind a state machine (to have reconnect and similar easily)
* replaced [id3](https://github.com/jameshurst/rust-id3) with [taglib](https://github.com/ebassi/taglib-rust/) (more external libs, but many more available media tags). Unfortunately it took me quite some time to find some strange difference (didn't work) between [crates.io](https://crates.io/crates/taglib) and original [github.com](https://github.com/ebassi/taglib-rust/) version, so I had to use the git pull rather than the convenient crate.io dependency usage in `Cargo.toml`.
* I suspended the usage of [trust](https://github.com/japaric/trust) which uses [cross](https://github.com/japaric/cross), since the develop cross compiling docker images are based on ubuntu 12.04 (deb jessie), and the libsodium, libavahi uses ubuntu ppa from newer versions. I might even go to xenial (deb stretch), then both libs are included by default. But I would have to create my own dockerfile for that, and not just extend the well prepared dockerfiles from cross. :unamused:
* [documentation](https://electricherd.github.io/audiobookfinder/audiobookfinder/index.html) deployed, awesome: Rust + github + travis +... (needs javascript enabled)
* applied single test file for travis run: took Bach's Toccata And Fugue In D Minor by Paul Pitman (licence PD)  [orangefreesounds](www.orangefreesounds.com/toccata-and-fugue-in-d-minor/) in rememberring [Monthy Python's grand rugby match](https://www.youtube.com/watch?v=HKv6o7YqHnE).
* travis CI working
* more documentation locally as html: `cargo doc --no-deps --open`
* file logging in (use [glogg](http://glogg.bonnefon.org/))
* logging mechanism introduced (`logit.rs`). It was needed because of tui console output was not readable (either syslog or console)
 * run e.g. with `ADBF_LOG=file RUST_LOG=adbfbinlib::net=debug RUST_BACKTRACE=full cargo run -- -n ~/Audiobooks`
* ssh client with example key works, key now external
* found emojis :grin:

Screenshots

ToDo

leave a good, well defined version
create secret by github if also usable by other build systems
fix build systems github actions, appveyor, travis (if still possible)
~~remove TUI, since it is not working properly~~
add kademlia push of android data found to network to all clients for adbfflutter
add signature key, and google developer key to android app, and some time later process for a correct ios app
try crate vfs for unit test with files!! interesting and needed!
look for other tag libraries (e.g. symphonia-metadata symphonia)
a good and fast data collection with few more further lifetimes optimizations
add a nice way to collect data (string distance + time, maybe 2nd duration hash set?, filter empty tags)
add memory consumption monitoring for collection - started for BKTree
look into state and extra data usage of already used smlang-rs
think of a protocol what adbf clients agree on and exchange (e.g. still searching, files found, etc)
maybe a little AI layer on determining audio books duplicates/same author by similar spelling, etc.
internationalization (which is not really supported yet by Rust)
~~flutter app:~~
- ~~start network thread on start~~
- ~~add list of found peers~~
- ~~add UiUpdateMessages somewhere~~
~~fix github actions CI for automatic build~~
~~implement as android/ios app using flutterust.~~
~~change webui to be started without collection start, to be able use path selection from within webui later~~
~~add ctrlc functionality for signal handling in main~~
~~let state machine talk (as ipc) with data collection via crossbeam (first only the finish search status)~~
~~make div from html page to extra single file for later multiple clients on one page~~

CI

The Continuous Integration is done on 2 services, Travis and AppVeyor but will probably once completely moved to AppVeyor because Travis recently only had old LTS 16.04 images, and no possible Windows compilation (they are working on it), so there is:

On Appveyor
On Travis
- Audiobookfinder Build Ubuntu 16.04
- Audiobookfinder Build Debian Buster Raspberry
- only until version 0.1.18: Build Linux 14.04
On Github with Github Actions
- Android-Build:
- iOS-Build:
- MacOS-Build:

Goals

The primary goal is to learn Rust and to cover various aspects of the language, of which some of I already used inside the program, such as:

Dependencies

no non-Rust libraries, it's crossplatform now 😊

Tools

Editors:
- IntelliJ IDEA, and then Rust plug-in: easy, refactoring, spell-check, nice, first choice now, because of easy type look-up, and other good features
- sublime text is good and fast, setup was ok, using it now, works very well
- atom was for a long time my choice for development, on my Eee Pc sublime, because of small footprint and performance, but now that is too slow though I really like the Git feature of it, has README.md syntax
Logging:
- glogg a good logger on linux - since log has a coloring problem it still works pretty good
Git:
- gitui very good
- gitahead I like that
Documentation:
- drawio-desktop nice
- mermaid can be used with drawio and github in a very natural way

Useful links

https://jsfiddle.net/boilerplate/jquery : for people who don't really like js but need it:
https://thoughtbot.com/blog/json-event-based-convention-websockets : websockets to js commands
https://github.com/Ragnaroek/rust-on-raspberry-docker : headachefree compiling for raspberry pi locally with docker
https://learning-rust.github.io/docs/a5.comments_and_documenting_the_code.html howto documentation
https://github.com/actions : a better CI with repo of scripts

electricherd/audiobookfinder