/modb-app

The tool that creates the content of anime-offline-database.

Primary LanguageHTMLGNU Affero General Public License v3.0AGPL-3.0

Tests codecov jdk21

modb-app

modb stands for Manami Offline DataBase. The applications and libraries of this repository are used to create the manami-project/anime-offline-database. Don't use these libraries and applications to crawl the websites entirely. Instead check whether the dataset already offers the data that you need.

  • analyzer: Allows to review the entries of the dataset and create merge locks.
  • anidb: Config, downloader and converter for anidb.net
  • anilist: Config, downloader and converter for anilist.co
  • anime-planet: Config, downloader and converter for anime-planet.com
  • anisearch: Config, downloader and converter for anisearch.com
  • app: The application that runs the crawlers, merges anime and updates the repository.
  • core: Core functionality used by all other modules.
  • kitsu: Config, downloader and converter for kitsu.app
  • lib: A library that drives the applications "app" and "analyzer".
  • livechart: Config, downloader and converter for livechart.me
  • myanimelist: Config, downloader and converter for myanimelist.net
  • notify: Config, downloader and converter for notify.moe
  • serde: Serialization and deserialization of the finalized dataset files.
  • simkl: Config, downloader and converter for simkl.com as well as config for animecountdown.com.
  • test: All essential dependencies as well as some convenience functions and classes for creating tests.

Documentation

Requirements

  • JDK/JVM 21 (LTS) or higher
  • Linux/Unix system supporting ifconfig
  • ipv6 based internet connection with SLAAC enabled

Getting started

Setup is identical for app and analyzer.

Setup configuration

Create a configuration file. Set all the properties from the "Configuration" section down below which don't offer a default value.

Optional: Logback configuration

Optionally you can create a logback configuration to override the default setup.

Start using IDE

Run main() in io/github/manamiproject/modb/app/App.kt of the app module or io/github/manamiproject/modb/analyzer/Analyzer.kt of the analyzer module with the following VM parameter:

  • -Djava.net.preferIPv6Addresses=true
  • -Djava.net.preferIPv4Stack=false

Start using *.jar file

Run

  • either java -Djava.net.preferIPv6Addresses=true -Djava.net.preferIPv4Stack=false -jar modb-app.jar
  • or java -Djava.net.preferIPv6Addresses=true -Djava.net.preferIPv4Stack=false -jar modb-analyzer.jar

Configuration

For more configuration options see the README.md files of the respective modules.

parameter type default description
modb.app.downloadsDirectory String - Root directory in which the raw files and converted files are stored.
modb.app.outputDirectory String - Target output directory. Normally this should be the directory in which you cloned the anime-offline-database
modb.app.downloadControlStateDirectory String - Root directory of download control state files.
modb.app.logFileDirectory String A directory called logs within the working directory of the current week. Defines the directory in which the logs saved.
modb.app.keepDownloadDirectories Long 1 Number of download directories to keep. Download directories contain both raw data and conv files (intermediate format). Default is 1 which means that only the most recent download directory is kept.