/nabu

nabu is a digital media item management system that provides a catalog of audio and video items, metadata for these items, and information about the workflow status of the items.

Primary LanguageRubyGNU General Public License v3.0GPL-3.0

Nabu Catalog

Setting up your dev environment

Use direnv to add bin to your path

PATH_add bin

Bring up the environment

# Build the base image
docker compose build

# Install the gems
nabu_run bundle

# Bring up all the containers
docker compose up

This brings up the following containers

  • app - the rails app
  • search - Solr instance for search (dev + test)
  • proxyist - S3 proxy
  • db - mysql data base (dev + test)
  • s3 - s3 mock

You can then easily run all the standard commands by prefixing with nabu

nabu_run bundle install
nabu_run bundle exec rake db:create
nabu_run bundle exec rake db:schema:load
nabu_test bundle exec rake db:schema:load
nabu_run bundle exec guard # Test runner

Production

The application is designed to be deployed with containers into an AWS account using CDK

To bootstrap a new account

# Setup an AWS account and credentials as per your preferred method and set the environment to use it
AWS_PROFILE=nabu
REGION=ap-southeast-2
ACCOUNT=$(aws sts get-caller-identity | jq -r .Account)
cdk bootstrap aws://$ACCOUNT/$REGION

Deployment

Use CDK to deploy new code via docker as well as any infrastructure changes

cd cdk
cdk --profile nabu-stage diff nabu-appstack-stage
cdk --profile nabu-stage deploy nabu-appstack-stage

If necessary:

bin/aws/ecs_rake deploy:migrate
bin/aws/ecs_rake sunspot:reindex

Importing a production database into your development environment

bin/aws/ecs_shell bash -c 'mysqldump -u root nabu | bzip2' > ../nabu.sql.bz2
bzip2 -dc nabu.sql.bz2 | mysql -h 127.0.0.1 -u root nabu_devel
rake sunspot:reindex

Production Tasks

import archive files

RAILS_ENV=production bundle exec rake archive:update_files

check if all files that have been uploaded are ok:

cd /srv/www/nabu/current
RAILS_ENV=production bundle exec rake --trace archive:update_files > log/update_files.log

check if all *-CAT-PDSC_ADMIN.xml files exist and create if necessary:

cd /srv/www/nabu/current
RAILS_ENV=production bundle exec rake --trace archive:admin_files > log/admin_files.log

delete a collection with all its items:

cd /srv/www/nabu/current
RAILS_ENV=production bundle exec rake archive:delete_collection[PA1]


# New Ethnologue data

We use the following source locations
* https://www.ethnologue.com/codes/
* https://iso639-3.sil.org/code_tables/download_tables

Run the following rake task to import everything

``` bash
bundle exec rake import:ethnologue

This will

  • Add new countries and update names
  • Update country names
  • Add new languages and update names (Only Living languages)
  • Add mappings of language to countries
  • All existing languages that are retired are marked as such, incl name change.
  • Where name changes occurred items in CollectionLanguage, ItemContentLanguage, ItemSubjectLanguage are updated with the replacement language code.
  • Where splits happened, a message is printed.

OAI-PMH

OLAC available at:

The feeds that OLAC harvests:

Individual item:

RIF-CS available at:

testing:

gem install localtunnel
rbenv rehash
localtunnel 3000

use resulting server on an OAI repository explorer:

URLs to test:

The feed that ANDS harvests:

Test at ANDS:

Feed for a single collection:

Setup Secrets

aws secretsmanager list-secrets

aws secretsmanager put-secret-value --secret-id ARN --secret-string "{\"site_key\":\"***\", \"secret_key\":\"***\"}"

Upgrades

We should regularly make sure we are running the latest versions of third-party packages

# Ruby gems
nabu_run bundle outdated
nabu_run bundle update

# npm
nabu_run bin/importmap audit
nabu_run bin/importmap outdated

# Sentry
# https://docs.sentry.io/platforms/javascript/install/cdn/
vi app/views/layouts/application.html.haml