/zpaq-backup-linux

Incremental directories backup with ZPAQ archiever and following check of archieves with Linux Shell script

Primary LanguageShell

About

ZPAQ is an open source command line archiver for Windows and Linux. It uses a journaling or append-only format which can be rolled back to an earlier state to retrieve older versions of files and directories (https://en.wikipedia.org/wiki/ZPAQ, http://mattmahoney.net/dc/zpaq.html).

This system is intended for incremental archiving and long-term storage of archives (databases) and verification of created archives in Linux OS.

The system includes:

  • Directory with "zpaq" application
  • Main script "zpaq-backup.sh"
  • File containing a list of databases "dbsList.txt"
  • Dynamically created directories:
    • "dbs_backup" - to store created archives
    • "dbs_extracted_for_check" to extract and check generated archives
    • "dbs_log" for logging and backup status

Installation

  • On the server (Linux OS) where the backup system will be used:
    • Fill in the list of directories (bases) for backup (absolute paths are required) in the dbsList.txt file. Example: 
      /tmp/dbs/db01
/tmp/dbs/dir02
    • Change the paths directories, files and utilities in the "paths" section of the "zpaq-backup.sh" file
    • Run "zpaq-backup.sh"

Work algorithm

  • zpaq-backup.sh script gets a list of directories (bases) from file "dbsList.txt"

  • Actions are performed for each directory:

    • File "checkMarker.txt" containing the current date in format YYYY-mm-dd is written to the directory
    • Directory is archived along the path dbs_backup/{dbName}/ with the archive name {dbName}_???.zpaq, where ??? - sequence number of the archive (001 - full archive, the following ones are incremental)
    • Upon completion of the archiving, the status of the operation is written to the "dbArchStatus" variable (-ARCH_OK- | -ARCH_FAIL-)
    • The file checkMarker.txt is removed from the directory
    • The archive is extracted along the path dbs_extracted_for_check/{dbName}
    • Upon completion of the extraction, the status of the operation is written to the dbExtrStatus variable (-EXTRACT_OK- | -EXTRACT_FAIL-)
    • The dbs_extracted_for_check/{dbName}/checkMarker.txt file is checked to see if it contains the current date in YYYY-mm-dd format. As a result, dbCheckMarkerStatus is set to (-CHECKMARKER_OK- | -CHECKMARKER_FAIL-)
    • A line like this is written to the backup logging file "dbs_log/dbsBackupLog.txt": 
      /tmp/dbs/!dbName! : -ARCH_OK- -EXTRACT_OK- -CHECKMARKER_OK-

  • As a result, the following structure is created in the dbs_log\dbsBackupLog.txt file:

/tmp/dbs/db01 : -ARCH_OK- -EXTRACT_OK- -CHECKMARKER_OK-
/tmp/dbs/dir02 : -ARCH_OK- -EXTRACT_OK- -CHECKMARKER_OK-
  • Checks for the presence of a sequence of characters in the file "dbs_log/dbsBackupLog.txt" (_OK- | _FAIL-)
  • Depending on the result, OK or FAIL is written to the final backup status file "dbs_log/dbsBackupStatus.txt"