Get realtime output from `restic`
andreasnuesslein opened this issue · 5 comments
andreasnuesslein commented
show the restic process while running instead of just printing the result in the end.
ozboss commented
Exporting the progress of a backup would be amazing.
andreasnuesslein commented
i've been throwing many hours into this but I guess my pipe/stdout/stdin/dialog/whatnot-fu was just not good enough. even trying to use the most low level libs in python i was not able to capture the restic output "live". it just kept buffering, every time.
so if you have some insight @ozboss .... ? :)
ozboss commented
So I had a go at writing my own script and basically what I do is this:
...
proc = subprocess.Popen(command, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
if task == "backup":
while True:
output = proc.stdout.readline().decode()
if output == '' and proc != None:
break
if output:
metrics = parse_output(output)
create_and_write_textfile(metrics)
else:
proc.wait()
output = proc.stdout.read().decode()
metrics = parse_output(output)
create_and_write_textfile(metrics)
...
So I write a separate textfile for every command and during backup I just keep rewriting the backup-textfile.
This can probably be done more elegantly but it works flawless so far.
During backup the "backup-file" contains following status messages:
_restic_backup_status = """
# HELP restic_backup_percent_done Percent done
# TYPE restic_backup_percent_done gauge
# HELP restic_backup_total_files Total number of files to backup
# TYPE restic_backup_total_files gauge
# HELP restic_backup_files_done Number of files processed
# TYPE restic_backup_files_done gauge
# HELP restic_backup_total_bytes Total size in bytes to backup
# TYPE restic_backup_total_bytes gauge
# HELP restic_backup_bytes_done Size in bytes processed
# TYPE restic_backup_bytes_done gauge
restic_backup_percent_done{{config="{name}",repository="{repository}"}} {percent_done}
restic_backup_total_files{{config="{name}",repository="{repository}"}} {total_files}
restic_backup_files_done{{config="{name}",repository="{repository}"}} {files_done}
restic_backup_total_bytes{{config="{name}",repository="{repository}"}} {total_bytes}
restic_backup_bytes_done{{config="{name}",repository="{repository}"}} {bytes_done}
restic_backup_duration_seconds{{config="{name}",repository="{repository}"}} {duration_seconds}
"""
Once it's done it contains the summary:
_restic_backup_summary = """
# HELP restic_backup_files_new Number of new files
# TYPE restic_backup_files_new gauge
# HELP restic_backup_files_changed Number of changed files
# TYPE restic_backup_files_changed gauge
# HELP restic_backup_files_unmodified Number of unmodified files
# TYPE restic_backup_files_unmodified gauge
# HELP restic_backup_dirs_new Number of new dirs
# TYPE restic_backup_dirs_new gauge
# HELP restic_backup_dirs_changed Number of changed dirs
# TYPE restic_backup_dirs_changed gauge
# HELP restic_backup_dirs_unmodified Number of unmodified dirs
# TYPE restic_backup_dirs_unmodified gauge
# HELP restic_backup_data_blobs Number of data blobs
# TYPE restic_backup_data_blobs gauge
# HELP restic_backup_tree_blobs Number of tree blobs
# TYPE restic_backup_tree_blobs gauge
# HELP restic_backup_processed_files Number of processed files
# TYPE restic_backup_processed_files gauge
# HELP restic_backup_processed_size_bytes Processed size bytes
# TYPE restic_backup_processed_size_bytes gauge
# HELP restic_backup_processed_duration_seconds Backup processed duration in seconds
# TYPE restic_backup_processed_duration_seconds gauge
# HELP restic_backup_added_to_repo Number of files added to repo
# TYPE restic_backup_added_to_repo gauge
# HELP restic_backup_duration_seconds Backup duration in seconds
# TYPE restic_backup_duration_seconds gauge
restic_backup_files_new{{config="{name}",repository="{repository}"}} {files_new}
restic_backup_files_changed{{config="{name}",repository="{repository}"}} {files_changed}
restic_backup_files_unmodified{{config="{name}",repository="{repository}"}} {files_unmodified}
restic_backup_dirs_new{{config="{name}",repository="{repository}"}} {dirs_new}
restic_backup_dirs_changed{{config="{name}",repository="{repository}"}} {dirs_changed}
restic_backup_dirs_unmodified{{config="{name}",repository="{repository}"}} {dirs_unmodified}
restic_backup_data_blobs{{config="{name}",repository="{repository}"}} {data_blobs}
restic_backup_tree_blobs{{config="{name}",repository="{repository}"}} {tree_blobs}
restic_backup_added_to_repo{{config="{name}",repository="{repository}"}} {data_added}
restic_backup_processed_files{{config="{name}",repository="{repository}"}} {total_files_processed}
restic_backup_processed_size_bytes{{config="{name}",repository="{repository}"}} {total_bytes_processed}
restic_backup_processed_duration_seconds{{config="{name}",repository="{repository}"}} {total_duration}
restic_backup_duration_seconds{{config="{name}",repository="{repository}"}} {duration_seconds}
"""
```