Dealing with attic issues
maltefiala opened this issue · 63 comments
Here is a list of all open issues in attic, acquired with this dirty python script . I suggest we go through all of them and tick them when fixed.
- init: missing parameter to avoid error if repository already initialized
- separable repository format
- ssh:// after network re-connection and SIGINT issues
- Suggestion: Allow multiple servers to back up to the same repo at the same time
- Add metadata search index for more responsive FUSE
- Force-include the specified paths in 'create'
Done / tracked here / invalid / out-of-scope:
- Update installation instruction in README
- Cannot build with pip because of EVP_CIPHER_CTX error
- transaction recovery issue?
- Attic stores hard link to inaccessible file
- attic mount doesn't work
- build on fedora
- safe to interrupt backup?
- Has anyone got working on windows
- Cannot build with pip because of EVP_CIPHER_CTX error
- Attic stores hard link to inaccessible file
- File search Request
- Allow 'list' while backup is in progress
- --include switch
- Expose External Crypto providers
- New security changes broke remote access to repos
- Incorrect directory permissions during extract
- where shall parameters get stored?
- Flexibility and cleanup
- "extract" command has no destination switch?
- Has anyone got working on windows
- Repository clone command?
- Parallel backups from different hosts crash attic
- Cannot mount due to llfuse error
- ZFS snapshot ingestion support
- Let attic know it should wait before beginning
- aes-gcm support
- aes-gcm or aes-ocb?
- exclusive=x flag and RemoteRepository
- "Initializing cache..." does not save progress
- I/O Error on CIFS Mounts?
-
undefined symbol: EVP_aes_256_ctrerror when runningattic - draw a picture from the internals documentation
- How to list files corresponding to corrupted data segments?
- attic-mount misses files that can be extracted through attic-extract
- PermissionError
- setup.py: seek
/opt/pkg(pkgsrc OS X) for openssl - Unable to download precompiled binaries
- Use environment variable to set the tmp directory
- msgpack.exceptions.UnpackValueError
- attic check with ATTIC_PASSPHRASE -- getpass.py/terminal issue
- non-zero exit status -11
- SSH and Ctrl-C leave attic serve instances alive and prevent further upload
- Typo: fileystem -> filesystem
- Parity check/recover considered?
- Can't subscribe to mailing list
- Installation problems on CentOS 6.7
- Locally backup S3 buckets
- Tests failing on Gentoo Linux
- Debian Jessie installation instructions for pip3
- acl_use_local_uid_gid has suspicious code
- Cache location
- Data integrity error after running out of disk space
- OverflowError when attempting to create a backup
- attic crashed with "malloc failed"
- ValueError on attic check --repair
- Error with UnicodeDecode
- Conversion from rdiff-backup
- // , Installation fails on CEntOS 7.1.
- // , gpg key for attic-backup.org
- Default repository
- low-impact - lrucache code appears to leak fds without closing them
- Optimization: asynchronous writeback
- repository: two minor improvements
- Tiny fixes: printf error messages and gitignore
- attic remote backup ends in broken pipe
- attic/_version.py from github releases change over time
- Attic segfaults
- Let attic take a remote path option for cases where attic cannot be installed in standard places (like rsync, git, etc)
- Honor nodump flag
- osx pip3 install Attic - command not found
- Allow to ignore file by size
- Make the script for automatic backup finding the hostname
- --include feature
- OverflowError: value too large to convert to int
- Undocumented standalone binaries
- Undocumented environmente variables
- Restore remote data inplace with deduplication or rsync algorithm
- Add --files-from and --files-from0 options
-
createflag to print backed up files - "Failed to acquire write lock" FUSE
- int32 chunk reference counter, is it a problem?
- merge multiple repos into one
- Exit gracefully when output pipe is broken
- Performance: attic create stalls while index is read back
- typo in docs theme css
- st_mtime_ns precision or rounding related test failure
- memory usage is too high
- Question: list files modified/added in archive
- Exception: hashindex_set failed (malloc failed)
- Exception raised during create
- Trailing slash on "init" confuses attic
- Add disk space preview command
- speed / compressed tradeoff parameter?
- Doc symlinks are not followed on
attic create. - sigusr1 / sigint signal handler to print current file infos
- implement --timestamp, iso8601-like utc timestamp string or reference file
- simple sparse file support, fixes #256
- crc32 computation partly redundant?
- use posix_fadvise to avoid spoiling the OS cache
- Documentation for 'attic serve'
- attic serve is not documented on attic-backup.org
- Question about subsequent backups, and about pruning
- improve and simplify key modes
- potential counter/IV uniqueness issue
- use poly1305-AES for faster MACs
- attic create -n/--dry-run
- ACLs do not work when using attic mount
- Data integrity error due to index mismatch for key
- Specify the date and time of the backup
- Remove a path prefix on backup and/or restore
- Specify the "extract to" location
- improve restores by posix_fallocate
- attic does not backup UNIX domain sockets
- upgrade versioneer so it is pep440 compliant
- handle sparse files
- Adds a note about the ACCDCC2D signing key to the installation documentation
- Fix test setup for 32bit platforms, partial fix for #196
- wrong code in /bin/attic script
- make system more responsive by using the fadvise DONTNEED
- implement attic rename repo::oldname newname
- implemented create --progress
- own lrucache?
- catch all Exceptions, transmit traceback back from remote
- fix some exception handlers
- fix noatime mode, fixes #243
- atime preservation code is broken
- Refactor 'list' and 'prune' operations on remote repositories to run on the server side
- fix misleading hint the fuse ImportError handler gave, fixes #237
- misleading error msg when importing fuse support fails
- give specific path to xattr.is_enabled(), disable symlink setattr call ...
- Traceback when running 'check --repair' on a repository
- check --last N
- avoid defect python-msgpack releases, fixes #171, fixes #185
- check unpacked data from RPC for tuple type and correct length, fixes #127
- improve on ready-only targets, see issue #202
- Delete repo
- better attic create -v output
- add global option --no-cache-files to lower memory consumption
- fix traceback when trying to do unsupported passphrase change, fixes #189
- datetime does not like the year 10.000, fixes issue #139
- Implement mid-file checkpoints?
- improve exception handling / diagnosis
- helping out with merge reviews and issue triage
- compatibility policy / support timeframe
- cleanup crypto.pyx, make it easier to adapt to other modes
- OpenSSL's EVP_* interface not used correctly?
- version number creates wrong impression
- hash performance sha256 vs. sha512
- stdin and stdout support
- Misc. Fixes / Improvements
- Error while creating backup - 'Repository' object has no attribute 'segments'
- Document source of python 3.4 libraries bundled with releases
- More user friendly message if repository is on read-only storage
- attic hangs when initializing repository on a cifs mount
- Attic mount dies (Input/output error) when backgrounded
- Win32
- Extra debug information for 'fread failed'
- attic create fails with "assert segments[segment] == 0"
- How to delete or rename a repository
- add more .c files to .gitigore
- Retry on Input/Output error
- Changing passphrase fails: AttributeError: 'PassphraseKey' object has no attribute 'change_passphrase'
- Changing/removing keyfile password?
- FUSE: reflect deduplication in allocated blocks
- attic does not respect XDG_CACHE_HOME
- Creating archive with a slash breaks mount option
- Suggestion: attic check --last N
- extended attributes in 0.14 / linux
- KeyError: 'cache'
- attic check --repository-only reports "Error reading segment ###" at different locations
- Pull mode
- Enhancement: Add archive diff command
- Only allow whitelisted RPC calls in server mode
- Some file operations not working correctly in mounted repo
- Issue with modification dates of parent directories in mounted archives
- Traffic needed for 'list' and 'prune' operations on a remote repository increases linearly with time
- Integrity error: 300k files 100GB of data.
- Restrict "serve" command to "create" operation only
- Traceback on a frequent backup
- list changed files in info
- Attic leaks file descriptors during checkpoint
- document that multiple clients can backup to a single remote
- document Internals better
- document better different dependencies
- document that hardlinks and symlinks are supported
- Normalize source/exclude paths before matching
- Suggestion: attic info --latest
- Suggestion: Add a comment string to an archive
- Suggestion: Mark a specific archive as "never prune"
- Error backing up a changing filesystem
- Implement --exclude-if-present
- Specifying date or ability to change it.
- attic list bombs with "ValueError: year is out of range"
- Mounting a entire repo is not working
- Support for Amazon S3?
-
pip installon Ubuntu 14.04 fails - FUSE 2.8.0 required?
- Document that hard links work
- Prune help unclear about what it will delete
- Attic doesn't handle connections issues when using remote repo
- --exclude-if-present
- TypeError when trying to connect to a remote repository
- Cache is newer than repository, refusing to continue
- UnicodeEncodeError
- cifs destination exception: listxattr failed
- attic freezes sometimes
- Subkeys
- Add a interface to report current progress
- Include other compressors like XZ for smaller size and LZO for speed
- The "self contained" builds are missing the llfuse dependency
- Add bandwidth limit support
- Slow backup speed (~12MB/s)
- Support custom ssh binary / ssh options?
- description of exclude wildcards
- Backup speed getting worse after cleanup
- Copy repository to secondary location
- Fixed buffer size a waste of resources/overhead?
- Speed up directory scanning by ordering by inode
- options to not compare inode numbers, and disable acl / xattr
- Re-encrypt a repository with a new key
- Convert unencrypted repositories to encrypted repositories
- Document at "Usage->Attic" that init remote need installation on remote
- create directories for mount points
- docs bounty: let's crowdfund the attic docs wrinting ( 100 usd is my personal pledge. )
- Poor FUSE performance
- Windows Support
- Attic and large files
- Documentation on how I used Attic Backup (full system backup)
- print current status/filename on
SIGUSR1in non-verbose mode - More informative "attic create -v" output
- Document exit code(s)
- Use logging lib for messages
- Document known file cache and add an option to disable it
- Add option to disable file chunks cache to reduce memory usage
- Document how repository encryption works
- Document how deduplication works
- Document repository format
- BlockingIOError in verbose mode
- Archive stdin or fifo
- config file whis tasks
I ticked all the stuff that's currently in borg master branch. There is some more stuff already fixed/improved that is not merged into there yet.
I am also ticking the items now that I think should be closed (because duplicate, irrelevant, stale, not enough infos, ...).
I'm also ticking the stuff that has own issues in the borg issue tracker.
about jborg/attic#252 (fadvise DONTNEED): it is merged (and AFAICS using fadvise DONTNEED had positive effects).
https://github.com/borgbackup/borg/blob/master/borg/_chunker.c#L160
People seem to disagree whether that is beneficial or not.
I'll reconsider as soon as facts or at least good reasoning is available that indicates using it is wrong.
about pull mode: it is ticked because we have an issue (which is a bit more general) in our issue tracker: #36
it would have been nice to clarify which tickets are resolved, in the issue tracker or ignored instead of just ticking them off here... now we don't clearly know what's what...
@anarcat I see your point, but I also don't want to make it even more work.
I mainly see the unchecked attic issues as stuff that has still to be looked at (and maybe dealt with), so that one does not have to go through all the issues there again and again. Of course we also need to deal with the issues in borg issue tracker.
What's really fixed can be seen more easily in CHANGES.rst.
jborg/attic#99 implemented through #248
jborg/attic#181 implemented through #247
After doing a large volume test myself (see #216) and not finding issues, I'll check these ones as "can't reproduce" now:
Note: I put a checkmark in that (native) "windows support" attic issue. Not because we have native windows support (only cygwin), but because the related changesets there are rather messed up.
jborg/attic#18 has been reopened in #315
jborg/attic#70 is 404 (huh? how can that happen?)
but as we recently improved FUSE performance, I've just checked it as solved.
i checked jborg/attic#367 in the list as we track this in #225.
Hi
In the list there is a point named "Pull mode" which is ticked. That is great since it is exactly what I need (schedule and manage backups from one central server). Unfortunately I can't find any hint in the docs how to make backups in pull mode.
so does that mean it is fixed/implemented but I am missing the docs or is it ticked by mistake?
(I would love to see pull mode implemented)
@pacman88 see #5 (comment) and #36
Hi,
I'm unable to run borg create as I'm running into #317 (something related to NFS). Would you like me to open a separate issue in this repo?
info:
Platform: OpenBSD orchid.home 5.9 GENERIC.MP#1854 amd64 amd64
Borg: 0.30.0 Python: CPython 3.4.4
@pyrohh it's unclear what your issue is. sure open a new ticket and provide enough information so we can reproduce.
@ThomasWaldmann I'll do that, and I meant to link to jborg/attic#317, not #317 in this repo.
@pyrohh ah :) but i doubt it is same issue as that one as we do not use fcntl locking any more.
False alarm, everything works after a reinstall... must have been the boogie man. Anyway, this is such a nice piece of software, thanks for working on it :D
jborg/attic#103 checked, we have it now in ticket #661.
jborg/attic#304 i checked that as solved as we require py 3.4+ now and it only happened on 3.2.
I've checked the "win32" attic issue, see our "windows" branch.
checked jborg/attic#93 - the original question was answered "use rsync". in the discussion, the idea implemented by "borg with-lock subcommand" came, which we also have implemented now.
jborg/attic#380 fixed in borg by PR #1019.
jborg/attic#266 is now tracked as #1042 here.
IMHO could be ticked:
jborg/attic#362 (locally backup s3 issues) should use fuse
jborg/attic#323 (ssh:// after network re-connection and SIGINT issues) locking was improved on borg
jborg/attic#296 (Add disk space preview command) borg has --dry-run
jborg/attic#260 (improve restores by posix_fallocate) Fixed already in borg?
jborg/attic#227 (exclusive=x flag and RemoteRepository) This and several other issues are about parallel backups into the same repository. Borg has #1220 and some locking refinement discussions that try to solve this AFAIK.
jborg/attic#220 (Implement mid-file checkpoints?) We got that now! =D
jborg/attic#219 (aes-gcm support) TW PR, already in borg?
jborg/attic#216 (draw a picture from the internals documentation) Borg has extensive documentation with some sample ascii-art. Guess that can be also ticked
jborg/attic#207 (Flexibility and cleanup) TW PR, already in borg?
jborg/attic#211 (aes-gcm or aes-ocb?) TW issue, already in borg?
jborg/attic#210 (where shall parameters get stored?) TW issue, already in borg?
jborg/attic#131 (Parallel backups from different hosts crash attic) See above
jborg/attic#145 (Repository clone command?) According to #683 can be ticked
What we should test/discuss
jborg/attic#276 (Incorrect directory permissions during extract) Interesting and easy to test
jborg/attic#169 (Issue with modification dates of parent directories in mounted archives) same
jborg/attic#195 ("extract" command has no destination switch?) Do we want this?
jborg/attic#151 (Force-include the specified paths in 'create')
update
jborg/attic#110 : Borg has #210 and #773 for the lock timeout feature. As with all issues above that are about parallel operations (list while creating, create while mounted etc) we have #768
A few others simply need some discussion if this can be easily implemented and if it makes sense to implement. But I'm leaving that to others. :)
ticked PermissionError jborg/attic#381 fixed in 9ebc53a
ticked now: 362 296 260
323: unclear, we have recently documented some ssh settings though, so maybe could be checked
not fixed, still open: 216
in the works, PR against master/1.1 exists: 220
planned for 1.2, but not finished yet: 219, 211, 210
207: partially done (compression), crypto still todo
131: no crash (AFAIK), but a deadlock, not much better
145: rsync does not change the repo UUID (and that was the point of that issue)
(for transparency) jborg/attic#260
I'd tick "improve restores by posix_fallocate" because it has next to none positive impact on traditional FSes and (reportedly) negative impact on CoW FSes like ZFS and btrfs
Another very bad thing (tm) about fallocate is that if the FS doesn't support it glibc implements the worst-possible fallback ever.
OK, ticked 260 off the list. @enkore can you add your comment to the attic ticket and suggest closing it?
checked attic 169 -> it is not designed to do that, should be closed there also.
checked attic 177 -> safe to say it's an "issue" outside attic. i.e. crash before the file was committed to disk and stuff like that. #1060 goes in the direction.
checked attic 372 - solved by borg check --repair since 1.0.4 (remember correct chunks) and 1.0.6 (list missing chunks, warn at extract time, fuse EIO, healing capability).
checked attic 197 - issues on openssl 1.0.0j, which we don't support
checked attic 107 - borg caches per-archive hashindexes locally, so it only needs to fetch metadata from new archives.
checked attic 100, quite old and no response there.
FYI, just noticed jborg/attic#219 is closed.
@JensRantil yes, but in borg, we'll do aes-gcm, likely in borg 1.2.
attic 120 is borg #1462.
checked attic 227 - in PR #1371 rpc api for exclusive flag was added, so remote and local repos are now more similar in how they lock. also, the deadlock was fixed.
checked attic 383 - can be solved with borg with-lock ...
checked attic 384, out-of-scope / too special.
attic 131 checked. It is a rather old ticket, the tracebacks do not contain enough information.
Also, this issue is likely related to repository locking which is very different in borg (mkdir-based and since 1.0.7 immediately-exclusive) than in attic (posix locks, lock upgrades).
Checked attic 387 - borg works on cygwin and there is an ongoing windows port in branch "windows".
Checked attic 145, is locally tracked in #1695 now.
attic 276 is now tracked locally in #1751.
attic 195 is now tracked locally in #1783.
attic 210 is implemented with Manifest.config and borg_security_dir. Maybe create a internals doc ticket there?
from tw: yes that would be good, also for the changelog.
attic 207 is long implemented in Borg :) The crypto stuff has it's own tickets (see above).
from tw: some is implemented, some will (hopefully) come with 1.2. but yes, we can check it as we have local tickets.
attic 182: fuse archives are cached via a tempfile in the page cache. Resolved?
from tw: no, attic 182 is about a search index to optimize access to the right chunks.
attic 117 resolved in attic (also: fuse versions view, fuse repository mount, borg diff).
from tw: you mean "resolved in borg"? yes, fuse versions help, although it is not a general "file search" functionality. it only helps if you know the precise path.
attic 110: better locking is now done, otherwise it's covered by #768 and the like
from tw: ok, similar enough.
attic 393 is handled by borg #2092.
attic 395 checked - borg can be built with openssl 1.1.
I'm closing this now. If we are missing something important here in the borg issue tracker, please file an issue here (after checking we do not already have one for it).