The -c option doesn't seem to be working correctly
Closed this issue · 12 comments
In order to only copy files added to our bucket in the last 24 hours, I'm running with the parameter: -c 1
Yet I've just added a file to our bucket, and run it again with this command, and no copy operations are happening (the file does not exist in the destination either).
The Last Modified timestamp for the file shows:
Wed Dec 04 16:05:41 GMT-800 2013
And as I write this it's 16:22 GMT-800 (4:22PM Pacific), so the Last Modified is definitely within the last 24 hours.
Incidentally, it would be awesome if the -c parameter could be specified with an indicator as to the period, and just defaulting to days if none is specified, for backwards compatibility. This would allow us to specify at a different granularity, such as just the last two hours: -c 2h
I wonder if it might be a timezone issue, e.g. comparing a GMT timestamp to a Pacific timestamp on accident.
@winzig I was unable to duplicate the bug, so I added some logging to see what the heck is going on. Can you run again with the "-v / --verbose" flag? You should see lines that read something like this:
pool-1-thread-2 INFO : org.cobbzilla.s3s3mirror.KeyJob - key some_key (lastmod=Fri Oct 12 10:30:01 PDT 2012) is older than 1 days (cutoff=Wed Dec 04 18:16:42 PST 2013), not copying
Could you let me know what that log line looks like for your file that should be getting copied but is not?
OK, in order to make it more reasonable, I added a prefix so that it would basically only be trying to copy this file that I just put into my source bucket, named /large/bd2d03d2-9087-4730-a143-9f0949b3a97c.jpg.
So I ran it first with -c 1 and it failed, and then ran it without -c 1 and am getting the same results. This error is especially strange since I am absolutely positive that the file is nowhere on the destination bucket!
c:\s3s3mirror>s3s3mirror.bat -m 1000 -t 1000 -v -p large/bd2d03d2-9087 source dest
main INFO : org.cobbzilla.s3s3mirror.KeyLister - added initial set of 1 keys
main INFO : org.cobbzilla.s3s3mirror.MirrorMaster - 1 keys found in first batch from source bucket -- processing.
main INFO : org.cobbzilla.s3s3mirror.MirrorMaster - Lister has no keys queued, but is not done, waiting and retry
pool-1-thread-1 INFO : org.cobbzilla.s3s3mirror.KeyLister - starting...
pool-1-thread-1 INFO : org.cobbzilla.s3s3mirror.KeyLister - No more keys found in source bucket, exiting
pool-1-thread-1 INFO : org.cobbzilla.s3s3mirror.KeyLister - KeyLister run loop finished
pool-1-thread-2 INFO : org.cobbzilla.s3s3mirror.KeyJob - Destination file is same as source, not copying: large/b
-9087-4730-a143-9f0949b3a97c.jpg
pool-1-thread-2 INFO : org.cobbzilla.s3s3mirror.KeyJob - done with large/bd2d03d2-9087-4730-a143-9f0949b3a97c.jpg
main INFO : org.cobbzilla.s3s3mirror.MirrorMaster - No more keys found in source bucket -- ALL DONE
Thread-1 INFO : org.cobbzilla.s3s3mirror.MirrorStats -
STATS BEGIN
read: 1
copied: 0
copy errors: 0
duration: 0:00:01
read rate: 54.151624548736464/minute
copy rate: 0.0/minute
bytes copied: 0 bytes
GET operations: 2
COPY operations: 0
STATS END
this output seems to indicate that the source file is in fact already present in the destination (with the same name, size and etag). for kicks, try to copy it from the destination back to the source, I'll bet you'll see that it's already there, for example:
s3s3mirror.bat -v -p large/bd2d03d2 -d test_prefix dest source
You should then have a file at the root level of your source bucket named test_prefix-9087-4730-a143-9f0949b3a97c.jpg
I saw the latest fix you checked in, downloaded and rebuilt, and I'm still seeing the issue described above. If I try to copy the file from src to dest with the prefix given, it tells me the file already exists on dest (which I don't believe is true). If I then run your test to see if I can copy it back over from dest to source, that fails as well:
c:\s3s3mirror>s3s3mirror.bat -v -p large/bd2d03d2 -d test_prefix dest source
main INFO : org.cobbzilla.s3s3mirror.KeyLister - added initial set of 0 keys
main INFO : org.cobbzilla.s3s3mirror.MirrorMaster - 0 keys found in first batch from source bucket -- processing...
main INFO : org.cobbzilla.s3s3mirror.MirrorMaster - Lister has no keys queued, but is not done, waiting and retrying
pool-1-thread-1 INFO : org.cobbzilla.s3s3mirror.KeyLister - starting...
pool-1-thread-1 INFO : org.cobbzilla.s3s3mirror.KeyLister - No more keys found in source bucket, exiting
pool-1-thread-1 INFO : org.cobbzilla.s3s3mirror.KeyLister - KeyLister run loop finished
main INFO : org.cobbzilla.s3s3mirror.MirrorMaster - No more keys found in source bucket -- ALL DONE
Thread-1 INFO : org.cobbzilla.s3s3mirror.MirrorStats -
--------------------------------------------------------------------
STATS BEGIN
read: 0
copied: 0
copy errors: 0
duration: 0:00:01
read rate: 0.0/minute
copy rate: 0.0/minute
bytes copied: 0 bytes
GET operations: 1
COPY operations: 0
STATS END
--------------------------------------------------------------------
@winzig Are you using the latest s3s3mirror? I just fixed a really stupid bug yesterday that would have manifested itself in just this way.
I am 99% sure that I have it, although I do have some issues building it on Windows that I haven't had time to sort through (I think just some tests fail due to how things are on Windows). I think next release should print a version number along with the usage parameters so we'll know for sure ;)
Good idea. The first log line now states the version, which I have just bumped to 1.1.0. Let me know if you still see this problem with this latest version.
OK I think I am now getting the new stuff, and the copy now seems to be working again, however two things I noticed:
.sh and .bat file weren't updated to point to the new 1.1.0 jar, so they kept loading the old version for me.
When I run it, it says:
main INFO : org.cobbzilla.s3s3mirror.MirrorMaster - version null starting
Oops. Missed some files in my last commit. It's now been updated and should print the correct version string.