Sync to S3 always copies all files

[willkoehler]

When syncing to S3, all files are copied each time. Deleted files are not cleaned up. I'm not sure if the fix needs to be in middleman-sync or asset_sync.

The problem is get_remote_files() in lib/asset_sync/storage.rb always returns a blank list of files. This can be fixed by commenting out the :prefix option in bucket()

def bucket
  # fixes: https://github.com/rumblelabs/asset_sync/issues/18
  @bucket ||= connection.directories.get(self.config.fog_directory)#, :prefix => self.config.assets_prefix)
end

The :prefix option is obviously needed in other situations, so a solution upstream is needed. Maybe something with middleman-sync options.prefix = "**"? I'm getting over my head at this point.

I'm happy to dig further and submit a pull request, but I need some guidance. Thanks.

[willkoehler]

Just noticed that my "fix" has an undesired side effect. New files are uploaded to S3, deleted files are cleared from S3, but changed files are not uploaded. Looking at the code in asset_sync, I can't see where it would have the ability to detect changed files. Maybe uploading all the files each time was the intended behavior?

[irvinebroque]

+1

[stationkeeping]

+1

[willkoehler]

It looks like the fundamental design of asset_sync (the gem that middleman-sync uses to synchronize with S3) conflicts with the needs of middleman-sync. This is because asset_sync is designed to synchronize precompiled Rails assets with S3. Specifically:

• asset_sync does not detect when the content of a file changes
• asset_sync sets far-future cache headers on all files (including HTML files).

All this makes sense in the context of asset_sync's mission but does not work when synching middleman apps with S3.

@fredjean has built a new gem: https://github.com/fredjean/middleman-s3_sync that is specifically designed to sync middleman apps with S3. It handles new files, deleted files, and changed files. It also sets cache headers correctly. I've been using it several months now and it has worked flawlessly.

[karlfreeman]

@willkoehler +1, Im currently in the process of re-writing an asset_sync alternative so fingers crossed I'm able to swap out asset_sync in the future.

[willkoehler]

@karlfreeman you might want to review the work that @fredjean is doing in middleman-s3_sync. He has a nice solution for pushing middleman projects up to S3 and has solved all the difficult problems. You guys might want to sync up your work (no pun intended :) )

[karlfreeman]

@willkoehler #16 :) Hopefully we'll organise something soon.