existing_remote_files to keep X previous builds

Feature Request/Discussion

Keep a number of previous build's files on hand for things like rollback and zero downtime.

Current Solution

existing_remote_files = 'delete' will immediately delete old assets after uploading new ones.

Problems of Current Solution

Our deploy process uploads the new assets but it takes about 2 minutes before the first container can rotate and 6 minutes before the last one gets changed out based on health.

This potentially leaves old containers referencing deleted assets.

Benefits of Proposed Solution

No downtime between syncing assets and rotation of code/containers.
Ability to rollback to previous builds/old-assets.

Thanks for this great gem!

How about adding a task/API to let you delete unused assets
Then you can use existing_remote_files = 'keep' and delete unused assets afterward
And you can delete unused assets whenever you like

asset_sync/lib/asset_sync/storage.rb

Line 318 in cc9d971

delete_extra_remote_files unless keep_existing_remote_files?

How about adding a task/API to let you delete unused assets
Then you can use existing_remote_files = 'keep' and delete unused assets afterward
And you can delete unused assets whenever you like

Yeah a rake task works. It would need something like RUNS_TO_KEEP=3 to maintain the history for rollbacks.

History could be stored in remote_file_list_cache_file_path?

{ 'TIMESTAMP': [ files... ], 'TIMESTAMP_2': [ files... ] }

I mean keeping remote files with existing_remote_files = 'keep'
And use new rake task / API which calls delete_extra_remote_files when you want to

As an automated solution, that will not work for our case. Removing the ability to rollback is not an option we want to lose because we won't know, automatically, when it is safe to delete the rollbacks.

It's the failsafe we never use until that one time.

Currently, we use keep and manually log into a console to run the delete function, which works well enough.

I imagined an automated solution would be ideally be similar to rake assets:clean that could be run with 100% certainty.

Unless sprockets provide a stable API to return files in N versions
Otherwise I doubt we would be supporting that directly

But maybe a new options can be added to be used in delete_extra_remote_files

asset_sync/lib/asset_sync/storage.rb

Lines 158 to 175 in cc9d971

    
           def delete_extra_remote_files 
        
             log "Fetching files to flag for delete" 
        
             remote_files = get_remote_files 
        
             # fixes: https://github.com/rumblelabs/asset_sync/issues/19 
        
             from_remote_files_to_delete = remote_files - local_files - ignored_files - always_upload_files 
        
             log "Flagging #{from_remote_files_to_delete.size} file(s) for deletion" 
        
             # Delete unneeded remote files, if we are on aws delete in bulk else use sequential delete 
        
             if self.config.aws? && connection.respond_to?(:delete_multiple_objects) 
        
               from_remote_files_to_delete.each_slice(500) do |slice| 
        
                 connection.delete_multiple_objects(config.fog_directory, slice) 
        
               end 
        
             else 
        
               bucket.files.each do |f| 
        
                 delete_file(f, from_remote_files_to_delete) 
        
               end 
        
             end 
        
           end

Probably a proc that returns a list of file keys/paths when delete_extra_remote_files is called
Those keys/paths returned will be removed from from_remote_files_to_delete and they won't be deleted
Then you can use whatever Sprockets documented or not documented methods to generate that list (and you are in charged of updating the proc when API updated / broken of course)
What do you think?

Definitely understood that supporting this feature directly isn't something we'd want to take on.

Looking at the Sprockets clean and the Webpacker clean, not really accessible API. Solved by: Copy/paste of code... ew?

Anyway, feel free to close this. Thanks!

If you can come up with a PR or there are more people needing this I can introduce it more quickly
Otherwise I am busy at work usually so only have limited time to open source projects
And I can't test the function myself coz I don't use this gem anymore OTL

Admittedly, we've moved off of this gem a while ago partly for this reason. A consideration was either going back to storing assets in the container itself or develop this feature.

Sadly, with it being simpler to go back to container based assets, we won't be coming up with a PR.

I did want to make the suggestion on the off chance there was interest and it could help others.

I did moved away from this gem to container hosted, CDN cached structure too

OK unless there are more people having this issue I am closing this 😃

@nitsujri we were in a similar situation since we perform rolling deploys. We started with asset_sync and took a lot of inspiration from it and then ended up rolling our own tool. We just open sourced it - https://github.com/Loomly/s3_asset_deploy. Hopefully it's helpful!

	def delete_extra_remote_files
	log "Fetching files to flag for delete"
	remote_files = get_remote_files
	# fixes: https://github.com/rumblelabs/asset_sync/issues/19
	from_remote_files_to_delete = remote_files - local_files - ignored_files - always_upload_files

	log "Flagging #{from_remote_files_to_delete.size} file(s) for deletion"
	# Delete unneeded remote files, if we are on aws delete in bulk else use sequential delete
	if self.config.aws? && connection.respond_to?(:delete_multiple_objects)
	from_remote_files_to_delete.each_slice(500) do \|slice\|
	connection.delete_multiple_objects(config.fog_directory, slice)
	end
	else
	bucket.files.each do \|f\|
	delete_file(f, from_remote_files_to_delete)
	end
	end
	end