Add Support for "Overwrite" instruction in appspec.yml "Files" section
Closed this issue · 176 comments
Feature Request:
Would like the ability to have an "overwrite" instruction in the appspec.yml "Files" section.
Use:
In my initial testing, I wanted to clobber all files in /var/www/html with new files. This resulted in a failure because the files already existed. Specifically, the following failure: 2015-06-13 20:58:40 ERROR [codedeploy-agent(7589)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: RuntimeError - File already exists at location /var/www/html/index.html - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:113:ingenerate_normal_copy'`
Desired Behavior:
I'd like to be able to create an appspec.yml file with the following section:
files:
- source: web
destination: /var/www/html/
overwrite: true
Notice I've added the "overwrite" instruction, which is currently not offered.
Current Workaround:
I've been working around by utilizing the hook section to do file copies. The hooks/shell script workaround is sufficient, rendering this a lower priority feature request.
Hi,
Thanks for the feedback. I've added this feature request to our backlog.
I would also love this feature.
Add one more name to the list.
Of course we need this feature!
But it should not be simply overwrite: true -- it should allow for a few options:
overwrite: none (default)
overwrite: purge
overwrite: merge
Purge removes all files in the destination before performing copying the new files.
Merge leaves the destination as-is, adds the files from source and will replace an files in the destination that also exist in the source.
As for now I guess I need to stage my source files in a temp location and then run a script to achieve this. 🐼 😢
+1 for this feature as well.
Side note: the Windows agent does NOT exhibit this behavior. I've been writing several appspecs for Windows services and I've never run into this. Only when we went to deploy to an Ubuntu server for the first time did we see this behavior.
+1
+1
+1
+1
+1
+1
So, I hope this helps other people find a solution. In my situation I work a lot with Magento so I deal with PHP, nginx. Composer is also tossed in the mix. I found a solution that works for me and for my case. If anyone has any comments or suggestions, please let me know.
I wanted to be have the production directory synced with the repository that I was using so any file modifications would be reflected. Not just updates to files, but delete the files that are no longer required. I think this is what everyone really wants. The solution I found was to deploy the code into a temporary directory (/var/www/release) and then rsync that directory over to the production directory (/var/www/magento).
appspec.yaml
version: 0.0
os: linux
files:
- source: ./
destination: /var/www/release/
permissions:
- object: /var/www/magento/
owner: www-data
group: www-data
hooks:
BeforeInstall:
- location: scripts/beforeInstall.bash
AfterInstall:
- location: scripts/afterInstall.bashscripts/beforeInstall.bash
#!/usr/bin/env bash
# I want to make sure that the directory is clean and has nothing left over from
# previous deployments. The servers auto scale so the directory may or may not
# exist.
if [ -d /var/www/release ]; then
rm -rf /var/www/release
fi
mkdir -vp /var/www/releasescripts/afterInstall.bash
#!/usr/bin/env bash
# I have left a few things in here and will explain this further (see below)
rsync --delete-before --verbose --archive --exclude "htdocs/media/" --exclude ".*" --exclude "htdocs/var/" --exclude "htdocs/app/etc/local.xml" /var/www/release/ /var/www/magento/ > /var/log/deploy.logThe rsync command is what allows you to overwrite all the files you need and it also deletes files if they are deleted from the repository. In my case I am deploying a git repo and there are times when extensions/files are removed and those need to be removed on the server.
There are also directories that are excluded from the rsync command. You do not need to sync cache directories and in my case the htdocs/media directory is a mounted S3 bucket and there are lots of files we do not need to delete.
All of the output is put into a deployment log (/var/log/deploy.log) file that I can use in case something goes wrong and can see the exact files deleted and synced over. When I was first experimenting with this, I ran rsync in dry-run mode and checked the log file to make sure files were synced correctly. I HIGHLY recommend you do the same.
Some things I want to try when I make the time, is to rsync files directly from the code-deploy directory. For each deployment, AWS CodeDeploy puts the code into a specific directory which you can reach by some of the environment variables it sets. This should speed up the deploy a little since it isn't putting the files in two places.
There are some things that have been left out of the scripts. Certain directories are chowned with www-data user/group since the files are served used nginx. You may need something similar. The code that places Magento into maintenance mode and removes it from maintenance mode has also been removed since this needs to be generic. You may need similar logic.
You can read more about the various hooks and when they occur in the docs at http://docs.aws.amazon.com/codedeploy/latest/userguide/app-spec-ref.html and also see the environment variables that can be used within the scripts.
+1
+1
+1
@JoshuaEstes You beautiful SOB, this is exactly what I needed.
+1
Thanks, glad I could help!
+1
+1
+1
+1
This is a good workaround. The issue I have is with Amazon and the fact that it's been over a year and yet this feature is still not implemented.
+1. It's a must.
+100
+1
any update on this feature?
+1
Any update on this?
Hello, any update on this feature?
+1
+1
+1
+1 for overwrites of files section
+1 to also add to permissions section as well, stuck for 10 hours with this back and forth to get symfony ./app/console and all related shell scripts executable, and it keeps whining about "The deployment failed because the permissions setting for is specified more than once in the application... blah blah". Now it's extremely backwards and counter-intuitive. (Sorry if sounds negative)
Looks like this has been solved: https://aws.amazon.com/about-aws/whats-new/2017/05/aws-codedeploy-adds-file-handling-support
From http://docs.aws.amazon.com/codedeploy/latest/userguide/deployments-create-console.html
In Content options, you can specify how AWS CodeDeploy handles files in a deployment target location that weren't part of the previous successful deployment.
Choose from the following:
Fail the deployment — An error is reported and the deployment status is changed to Failed.
Overwrite the content — If a file of the same name exists in the target location, the version from the application revision replaces it.
Retain the content — If a file of the same name exists in the target location, it is kept and the version in the application revision is not copied to the instance.
Edit: I'm using 'solved' in a loose way
Thanks for letting us know @iskandar
I saw that option a few weeks ago but never really took the time to read the docs about it. I'll be giving it a test in a couple of weeks.
still an issue if you are scaling up an autoscaling group.
+1
Thanks @JoshuaEstes for the work around.
+1
+1
This option is not working in the AWS CLI create-application --file-exists-behavior subcommand. Do I need to setup anything special to make this work?
Usage: aws codedeploy --application-name <name> --description <disc> --file-exists-behavior OVERWRITE --s3-bucket "<info>"
Error message: "Unknown options: --file-exists-behavior, OVERWRITE"
@jeremyroelfs It works with "aws deploy create-deployment" not with "create-application", even URL you mentioned as AWS CLI create application leads to "Create-deployment" documentation.
Typed it in fast and left out create-deployment and it should be create-deployment instead of create-application... aws deploy create-deployment --application-name <appname> --deployment-config-name CodeDeployDefault.OneAtATime --deployment-group-name <groupname> --description "Deploying App" --file-exists-behavior OVERWRITE --s3-location "bucket=<bucketname,bundleType=zip,key=<key>.zip"
And no... it is not working for me.
Here's a snippet from my deployment command (deploying from github):
aws deploy create-deployment --application-name my_app
--deployment-group-name my_app_deployment_group
--github-location commitId=020398409283049820938a0e234223409280398402989, repository=My_github_acc/my_repo
--deployment-config-name CodeDeployDefault.OneAtATime
--description "Test Description"
--file-exists-behavior OVERWRITE
Worked fine.
Crazy. Yeah, I can't get it to work at all. I've tried --file-exists-behavior OVERWRITE infront of/behind every sub-command. I am always getting this error:
21-Sep-2017 17:58:06 | usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
-- | --
21-Sep-2017 17:58:06 | To see help text, you can run:
21-Sep-2017 17:58:06 |
21-Sep-2017 17:58:06 | aws help
21-Sep-2017 17:58:06 | aws <command> help
21-Sep-2017 17:58:06 | aws <command> <subcommand> help
21-Sep-2017 17:58:06 |
21-Sep-2017 17:58:06 | Unknown options: --file-exists-behavior, OVERWRITE
I am running the command in an inline script on a Bamboo server... the --file-exists-behavior doesn't require specific permissions does it? I don't see a reference for any permissions in the docs or any other reading I've come across.
EDIT: My role has AWSCodeDeployRole, AmazonEC2RoleforAWSCodeDeploy, AWSCodeDeployFullAccess, all the S3 permissions and AdministratorAccess.
@jeremyroelfs you just forgot to update your AWS CLI, obviously old version doesn't know about this key :)
Is there anyway I can use this --file-exists-behavior option with codepipeline?
@hanzhimeng CodePipeline / CodeDeploy via the web interface does not support it. Short of modifying the source code of the agent (which would just be overwritten when the agent auto updates), there is no way to change the default file_exists_behavior. Would be awesome if the codedeploy agent had a conf file that would provide the same type of options that users of the AWS CLI receive. I wonder if they would accept a PR for this?
Hi all.
I could create a new deploy with fileExistsBehavior RETAIN (usind cmd line: aws deploy create-deployment), but in the case the deploy is started by autoscalling, like increase a desired capacity the deploy fails. I think that the default create-deployment used by autoscaling is assuming the default fileExistsBehavior confog that is DISALLOW.
How can I change this default behaviour in new instances in autoscalling that is part of a codedeploy?
Thanks.
+1
+1
@comicat-hu @ALL
It's been implemented already...
Via the appspec.yml file? Could u place a link where we can find how to use that (because it is not on the docs...)
@renatosc
not in appspec.yml, you path this option when creating deployment.
http://docs.aws.amazon.com/codedeploy/latest/userguide/deployments-create-console.html
example (deploying from GitHub):
aws deploy create-deployment --application-name client_applications
--deployment-group-name client_applications_deployment_group
--github-location commitId=09b0180cf1aa0c952ef00f407b5,repository=GIthub_User/repo_name
--deployment-config-name CodeDeployDefault.OneAtATime
--description "Test Description"
--file-exists-behavior OVERWRITE
@AntonChernysh, isn't this feature request/Issue to add support to the appsec.yml?
@renatosc what's the point? it's implemented in a different way that does the what was requested... And issues is closed already
Adding support to AWS CLI is not the same as adding via appspec.yml since it does not make the feature available to those who use the automatic deploy from Github repo.
I don't know who closed the issue but as you just said, the fix is not for what the issue was opened for.
@renatosc It's not on AWS CLI, neither appspec.ym nor AWS console level. It's now supported on AWS API for codedeploy. The way you call it is up to you.
See this for automatic deployment from github. on "Create New Deployment" step you have an option to specify file exist behavior.
https://aws.amazon.com/blogs/devops/automatically-deploy-from-github-using-aws-codedeploy/
Yes, it is supported by AWS CLI (the example is pasted above) and via AWS Portal (the link you provided). But it is not supported via appspec.yml (i.e, there is no keyword that I can use inside the file to make the deployment overwrite any existing file), which makes the Automatic Github deployment unable to use it.
The whole purpose of having automatic deployment is to not have to manually create deployments via CLI or portal.
@renatosc just read documentation (link above) till th end, there is info how to set webhooks and make sure deployment happens automatically.
I will test if it works Tomorrow and reply here either with an apology or confirmation.
Yes, I did that and was when I noticed this problem and then I googled it and I found this Github issue about it.
When you setup the Automatic Deployment (using the AWS CodeDeploy + GitHub Auto-Deployment services), every time you update your Github repo (via a push/commit), the AWS Code Deploy creates a new deployment for you automatically and it knows what to do by reading the appspec.yml. If any of the files that you try to deploy is already on the destination, the deployment will fail.
When creating the deployment via CLI or portal, you have a way to indicate to overwrite any existing file, but when you have the Automatic Deployment, you don't. I think the expected place to indicate that would be inside the appspec.yml. Of course, AWS could add that option on another place (maybe inside the AWS Code Deploy GitHub service configuration or even directly on the CodeDeploy application settings in the portal) and I would be fine with that, but right now there is no option.
The only "hack" is to add a script (to delete all existing files) to be run before the deployment That works, but it is not efficient specially when you just change one file..
And, of course, if you find out a way to have the Automatic Deployment overwrite existing files, please let us know.
@renatosc no apologies. If you follow all steps from a link I sent earlier it work fine (autodeployment with overwrite).
I spent a hour on this Today, started new EC2 instance, installed CD agent, configured all roles and profiles, created CD application, CD deploymentGroup and first deployment, configured auto-deployment as per instructions in my post above and it just works fine.
I've added you as a collaborator to my github repo.
You can try to update index.html and watch your changes are applied here:
http://34.235.119.225/
Looking forward to update from you here.
@AntonChernysh. Did u have put an index.html file in your server manually before trying to trigger the GitHub (with a commit) to make the deploychange it contents, for example?
I am asking that because it the Deployment will work and overwrite any file that already belonged to the commit history. But if it sees a existing file during it 1st deployment (for example), it will fail.
A simple way to test now is the following:
- Create a text.html ( "Hello 1") file directly inside your server
- Add a text.html ("Hello from deploy") inside your GitHub repo and see if it will update your existing text.html.
DIdnt you say this setup is not working at all?
- manually created index file with dummy content
- First deploy is always semi-manual, where you creating codedeploy deployment and setting flag “overwrite”. Later on this configuration is used for all subsequent auto deployments.
- Configured auto deployment from GitHub as per instructions.
- Commited changes to index.html
As I wrote above, yes initial index.html was created manually on the server, later overwritten with codedeploy deployment (first deployment).
Feel free to contact me in skype for consultations: anton.chernysh just so we don’t leave trashy comments on this issue.
That is the thing: When you create your first manually and flag it to overwrite the files, it will work.
But again, the whole purpose is to not have any manual deploy.
I am pretty sure that if I add a 2nd file manually to your server and later add it you your repo, the deployment will fail because it will say that the destination file already exists.
I see. Yes first deployment has to be manual-ish, then all following ones will use those settings.
If we are talking about fully automated dynamic systems, where you spin up new instances every few minutes, each with loads of pre-cocked files, and adding it to deployment group then I agree, this is quite possible scenario.
few other options:
1.
- have a separate folder ("CD_TARGET" for example) for codedeploy deployment (initially empty).
- BeforeInstall.sh Script: Stop web server (might not need this, just an example)
- Install: Deploy to "CD_TARGET" .
- AfterInstall.sh Script: Copy files from "CD_TARGET" to required location with overwrite.
There's another "dirty-recursive" fix that came to mind. You can initialize git repo in a target folder for deployment (or clone initially)
BeforeInstall.sh Script: Stop web server, "git fetch" changes from the same github repo where you have appspec.yml.
I don't think last one is good enough, just as a possible option...
... or they could just add support to overwrite files inside the appspec.yml as this issue is asking for and everything would work fine :)
I don't think it is quite as you say as " all following ones will use those settings.". They will not use whatever settings you used in the first deployment. The automatic deployment is set to (by default?) to not overwrite any existing file that it was not there already.
I tried with overwrite:yes, but didn't work. Any solution for this?
@thatianand - You will have to add overwrite flag while using aws createDeployment api.
""overwrite:yes"" will not work.
Pasting this example from above -
aws deploy create-deployment --application-name client_applications
--deployment-group-name client_applications_deployment_group
--github-location commitId=09b0180cf1aa0c952ef00f407b5,repository=GIthub_User/repo_name
--deployment-config-name CodeDeployDefault.OneAtATime
--description "Test Description"
--file-exists-behavior OVERWRITE
I wish this option can be configured from appspec.yml cuz sometimes CI/CD does not allow customize deployment configuration, so appspec.yml is the only option to change things.
is there a way to pass overwrite using appspec.yml file ? Everytime we have to create deployment manually to overwrite the files and then only codepipeline works
Yeah... this should really just be an easy to pass option rather than removing files in BeforeInstall:
+1
Can this be implemented, please? I am trying to integrate an existing instance into a CodeDeploy workflow and it fails because - since it's a pre-existing server - the files are already there; They should just be overwritten. I really don't like that I have to create a extra script just for this one instance. Especially for "in palce" deployments it would make sense to have an overwrite option, either in the appspec.yml or in the AWS Console.
What you want to do is delete existing files and do a first deployment with codedeploy including those files, once done Al subsequent deployments with codedeploy will overwrite those files with no issues.
@AntonChernysh yes, I could do that. But why can't I chose the overwrite option and have CodeDeploy overwrite the files if they exist? This project is not a simple "clear all" project; There are some folders on this instance which need to stay in-place and which are not in the appspec.yml, so I now have to manually delete them one by one.
When I use the AWS Console and create a new Deployment from "CodeBuild -> Deployments -> Create Deployment" I have these options for overwriting:
-
Fail the deployment
An error is reported and the deployment status is changed to Failed. -
Overwrite the content
The file in the application revision is copied to the target location on the instance, replacing the previous file. -
Retain the content
The file in the application revision is not copied to the instance. The existing file is kept at the target location and treated as part of the new deployment.
Which means CodeBuild is already capable of overwriting files - it just doesn't have an option via appspec.yml or in the Deployment Group configuration.
@KevinGimbel Obviously they don't want to implement this since this issue is open for a long time now.
What I normally do is select "Overwrite the content", delete all files that are part of deployment, and let CodeDeploy take care of updating content on a target env.
You do it only once...
@AntonChernysh that's what I am doing now, but this is only possible when manually creating a deployment via CLI or AWS Console - and that's the problem.
IMO it should just be an option in the AppSpec or Deployment Group config.
Anyway, I think you're right regarding this not being implemented given that the issue has been open for 3 years.
+1
We will look into a way to allow this setting when the deploy command is not run directly by the user (ex: autoscaling scale up, codepipeline)
Clearing all files will slow down the launch process if we have asset files which doesn't change frequently. We will be unnecessarily building assets every time which slows down the launch process.
Why is this not an option yet? Without appspec.yml option to specify overwrite behaviour, my auto-scaling groups cannot scale up because when Scale-Up event happens, the new instances do not CodeDeploy with overwrite behaviour!! This causes those launched instances to fail deployment ...
If the option was right there in the appspec.yml file - you know, since that file is supposed to dictate installation behaviour - we would be able to fully integrate code deploy with auto scaling but right now it fails ...
Is this package still maintained? Countless open issues like this with presupplied patches are sitting on deaf ears. Would be nice if someone on the AWS side would at least acknowledge the status.
Hi,
This is still a work in progress.
Thank you
Thanks, appreciate it @annamataws!
4 years and still no conclusion?
+1
This is a baffling omission. Please add support for this.
👍
4 years for such simple request. Shame. +1.
I faced this issue and I have to update script to clean directory to avoid this error
Well... even though this is 4 years in the making, at least I know other people are experience the same heartache that I am.
+1
From last four years no update for this small request. we need to some file configuration options other than using scripts in hooks. Can someone please prioritize this.?
AWS 4 years, this isn't really rocket science?
We're also using another framework they are building (amplify-js), update activity is too slow, not sure whether AWS responsible to maintain their open source frameworks
+1
still waitin'
CodeDeploy actually does support this feature, but it's not in the appspec.yml file. It's in the "CreateDeployment" API (unfortunately not exposed in CloudFormation).
Waiting for it...
waiting...
This would be helpful for our use case as well.