aws/aws-codedeploy-agent

Add Support for "Overwrite" instruction in appspec.yml "Files" section

Closed this issue · 176 comments

colinbjohnson commented

Feature Request:

Would like the ability to have an "overwrite" instruction in the appspec.yml "Files" section.

Use:

In my initial testing, I wanted to clobber all files in /var/www/html with new files. This resulted in a failure because the files already existed. Specifically, the following failure: 2015-06-13 20:58:40 ERROR [codedeploy-agent(7589)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: RuntimeError - File already exists at location /var/www/html/index.html - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:113:ingenerate_normal_copy'`

Desired Behavior:

I'd like to be able to create an appspec.yml file with the following section:
files:
- source: web
destination: /var/www/html/
overwrite: true
Notice I've added the "overwrite" instruction, which is currently not offered.

Current Workaround:

I've been working around by utilizing the hook section to do file copies. The hooks/shell script workaround is sufficient, rendering this a lower priority feature request.

RoKondo commented

+1

rana-amritanshu commented

+1 please add this feature

  • 😄3
  • 👀2
ApsRajput commented

+1

  • 👀3
ashwanijha04 commented

+1
This would help people prevent writing custom clean-up scripts and get to work quickly and swiftly. :)
Please add this feature.

The below manual cleanup is the current workaround:

Add it in your appspec. Something like:

hooks:
    BeforeInstall:
        - location: scripts/cleanup.sh

scripts/cleanup.sh

#!/usr/bin/env bash

# I want to make sure that the directory is clean and has nothing left over from
# previous deployments. The servers auto scale so the directory may or may not
# exist.
if [ -d /var/www/release ]; then
    rm -rf /var/www/release
fi
mkdir -vp /var/www/release
  • 👍7
  • 👀1
trungegd commented

Hi every body. Any update on it?

  • 👍3
  • 😄3
  • 🚀3
  • 👀3
Deleted user commented

+1

  • 👍4
Sneakr commented

5 years for an overwrite facepalm

  • 😄4
sshymko commented

Somebody should just implement this and submit a PR

  • 😄3
BartusZak commented

image

  • 😄20
  • 👀1
daaru00 commented

+1 really need this feature

  • 👍2
DanielViglione commented

+1 you shouldn't have to resort to shell scripts for such a fundamental task.

  • 👍4
edt-max commented

+1

  • 👍2
j0ton commented

+1

AntonChernysh commented

Time to celebrate first anniversary soon, almost 5 years 🎉

  • 😄4
  • 🎉9
yubangxi commented

It seems weird to update this request right now, but CodeDeploy already supports custom file override behavior back to 2017 - https://aws.amazon.com/about-aws/whats-new/2017/05/aws-codedeploy-adds-file-handling-support/

I'm closing out this issue now.

  • 👎21
  • 😕7
  • 👀3
cdshepherd commented

@yubangxi This does not solve the issue, as there's still no way to automate the behaviour and specify it in appspec.yml.

+1 would find this feature useful

  • 👍11
0cool-f commented

It seems weird to update this request right now, but CodeDeploy already supports custom file override behavior back to 2017 - https://aws.amazon.com/about-aws/whats-new/2017/05/aws-codedeploy-adds-file-handling-support/

I'm closing out this issue now.

By our side, this result in a not working API. You in AWS are developing a lot of good new stuff those times, I can't believe it's hard for you to fix this existing feature 😕 I really hope You decide to make happy all those people (customers) waiting for years for a fix.

  • 👍15
  • ❤️4
rubashanti commented

+1

  • 👍1
AstroTom commented

+1

wes-novack commented

I can't believe this was closed without actually addressing the issue. We know the feature exists from the AWS Web Console via a manual clicky process, but it does not exist in the appspec.yml file or the CodeDeploy API!

We need this feature in the API so that we can utilize Terraform, etc. Please re-open.

  • 👍4
Helen1987 commented

#145 should be adressed as well

  • 👍2
bkotch commented

It seems weird to update this request right now, but CodeDeploy already supports custom file override behavior back to 2017 - https://aws.amazon.com/about-aws/whats-new/2017/05/aws-codedeploy-adds-file-handling-support/

I'm closing out this issue now.

The documentation you linked to from 2017 is no longer valid in CodeDeploy for 2020. Or am I missing something

ashwanijha04 commented

o from 2017 is no longer valid in CodeDeploy for 2020

This support is from the CodeDeploy console while making a deployment. There is not an option to overwrite the files from the AppSpec file. Correct me if I am wrong.

ecommerce-technician commented

I need to overwrite files during pipeline builds, is there a way to do this in appspec.yml?

GCloud offers this and I'm pointing customers to GCloud instead of AWS because of how difficult writing my own cleanup scripts has become

  • 👍3
Justin-DynamicD commented

2020 and still not possible. This seems a little silly at this point. Guess scripts ftw?

  • 👍5
philstrong commented

Q: Is this deployment level overwrite or folder/target level?

cesc1989 commented

Q: Is this deployment level overwrite or folder/target level?

@philstrong folder level overwrite. In the deployment, if it tries to update an existing file, it would break. Leaving us to do cleanup before placing any file in destination folders.

reubensml commented

Holy Sh** . 5 years and yet no solution for such a simple requirement :O

waldoj commented

I regret that I didn't bake a cake for this issue's fifth birthday, back in June. In 2025, we're all gonna have a blow-out tenth birthday, though.

  • 👍4
  • 😄3
  • 🎉6
  • ❤️1
  • 🚀1
waldoj commented

The first frustrated comment didn't appear until Dec 20, 2018, or two and a half years after the issue was opened. There's a point at which it's reasonable for paying customers to be frustrated that a feature doesn't work correctly, and I think two and a half years is a reasonable point to switch to a new tactic, after "patiently waiting" failed to pan out.

  • 👍6
Zerquix18 commented

Just leaving a +1 but not really hopeful

  • 🚀2
andrzpg12 commented

I am aware that this feature can be made in the Console of CodeDeploy Service when "Create Deployment" of the Deployment Group is called, but, if we can force an overwrite by appspec.yaml would be much better.
Cause, i create a Codepipeline with the stage Deploy as CodeDeploy as provider but, when the pipeline is triggered by webhooks in Github and start to run, the default deployment is not enable the "overwrite content" so fails if you want to copy all the proyect in the folder in the EC2 autoscaling instance.

  • 👍6
markdelf commented

+1

0cool-f commented

The point is that this is more like a bug than a feature request, in fact, I consider this detail an obvious thing for everyone.

jjmontgo commented

Still hoping this will be added in 2021?

  • 👍2
decemvre commented
zwezheng commented

This issue was opened in 2015 and it still remains open 6 years after. Is the aws team maintaining this service still exists?

  • 😄2
philstrong commented

I'm the new dev manager on CodeDeploy (5 months). You can imagine that after so many years there have been a lot of changes to engineers and managers. I wonder if after 6 years the former team is sending you a message that this isn't making the cut ;).

Is this the number one feature request? Most comments doesn't always signify this.

  • 👍2
  • 👎2
cesc1989 commented

I'm the new dev manager on CodeDeploy (5 months). You can imagine that after so many years there have been a lot of changes to engineers and managers. I wonder if after 6 years the former team is sending you a message that this isn't making the cut ;).

Is this the number one feature request? Most comments doesn't always signify this.

@philstrong then shouldn't this be closed and locked?

There's already a decent workaround but of course an official solution would be better.

  • 👎1
0cool-f commented

I'm the new dev manager on CodeDeploy (5 months). You can imagine that after so many years there have been a lot of changes to engineers and managers. I wonder if after 6 years the former team is sending you a message that this isn't making the cut ;).

Is this the number one feature request? Most comments doesn't always signify this.

You are so wrong, this never was a feature request, this is a BUG of CodeDeploy, that you should have fixed long time ago, since your competitors have this feature working as it should 🤷‍♂️ are You able to fix this "simple" thing? All we see is procrastination and a will of searching for an excuse to explain why You don't fix this bug, in the meantime AWS build new things even into CodePipeline area (w/o fixing the current 🤔 isn't that a bad practice?), that's confusing.

Please do not try to justify this grotesque situation, we don't need a cloud provider that change one manager a day like pants, we need that AWS give attention to paying customers and fix issues, if they care about customers, otherwise it means they don't, that's it, clear and simple.

AWS, everyday, spam how much it's marvelous, we don't care about words, we are technician, we look facts, we look how AWS demonstrate it.

  • 👍5
0cool-f commented

I'm the new dev manager on CodeDeploy (5 months). You can imagine that after so many years there have been a lot of changes to engineers and managers. I wonder if after 6 years the former team is sending you a message that this isn't making the cut ;).
Is this the number one feature request? Most comments doesn't always signify this.

@philstrong then shouldn't this be closed and locked?

There's already a decent workaround but of course an official solution would be better.

this shouldn't be closed, this should be kept open until it's fixed, as the lifecycle of bugs, also afaik there's no workaround to replace the fully functional "fileExistsBehavior" operation (with rollback, etc.)

  • 👍2
cesc1989 commented

I'm the new dev manager on CodeDeploy (5 months). You can imagine that after so many years there have been a lot of changes to engineers and managers. I wonder if after 6 years the former team is sending you a message that this isn't making the cut ;).
Is this the number one feature request? Most comments doesn't always signify this.

@philstrong then shouldn't this be closed and locked?
There's already a decent workaround but of course an official solution would be better.

this shouldn't be closed, this should be kept open until it's fixed, as the lifecycle of bugs, also afaik there's no workaround to replace the fully functional "fileExistsBehavior" operation (with rollback, etc.)

@0cool-f (It's an old comment)this is the workaround I mentioned and the one I based my solution in the end back in the days.

Of course it isn't ideal and will be times better an official solution but it seems there's no interest in doing so.

0cool-f commented

I'm the new dev manager on CodeDeploy (5 months). You can imagine that after so many years there have been a lot of changes to engineers and managers. I wonder if after 6 years the former team is sending you a message that this isn't making the cut ;).
Is this the number one feature request? Most comments doesn't always signify this.

@philstrong then shouldn't this be closed and locked?
There's already a decent workaround but of course an official solution would be better.

this shouldn't be closed, this should be kept open until it's fixed, as the lifecycle of bugs, also afaik there's no workaround to replace the fully functional "fileExistsBehavior" operation (with rollback, etc.)

@0cool-f (It's an old comment)this is the workaround I mentioned and the one I based my solution in the end back in the days.

Of course it isn't ideal and will be times better an official solution but it seems there's no interest in doing so.

Yeah, I think all of us who haven't changed cloud provider did something similar with rsync (so did I), but they already have all we need, in facts we aren't asking for something new to deploy: https://docs.aws.amazon.com/codedeploy/latest/APIReference/API_CreateDeployment.html all we need is just the "fileExistsBehavior" as a parameter in appspec.yml

I could be wrong, but I think it should be pretty simple for them to allow this.

  • 👍5
philstrong commented

I thought I might learn something useful and I did. fileExistsBehavior was something I wasn't aware of. Let me dig on that and see where it goes.

I'll mark for triage and we'll look into this during grooming and next sprint.

  • 👍7
  • 🎉9
waldoj commented

Thank you, @philstrong!

  • 👍4
engineerakki commented

@philstrong
Was this picked up in your sprint ?
Is there any official timeline for this feature ?

quentinhayot commented

Is this really happening ?!!!
2021 WILL be a good year after all ! :')

dreaddesign commented

if this is fixed, i would be very happy. :) +1

philstrong commented
  • 👍2
  • 🎉4
philstrong commented

This will be coming soon in the next release of the agent. @chrisdibble did all of the work so he can explain the feature/fix a little better.

  • 👍4
  • 😄1
  • 🎉6
  • ❤️1
  • 🚀2
waldoj commented

Great news! Thanks so much for your attention to this.

  • 👍1
chrisdibble commented

Not a huge change - we're just allowing the file_exists_behavior to be set as part of the appspec file, which will take precedence over the corresponding command line option for file_exists_behavior for the upcoming release. We're not allowing fine grained control of file_exists_behavior per file/directory listed in the appspec files section, though. We'll consider that if demand from customers still exists following the upcoming release.

  • 🎉3
0cool-f commented

Not a huge change - we're just allowing the file_exists_behavior to be set as part of the appspec file, which will take precedence over the corresponding command line option for file_exists_behavior for the upcoming release. We're not allowing fine grained control of file_exists_behavior per file/directory listed in the appspec files section, though. We'll consider that if demand from customers still exists following the upcoming release.

Thanks a lot for fixing it. You may not have noticed that, but this is exactly THE huge fix thousands of people have been waiting for.

  • 👍3
alinoudev commented

On what date its planned to release?

quentinhayot commented

Not a huge change - we're just allowing the file_exists_behavior to be set as part of the appspec file, which will take precedence over the corresponding command line option for file_exists_behavior for the upcoming release. We're not allowing fine grained control of file_exists_behavior per file/directory listed in the appspec files section, though. We'll consider that if demand from customers still exists following the upcoming release.

Guys, this is it !

On what date its planned to release?

Hopefully soon ! I'll just put the file_exists_behavior in the appspec already...

Deleted user commented

Looking forward to this!

s-larbi commented

Any updates regarding the overwrite feature? It takes time to delete everything and setup apps from scratch. The feature would be a great time saver.

philstrong commented
  • 🎉3
  • 🚀5
  • 👀2
abhijith7025 commented

I think there is already a feature

files:

  • source:
    destination: c:\temp
    file_exists_behavior: OVERWRITE

read this out https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-files.html

  • 👎4
  • 😕1
s-larbi commented

I have tried this before and it didn't work: "The deployment failed because a specified file already exists at this location: /var/www/html/website-folder/sub-folder/file-name.php"...

0cool-f commented

I think there is already a feature

files:

* source: 
  destination: c:\temp
  file_exists_behavior: OVERWRITE

read this out https://docs.aws.amazon.com/codedeploy/latest/userguide/reference-appspec-file-structure-files.html

mate you should always read before write something: this bug exists because this feature, which it seems you haven't tried, does not work, in fact it's a bug...do you really think that an issue could remain open and debated for years, waiting for a doc page that we all haven't read? 😶

  • 👍3
  • 😄1
s-larbi commented

I hope this bug is fixed ASAP. It would be a great time saver.

0cool-f commented

I hope this bug is fixed ASAP. It would be a great time saver.

As Phil said, after the release cycle of 1.3.2 it will work.

Phil's team was fast, but "ASAP" is a relative concept, have you read the date of issue?

philstrong commented

1.3.2 is out in some regions e.g. us-east-1 but rolling out to the rest. We will update GitHub once all commercial regions are released which will be Thurs or Friday (5/13 or 5/14).
sa-east-1, ap-northeast-1, us-west-1, eu-west-1, ap-southeast-1 are also released

  • 👍4
  • 🎉4
quentinhayot commented

Jeff, Give this guy a cigar !

  • 😄3
philstrong commented

Issue is fixed in 1.3.2. Any and all feedback is welcome.

  • 🎉7
  • 🚀3
waldoj commented

Thank you, @philstrong, for getting this issue resolved. This will be a great little quality-of-life improvement for a lot of people. I'm grateful. 🙏🏻

andreprawira commented

hi @philstrong is file_exists_behavior: OVERWRITE also applicable to hooks section? because im also getting file already exist in codedeploy temp folder when i changed a folder name in the script. For example,

if my previous appspec.yml was

hooks:
  AfterInstall:
    - location: scripts\install_dependencies
      timeout: 300

and then i changed it to

hooks:
  AfterInstall:
    - location: scripts-linux\install_dependencies
      timeout: 300

i'd get that file already exist error, so if i add

hooks:
  AfterInstall:
    - location: scripts-linux\install_dependencies
      timeout: 300
      file_exists_behavior: OVERWRITE

will that solve the problem?

andreprawira commented

@philstrong sorry for the multiple comments but im also facing this The deployment failed because a specified file already exists at this location: C:\inetpub\www\ec2django/asgi.py error, this is on a windows appspec.yml, here is the script

`version: 0.0
os: windows
files:

  • source: \ec2django
    destination: C:\inetpub\www\ec2django
    file_exists_behavior: OVERWRITE`

any idea where i got it wrong? that should have overwritten the previously existing files right?

mwjones-aws commented

@andreprawira to address your first comment, no. file_exists_behavior is not supported in the hooks section.

With respect to your second comment, the service team will try to reproduce the problem. Can we have a little more information?

  1. Please post the full appspec.yml
  2. What file-exists-behavior are you passing to CreateDeployment?