Code deploy agent runs the wrong files in case of multiple deployments having same file path and name

Question

Code deploy agent runs the wrong files in case of multiple deployments having same file path and name

Closed this issue 8 years ago · 9 comments

Hi
I've noticed this several times - consider a scenario where two deployments/revisions have a same file in the same path:

In the first revision, the the file is scripts/ApplicationStop/application-stop.sh
In the second revision (For a different application), if you have similar file in the same location (but different) content, code deploy will execute the previous file (file from the previous deployment).
You deploy another revision, and this time code deploy will pick up the previous file (from revision 2).

we have a standard script/directory structure that we follow across deployments and this really causes code deploy agent to either run the wrong scripts or mess up data in some cases.

Deleted user commented 8 years ago

Thanks

Answer 1 · 2016-09-22T18:57:14.000Z

Here's excerpts from my log files:

[2016-09-22 18:44:43.242] [d-IJBVSF6XH]LifecycleEvent - ApplicationStop
[2016-09-22 18:44:43.242] [d-IJBVSF6XH]Script - scripts/ApplicationStop/application-stop.sh
[2016-09-22 18:44:43.276] [d-IJBVSF6XH][stdout]xxxxxxxxxxxxxx stop/waiting
[2016-09-22 18:44:51.539] [d-IJBVSF6XH]LifecycleEvent - BeforeInstall
[2016-09-22 18:44:51.541] [d-IJBVSF6XH]Script - scripts/BeforeInstall/api-before-install.sh
[2016-09-22 18:44:52.933] [d-IJBVSF6XH][stdout]Try deleting system user and group [xxxxxxxx:xxxxxxxxxx]
[2016-09-22 18:44:52.935] [d-IJBVSF6XH][stdout]Deleting system user: xxxxxxxxxx
[2016-09-22 18:45:00.625] [d-IJBVSF6XH]LifecycleEvent - AfterInstall
[2016-09-22 18:45:00.625] [d-IJBVSF6XH]Script - scripts/AfterInstall/api-after-install.sh
[2016-09-22 18:45:00.691] [d-IJBVSF6XH][stdout]Preparing...                          ########################################
[2016-09-22 18:45:00.702] [d-IJBVSF6XH][stdout]Creating system group: xxxxxxxxxxx
[2016-09-22 18:45:00.728] [d-IJBVSF6XH][stdout]Creating system user: xxxxxxx in xxxxxxxx with xxxxxxxxxxxx user-daemon and shell /bin/false
[2016-09-22 18:45:00.750] [d-IJBVSF6XH][stdout]Updating / installing...
[2016-09-22 18:45:04.148] [d-IJBVSF6XH][stdout]xxxxxxxxxx-1-1               ########################################
[2016-09-22 18:45:05.572] [d-IJBVSF6XH]LifecycleEvent - ApplicationStart
[2016-09-22 18:45:05.572] [d-IJBVSF6XH]Script - scripts/ApplicationStart/api-application-start.sh
[2016-09-22 18:45:05.632] [d-IJBVSF6XH][stdout]Starting xxxxxxxxxxxxx: [  OK  ]
[2016-09-22 18:45:07.115] [d-IJBVSF6XH]LifecycleEvent - ValidateService
[2016-09-22 18:45:07.115] [d-IJBVSF6XH]Script - scripts/ValidateService/api-validate-service.sh
[2016-09-22 18:45:07.427] [d-IJBVSF6XH][stdout]xxxxxxxxxxxxx (pid  32452) is running...```

Notice the second line where it says scripts/ApplicationStop/application-stop.sh. The actual appspec taken from the same deployment (by ssh'ing into the instance):

version: 0.0
os: linux
hooks:
  AfterInstall:
    - location: scripts/AfterInstall/api-after-install.sh
      timeout: 300
      runas: root
  ApplicationStart:
    - location: scripts/ApplicationStart/api-application-start.sh
      timeout: 300
      runas: root
  ApplicationStop:
    - location: scripts/ApplicationStop/api-application-stop.sh
      timeout: 300
      runas: root
  BeforeInstall:
    - location: scripts/BeforeInstall/api-before-install.sh
      timeout: 300
      runas: root
  ValidateService:
    - location: scripts/ValidateService/api-validate-service.sh
      timeout: 300
      runas: root
files:
  - source: content/xxxxxxxx-1-1.noarch.rpm
    destination: /home/ec2-user
permissions:```

Notice the name of the application stop script.

Answer 2 · 2016-09-23T20:58:14.000Z

Hi,

This is actually expected behavior. ApplicationStop lifecycle event is supposed to run the script from last successful revision, cause the lifecycle event is trying to stop the application revision installed last time. All other lifecycle events run scripts from the current revision. A more detailed explanation is here: http://docs.aws.amazon.com/codedeploy/latest/userguide/app-spec-ref-hooks.html

This ApplicationStop lifecycle event occurs even before the application revision is downloaded. You can use this event if you want to gracefully stop the application or remove currently installed packages in preparation of a deployment. The AppSpec file and scripts used for this deployment lifecycle event are from the last successfully deployed application revision.

Thanks,
Binbin

Answer 3 · 2016-10-06T14:41:49.000Z

But how to deal with it? For example, I have a wrong script set in ApplicationStop section. Now, everytime I deploy (or rollback), it call the wrong script. Then, the new deployment is stopped ....

I have delete this wrong script from my EC2 instance. But nothing changes.

How can I really stop this wrong script from running?

Answer 4 · 2016-10-06T18:59:06.000Z

I found the answer:

Call the create-deployment command, set the --ignore-application-stop-failures option, and deploy the application revision again.

http://docs.aws.amazon.com/zh_cn/codedeploy/latest/userguide/troubleshooting-deployments.html#troubleshooting-deployments-applicationstop

Answer 5 · 2016-10-06T19:02:45.000Z

Alternatively, you can delete all the files/directories located at /opt/codedeploy-agent/deployment-root and restart the code deploy agent.

Answer 6 · 2016-10-06T19:05:30.000Z

Just for record, I resolved my issue by creating two different applications -one for backend, one for frontend. Both the applications get downloaded to different folders on each machine (different folders under /opt/codedeploy-agent/deployment-rootand now the scripts don't interfere.

Think I'm gonna close this issue.

Answer 7 · 2018-11-07T08:59:36.000Z

I'm quite baffled as to why this is expected behaviour... Let's say I have revisionA that uses Apache and MySQL so revisionA.ApplicationStop will stop these applications. Then you replace MySQL with Postgres for revisionB so when you want to install revisionB, you'll want to stop Postgres not MySQL (which you might have already uninstalled)... or, perhaps you'd want to automate this even more so you'd want to uninstall MySQL and install Postgres in revisionB.ApplicationStop.
It's interesting that others have not pointed this out...

Answer 8 · 2018-11-07T22:53:50.000Z

@cristina0botez the key element here is to take into consideration that CodeDeploy has been designed to automate your deployments end-to-end. To that point and using your example:

revisionA has been deployed and left MySQL/Apache running
revisionA.ApplicationStop has instructions on how to stop those 2 components

When revisionB deployment starts, in theory MySQL/Apache are still running, so those are the ones that need stopping. That's why revisionA.ApplicationStop will be run. In your first example, you have implicitly assumed that the activities to replace MySQL with Postgres have been manually done outside of CodeDeploy. A revisionB deployment could look like this:

revisionA.ApplicationStop stops MySQL/Apache
revisonB.BeforeInstall removes MySQL and install/configures Postgres
revisionB.ApplicationStart starts Postgres/Apache
revisionB.ApplicationStop has instructions on how to stop Postgres/Apache, but will only be used in the next deployment.

I agree this can be slightly confusing at first, but from a development perspective it's probably easier to have your revision package designed as a single self-contained unit (i.e. with instructions that comprise the full lifecycle of the application) rather than having to figure out later how to properly stop something that's been already running. I hope this makes sense!! :)