beyond-sw-camp/beyond-sw-camp-be01_last-dance

AWS Github Action Deploy 2 EC2 - 2

Closed this issue · 13 comments

ec2 code deploy agent 오류 대응 방안

# AWS 자격증명 파일 삭제
$ sudo rm -rf /root/.aws/credentials

# codedeploy-agent 재시작
$ sudo systemctl restart codedeploy-agent

위의 명령어를 aws에서 입력 한 후, 다시 배포 시도하였는데, 실패하였습니다.

codedeploy-agent.log 내용은 아래 와 같습니다.

2024-03-19T05:18:59 ERROR [codedeploy-agent(2096)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: InstanceAgent::Plugins::CodeDeployPlugin::ScriptError - Script at specified location: start.sh run as user ubuntu failed with exit code 1 - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:202:in `execute_script'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:142:in `block (2 levels) in execute'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:130:in `each'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:130:in `block in execute'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:227:in `create_script_log_file_if_needed'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/hook_executor.rb:128:in `execute'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:195:in `block (3 levels) in map'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:193:in `each'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:193:in `block (2 levels) in map'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:115:in `execute_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:137:in `process_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:119:in `acknowledge_and_process_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:99:in `block in perform'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:363:in `run_task'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `block (3 levels) in create_worker'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:335:in `loop'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:335:in `block (2 levels) in create_worker'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `catch'
/opt/codedeploy-agent/vendor/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block in create_worker'
/opt/codedeploy-agent/vendor/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-03-19T05:18:59 WARN  [codedeploy-agent(2096)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Calling PutHostCommandComplete: "Code Error"
2024-03-19T05:18:59 INFO  [codedeploy-agent(2096)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.7.0-92_deb.
2024-03-19T05:18:59 INFO  [codedeploy-agent(2096)]: [Aws::CodeDeployCommand::Client 200 0.015427 0 retries] put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{\"error_code\":5,\"script_name\":\"\",\"message\":\"Script at specified location: start.sh run as user ubuntu failed with exit code 1\",\"log\":\"\"}"},host_command_identifier:"eyJiYXRjaElkIjoiYzdmNGM1NTQ5M2UxNDg5YThlZWJmN2M4MGM2OTI4MGQvcHVibGljMDA0IiwiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS9hcC1ub3J0aGVhc3QtMy9wcm9kL29ycGhldXM6cHVibGljMDA1Lzc2NzM5ODAxMTA0NDpkLU02VTVKWlE3RyIsImhvc3RJZCI6ImFybjphd3M6ZWMyOmFwLW5vcnRoZWFzdC0zOjc2NzM5ODAxMTA0NDppbnN0YW5jZS9pLTBkOTNjNDBiYTcwODg4NDQwIiwiY29tbWFuZElkIjoiQXBvbGxvRGVwbG95Q29udHJvbFNlcnZpY2V8YXJuOmF3czplYzI6YXAtbm9ydGhlYXN0LTM6NzY3Mzk4MDExMDQ0Omluc3RhbmNlL2ktMGQ5M2M0MGJhNzA4ODg0NDB8NnwwIiwiY29tbWFuZE5hbWUiOiJBcHBsaWNhdGlvblN0YXJ0IiwiY29tbWFuZEluZGV4Ijo2LCJhdHRlbXB0SW5kZXgiOjF9")

ERROR 메시지 해석 : "Script at specified location: start.sh run as user ubuntu failed with exit code 1" - 이는 start.sh 스크립트가 ubuntu 사용자로 실행되었을 때 오류가 발생했음을 나타냅니다. 이 오류로 인해 스크립트 실행이 실패했습니다.

위 에러 관련
AWS - 개발자 도구 - CodeDeploy - 배포 로그 를 보면 아래와 같은 오류
오류코드: ScriptFailed

LifecycleEvent - ApplicationStart
Script - start.sh
[stderr]
[stderr]ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
[stderr]
[stderr]Please set the JAVA_HOME variable in your environment to match the
[stderr]location of your Java installation.
[stderr]
[stderr]sudo: docker: command not found

오류 해결 진행상황....

EC2 인스턴스에 Java 설치

$ sudo apt install openjdk-17-jdk

JAVA 환경변수 설정 및 적용

$ sudo vi /etc/profile
# 이후 aws 재접속

$ echo $JAVA_HOME
/usr/lib/jvm/java-17-openjdk-amd64

다시 배포 시도하였는데, 실패하였습니다.

오류 코드: ScriptTimedOut

LifecycleEvent - ApplicationStart
Script - start.sh
[stdout]Downloading https://services.gradle.org/distributions/gradle-8.5-bin.zip
[stdout]............10%.............20%............30%.............40%.............50%............60%.............70%.............80%............90%.............100%
[stdout]
[stdout]Welcome to Gradle 8.5!
[stdout]
[stdout]Here are the highlights of this release:
[stdout] - Support for running on Java 21
[stdout] - Faster first use with Kotlin DSL
[stdout] - Improved error and warning messages
[stdout]
[stdout]For more details see https://docs.gradle.org/8.5/release-notes.html
[stdout]
[stdout]Starting a Gradle Daemon (subsequent builds will be faster)
[stderr]

CodeDeploy 에이전트가 start.sh 스크립트를 실행했지만, 이 스크립트가 60초 내에 완료되지 않아 에러가 발생했습니다.

오류 해결 진행상황...

$ vi appspec.yml
# timeout 시간 180초로 수정

다시 배포 시도하였는데, 실패하였습니다.
오류 코드: ScriptFailed: 스크립트 이름

[stderr]Please read the following process output to find out more:
[stderr]-----------------------
[stderr]
[stderr]
[stderr]* Try:
[stderr]> Run with --stacktrace option to get the stack trace.
[stderr]> Run with --info or --debug option to get more log output.
[stderr]> Run with --scan to get full insights.
[stderr]> Get more help at https://help.gradle.org.
[stderr]sudo: docker: command not found

docker가 설치되어 있지 않아서 발생한 오류 같습니다.

  • 저는 아래와 같은 오류가 발생하고 있습니다.
  • AWS Secret Access Key를 새로 생성했으나 여전히 같은 오류 발생.,

image

Error: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
오류 해석: 계산한 요청 서명이 제공한 서명과 일치하지 않습니다. AWS Secret Access Key 및 서명 방법을 확인하십시오. 자세한 내용은 서비스 문서를 참조하십시오

#108 (comment)
오류 해결 진행상황...

다시 배포 시도하였는데, 실패하였습니다.
오류 코드: ScriptTimedOut

LifecycleEvent - ApplicationStart
Script - start.sh
[stdout]Starting a Gradle Daemon, 1 busy Daemon could not be reused, use --status for details
[stderr]/opt/codedeploy-agent/deployment-root/015467d7-ee70-49a2-8c73-c1e6ad7a9920/d-OXJMQ10K3/deployment-archive/start.sh: line 5: 7136 Killed ./gradlew bootRun
[stderr]

로그를 자세히 보니 도커쪽 문제인 것 같아서
도커 파일 명령어를 하나씩 실행해 보니,
sudo docker compose -f docker-compose.yml up -d 명령어 실행시 아래와 같이 에러가 뜹니다.
=> ERROR [rest 2/2] COPY build/libs/*.jar app.jar

#108 (comment) 오류 해결 진행상황...

다시 배포 시도하였는데, 실패하였습니다. 오류 코드: ScriptTimedOut

LifecycleEvent - ApplicationStart
Script - start.sh
[stdout]Starting a Gradle Daemon, 1 busy Daemon could not be reused, use --status for details
[stderr]/opt/codedeploy-agent/deployment-root/015467d7-ee70-49a2-8c73-c1e6ad7a9920/d-OXJMQ10K3/deployment-archive/start.sh: line 5: 7136 Killed ./gradlew bootRun
[stderr]

로그를 자세히 보니 도커쪽 문제인 것 같아서 도커 파일 명령어를 하나씩 실행해 보니, sudo docker compose -f docker-compose.yml up -d 명령어 실행시 아래와 같이 에러가 뜹니다. => ERROR [rest 2/2] COPY build/libs/*.jar app.jar

폴더 권한이 루트로 되어 있어서 jar 파일이 생성되지 않는 문제입니다.

$ ls -l
total 64
-rw-r--r-- 1 root root  103 Mar 19 07:24 Dockerfile
-rw-r--r-- 1 root root 7007 Mar 19 07:24 README.md
-rw-r--r-- 1 root root  372 Mar 19 07:24 appspec.yml
-rw-r--r-- 1 root root  615 Mar 19 07:24 build.gradle
-rw-r--r-- 1 root root  687 Mar 19 07:24 docker-compose.yml
drwxr-xr-x 3 root root 4096 Mar 19 07:24 gradle
-rwxr-xr-x 1 root root 8692 Mar 19 07:24 gradlew
-rw-r--r-- 1 root root 2868 Mar 19 07:24 gradlew.bat
drwxr-xr-x 3 root root 4096 Mar 19 07:24 nginx
-rw-r--r-- 1 root root   28 Mar 19 07:24 settings.gradle
drwxr-xr-x 4 root root 4096 Mar 19 07:24 src
-rwxr-xr-x 1 root root  115 Mar 19 07:24 start.sh
-rwxr-xr-x 1 root root   36 Mar 19 07:24 stop.sh

$ pwd
/home/ubuntu/app/restdb

appspec.yml 의 내용을 아래와 같이 수정하면 됩니다.

permissions:
  - object: /
    pattern: "**"
    owner: ubuntu
    group: ubuntu

일단 저는 ./gradlew clean bootJar 명령을 실행앴는데 .... 메모리 문제도 있는 것 같습니다.
image

start.sh 파일에서 bootRun을 bootJar로 변경하고
인스턴스 유형을 t2.small로 올려서 문제 해결하였습니다

저도 어제 저녁에 지영님이 올려주신 이슈까지 진행하고 해결했습니다.

#108 (comment)

#108 (comment)

$ vi appspec.yml
# timeout 시간 180초로 수정

추가 문의 사항은 언제든 이슈 달아 주세요.