- AWS Console -> Kinesis -> Create Firehose Delivery Stream "StreamName" -> Direct PUT -> Next -> Choose Destination -> Create S3 Bucket “covid-vaccines-logs-diolive” -> Configure settings -> buffer size 5mb -> buffer interval 60s -> IAM Role -> create new role -> Review and create
- AWS Console -> EC2 -> Amazon Linux 2 AMI -> t2micro -> review and launch -> create new key pair -> download .pem file -> download putty -> puttygen -> load.pem file -> save .ppk file -> putty copy dns -> paste hostname -> SSH -> auth -> load ppk file -> login “ec2-user”
Commands:
-
sudo yum install -y aws-kinesis-agent
-
sudo yum install -y git
-
git clone https://github.com/cassianobrexbit/dio-live-aws-bigdata-2.git
-
unzip Dataset.zip
-
chmod a+x LogGenerator.py
-
nano LogGenerator.py
-
less country_vaccinations.csv
-
sudo mkdir /var/log/diolive
-
cd /etc/aws-kinesis
-
sudo nano agent.json
-
Copiar conteúdo do arquivo agent.json
-
agent.json -> "kinesis.endpoint": "kinesis..amazonaws.com"
-
AWS Console -> EC2 -> Instances -> Select Instance -> Security -> Modify IAM Role -> Create New Role -> EC2 -> Administrator Access -> rolename “ec2-admin-role” -> save
- sudo service aws-kinesis-agent start
- sudo chkconfig aws-kinesis-agent on (start with instance)
- _cd ~_
- sudo ./LogGenerator.py 500000
- tail -f /var/log/aws-kinesis-agent/aws-kinesis-agent.log
- AWS Console -> S3 -> select bucket -> selecionar arquivo e download
- sudo service aws-kinesis-agent restart
- sudo ./LogGenerator.py
- tail -f /var/log/aws-kinesis-agent/aws-kinesis-agent.log
- AWS Console -> Kinesis Streams -> select stream -> monitoring
- AWS Console -> glue databrew -> create new project -> create new role -> create project
- Create dataset -> S3 -> formato CSV