Big Data Journal Projects

Amazon-Redshift-cluster-to-analyze-USA-Domestic-flight-data

worked with an Amazon Redshift cluster to analyze USA Domestic flight data. Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse service that makes it simple and cost-effective to efficiently analyze all your data using your existing business intelligence tools. It is optimized for datasets ranging from a few hundred gigabytes to a petabyte or more and costs less than $1,000 per terabyte per year, a tenth the cost of most traditional data warehousing solutions

Analysing-Census-Data-using-aws

Use aws-emr and aws-redshift to analyse dataset of adult census of USA

Analyzing-Twitter-in-real-time-with-Kinesis-Lambda-Comprehend-and-ElasticSearch

Analyzing Twitter in real time with Kinesis, Lambda, Comprehend and ElasticSearch

AWS-Data-Lake

AWS Lake Formation makes it easy for you to set up, secure, and manage your data lakes also data discovery using the metadata search capabilities of Lake Formation in the console, and metadata search results restricted by column permissions.

aws-forest-fire-predictive-analytics

Big Data Engineering & Analytics Project

aws-serverless-data-lake-workshop

This workshop is meant to give customers a hands-on experience with mentioned AWS services. Serverless Data Lake workshop helps customers build a cloud-native and future-proof serverless data lake architecture. It allows hands-on time with AWS big data and analytics services including Amazon Kinesis Services for streaming data ingestion

big-data-ecosystem

Project developed during the Cognizant Cloud Data Engineer Bootcamp on the Digital Innovation One platform with the objective of extracting and counting words from a book in plain text format, displaying the most frequent word, through a python algorithm.

big-data-solutions

This repository provides Code examples written in Python,Spark-Scala using primarily boto3 SDK API methods and aws cli examples for majority of the AWS Big Data services. There are also nicley written Wiki articles for most of the common issues/challenges faced within BigData world.

Iot-and-Big-Data-Application-using-aws-and-apache-kafka

Iot,Big Data Analytics using Apache-kafka,spark and other aws services

IoT-Data-with-Amazon-Kinesis

Build a Visualization and Monitoring Dashboard for IoT Data with Amazon Kinesis Analytics and Amazon QuickSight

AWS-Big-Data-Projects/Airline_Data_Analysis

Process to gather streaming data from Airline API using NiFi & batch data using AWS redshift using Sqoop and build a data pipeline to analyse the data using Apache Hive and Druid and compare the performances ,to discuss the hive optimization techniques and visualise the data using AWS Quicksight

AWS-Big-Data-Projects/HeartRate-Monitoring-using-AWS-IOT-and-AWS-KINESIS

you run a script to mimic multiple sensors publishing messages on an IoT MQTT topic, with one message published every second. The events get sent to AWS IoT, where an IoT rule is configured. The IoT rule captures all messages and sends them to Firehose. From there, Firehose writes the messages in batches to objects stored in S3. In S3, you set up a table in Athena and use QuickSight to analyze the IoT data.

AWS-Big-Data-Projects/AWS_File_Trans_Lamda_S3_SNS

AWS Data Engineering Project using Lambda, S3 and SNS

AWS-Big-Data-Projects/awesome-opensource-data-engineering

An Awesome List of Open-Source Data Engineering Projects

AWS-Big-Data-Projects/aws-glue-job-tracker

AWS-Big-Data-Projects/dbt-glue

This repository contains de dbt-glue adapter

AWS-Big-Data-Projects/saas-analytics-infrastructure-on-aws

AWS-Big-Data-Projects/.github

AWS-Big-Data-Projects/amazon-kinesis-data-analytics-blueprints

Kinesis Data Analytics Blueprints are a curated collection of Apache Flink applications. Each blueprint will walk you through how to solve a practical problem related to stream processing using Apache Flink. These blueprints can be leveraged to create more complex applications to solve your business challenges in Apache Flink.

AWS-Big-Data-Projects/amazon-opensearch-batch-indexing-with-aws-lambda

AWS-Big-Data-Projects/arvados

An open source platform for managing and analyzing biomedical big data

AWS-Big-Data-Projects/aws-emr-serverless-using-terraform

AWS-Big-Data-Projects/aws-glue-cdk-cicd

Build, Test and Deploy ETL solutions using AWS Glue and AWS CDK based CI/CD pipelines

AWS-Big-Data-Projects/aws-glue-test-data-generator

AWS Glue Configurable Test Data Generator

AWS-Big-Data-Projects/aws-security-hub-glue-aggregator-terraform

These Terraform modules aggregate Security Hub findings to centralized account using Amazon Kinesis Firehose and AWS Glue

AWS-Big-Data-Projects/bigdata-file-viewer

A cross-platform (Windows, MAC, Linux) desktop application to view common bigdata binary format like Parquet, ORC, AVRO, etc. Support local file system, HDFS, AWS S3, Azure Blob Storage ,etc.

AWS-Big-Data-Projects/ClickHouse

ClickHouse® is a free analytics DBMS for big data

AWS-Big-Data-Projects/data-engineering

Construct a modern data stack and orchestration the workflows to create high quality data for analytics and ML applications.

AWS-Big-Data-Projects/data-engineering-zoomcamp

Free Data Engineering course!

AWS-Big-Data-Projects/data-science-on-aws

AI and Machine Learning with Kubeflow, Amazon EKS, and SageMaker

AWS-Big-Data-Projects/emr-studio-notebook-examples

This repository contains ready-to-use notebook examples for a wide variety of use cases in Amazon EMR Studio.

AWS-Big-Data-Projects/emr-trino-autoscale

AWS-Big-Data-Projects/host-openemr-on-aws-fargate

AWS-Big-Data-Projects/monitor-serverless-datalake

Alerting and notification in a serverless data lake during failures

AWS-Big-Data-Projects/msk-serverless-data-pipeline

AWS-Big-Data-Projects/nextflow

A DSL for data-driven computational pipelines

AWS-Big-Data-Projects/querypal

Web UI for Amazon Athena

AWS-Big-Data-Projects/serverless-datalake

AWS-Big-Data-Projects/serverless-etl-cdk

AWS-Big-Data-Projects/spark-on-aws-lambda