Pinned Repositories
azure-cloud-handler
Python library to interact with some resources on Azure as AKS (Azure kubernetes service) and Data Lake Storage Gen2
azure-functions-ingestion
Example of how to use Azure Functions to Ingest an API data to datalake
azure-spark-on-kubernetes
Spark on Kubernetes with Azure resources: Azure Kubernetes Service (AKS), Azure Data Lake Storage Gen2 and Azure Synapse
Chat-Gourmet-AI
A RAG system to generate recipe based on the ingredient available.
daft-lab
This repository showcases a data engineering project using Daft and Apache Iceberg to transform IMDB datasets
data-academy
project repo to work in Azure enviroment
desafio_bootcamp_data_engineer
ingestion-on-postgres
Ingestion of NY Taxy data on Postgres Database with Python or Spark
lineage-keeper
A lightweight lineage tool based on Spark and Delta Lake
spark-dev-env-docker
Spark development environment for kubernetes, spark-submit and jupyter notebook
otacilio-psf's Repositories
otacilio-psf/spark-dev-env-docker
Spark development environment for kubernetes, spark-submit and jupyter notebook
otacilio-psf/azure-cloud-handler
Python library to interact with some resources on Azure as AKS (Azure kubernetes service) and Data Lake Storage Gen2
otacilio-psf/azure-spark-on-kubernetes
Spark on Kubernetes with Azure resources: Azure Kubernetes Service (AKS), Azure Data Lake Storage Gen2 and Azure Synapse
otacilio-psf/azure-functions-ingestion
Example of how to use Azure Functions to Ingest an API data to datalake
otacilio-psf/Chat-Gourmet-AI
A RAG system to generate recipe based on the ingredient available.
otacilio-psf/daft-lab
This repository showcases a data engineering project using Daft and Apache Iceberg to transform IMDB datasets
otacilio-psf/data-academy
project repo to work in Azure enviroment
otacilio-psf/flights-price-watch
Watch prices and compare with historical for selected filghs
otacilio-psf/igti-cde-mod3-desafio
Challenge solved using Google Cloud Platform Dataproc and Storage services
otacilio-psf/llm-zoomcamp-otacilio
My space for LLM Zoomcamp class
otacilio-psf/data-eng-open-source-tools
Explore and test Open-sources tools for Data Engineering in standalone mode or integrated with a ecosystem
otacilio-psf/ingestion-on-postgres
Ingestion of NY Taxy data on Postgres Database with Python or Spark
otacilio-psf/lineage-keeper
A lightweight lineage tool based on Spark and Delta Lake
otacilio-psf/Daft
Distributed DataFrame for Python designed for the cloud, powered by Rust
otacilio-psf/data-engineering-roadmap
roadmap de engenharia de dados da jornada 2024
otacilio-psf/dev_api_with_flask
Developing an REST API with Flask MicroFramework
otacilio-psf/docker-bigdata
Big Data Ecosystem Docker
otacilio-psf/docker-nosql
Differents NoSql Databases with docker: mongoDB and Redis
otacilio-psf/DockerSwarm-MinIO
Deploy MinIO storage server in Docker Swarm
otacilio-psf/elastic-stack-labs
otacilio-psf/igti-cde-mod3-trab-pratico
Trabalho prático modulo 3 Cloud Data Engineer IGTI
otacilio-psf/igti-cde-practice-one-aws
Practice one from Cloud Data Engineer by IGTI
otacilio-psf/kafka-samples
Kafka Lab done with docker
otacilio-psf/llm-telegram-bot
This project is a sample project of how to create a Telegram Bot leveraging LLM and RAG
otacilio-psf/otacilio-psf
otacilio-psf/otacilio-psf.github.io
Portifolio
otacilio-psf/pyspark-codespace
Template in how to use PySpark + DeltaLake + Azure Storage + test cases in Codespace
otacilio-psf/spark-image
Spark image built-in with connectors to Common data sources and Delta lake
otacilio-psf/sql-challenge-platform
otacilio-psf/workshop-dw-pagando-pouco
workshop 03 - como montar um dw pagando pouco