Script_Kiddie

A list of hack scripts to use

spark_aws.sh:

Use this script to install pyspark to your jupyter linux environment or colab environment.
In your linux jupyter or colab use:

!wget https://raw.githubusercontent.com/aivan2798/Script_Kiddie/main/spark_aws.sh
!sh spark_aws.sh YOUR_AWS_KEY_ID YOUR_AWS_SECRET_KEY
from dotenv import load_dotenv
load_dotenv()

NB: Don't forget to replace YOUR_AWS_KEY_ID and YOUR_AWS_SECRET_KEY with your actual AWS KEY VALUE and AWS secret key respectively

To start the spark session for digital ocean:

from pyspark.sql import SparkSession

aws_endpoint = "nyc3.digitaloceanspaces.com"
spark = SparkSession \
.builder \
.appName("pyspark test") \
.config("spark.pyspark.python", "python") \
.config("fs.s3a.endpoint", aws_endpoint)\
.getOrCreate()

For aws: