Pyspark-tutorial

How to use spark using python by Colab or Docker

| Pyspark을 사용하며 실습 환경은 Google Colab과 개인컴퓨터에서 설치하는 Spark standalone 모드의 Spark입니다.

2023.07 (Docker)

docker를 이용한 spark 설치

root 경로에서 docker compose up 실행

.
├── 0.Set-up
├── 1.Pyspark-tutorial
├── 2.RDD, Dataframe
├── 3.Spark_SQL
├── 4.Spark_ML
├── Dockerfile ## here
├── README.md
├── docker-compose.yml ## here
├── hadoop
└── spark

만약 필요한 jdbc 드라이버가 존재한다면 /usr/local/spark-3.3.1-bin-hadoop3/jars 경로에 넣어주면 됩니다. (Dockerfile에 작성 요망)

RUN cd /usr/local/spark-3.3.1-bin-hadoop3/jars && \
  if ! wget -P /usr/local/spark-3.3.1-bin-hadoop3/jars https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/2.1.0.14/redshift-jdbc42-2.1.0.14.jar; then \
      echo "Failed to download Redshift JDBC driver"; \
  fi

2023.03.01 (Colab)

colab 내 java의 버젼 : 11.0.17

openjdk version "11.0.17" 2022-10-18
OpenJDK Runtime Environment (build 11.0.17+8-post-Ubuntu-1ubuntu220.04)
OpenJDK 64-Bit Server VM (build 11.0.17+8-post-Ubuntu-1ubuntu220.04, mixed mode,  sharing)

spark 3.3.1이 위의 버젼과 호환이 가능해짐
다음과 같이 설정이 바뀜

!pip install pyspark==3.3.1 py4j==0.10.9.5 
!pip install -q findspark

import findspark
findspark.init()

from pyspark.sql import SparkSession
from pyspark import SparkConf

conf = SparkConf()
conf.set("spark.app.name", "logistic-regression")
conf.set("spark.master", "local[*]")

# Singleton pattern
spark = SparkSession.builder\
        .config(conf=conf)\
        .getOrCreate()

# csv read to dataframe
df = spark.read.format("text").load("1800.csv")

2023 이전

Spark Version
- spark-3.3.1
- hadoop2
- JAVA : openjdk 1.8.0_352
Pyspark was run in colab and practiced.
colab 내부에 아래의 코드를 통해 spark 실습 환경을 구성한 후 spark context 및 session을 구동시켜야합니다.
- Within the colab, you must configure the spark practice environment using the code below before running the spark context and session.
need to update-alternatives (Java 버젼 세팅 , Java Version Settings)
- openjdk version "1.8.0_352"
  - (OpenJDK Runtime Environment (build 1.8.0_352-8u352-ga-1~20.04-b08))

!apt-get install openjdk-8-jdk-headless -qq > /dev/null
!wget -q https://downloads.apache.org/spark/spark-3.3.1/spark-3.3.1-bin-hadoop2.tgz
!tar -xvf spark-3.3.1-bin-hadoop2.tgz
!pip install -q findspark
!pip install pyspark

import os
os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-8-openjdk-amd64"
os.environ["SPARK_HOME"] = "/content/spark-3.3.1-bin-hadoop2"

# need to update-alternatives => openjdk version "1.8.0_352"
!update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
!java -version

import findspark
findspark.init()

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("spark3_test").master("local[*]").getOrCreate()

import pyspark

# sc object generation
from pyspark import SparkContext, SparkConf

conf = SparkConf().setMaster("local").setAppName("myfriend")
sc = SparkContext.getOrCreate(conf=conf)

HaJunYoo/Pyspark-tutorial

Pyspark-tutorial

How to use spark using python by Colab or Docker

2023.07 (Docker)

2023.03.01 (Colab)

2023 이전