sparksql-magic
Spark SQL magic command for Jupyter notebooks.
Prerequisites
- Python >= 3.6
- PySpark >= 2.3.0
- IPython >= 7.4.0
Install
pip install sparksql-magic
Usage
Load
%load_ext sparksql_magic
Config
%config SparkSql.limit=<INT>
| Option | Default | Description |
|---|---|---|
SparkSql.limit |
20 | The maximum number of rows to display |
Parameter
%%sparksql [-c|--cache] [-e|--eager] [-v|--view VIEW] [-l|--limit LIMIT] [variable]
<QUERY>
| Parameter | Description |
|---|---|
-c --cache |
Cache dataframe |
-e --eager |
Cache dataframe with eager load |
-v VIEW --view VIEW |
Create or replace temporary view |
-l LIMIT --limit LIMIT |
The maximum number of rows to display (Default: SparkSql.limit) |
variable |
Capture dataframe in a local variable |