/spark-scala-compose

up and running a local spark cluster in your machine

Primary LanguageScala

spark-scala

results

with stopwords included

image

with stopwords removed

image

A note on HDFS and Hadoop on windows:

It is silly to use windows in the first place, but you happen to be in such a dire situation, take the following path:

1- download winutils from https://github.com/kontext-tech/winutils

2- extract and put the \bin in C:\\hadoop

3- Add C:\hadoop\bin to your environmental variable as HADOOP_HOME

4- copy hdfs.dll and hadoop.dll from C:\\hadoop\bin to C:\\Windows\System32

and you are done!