I installed Spark cluster on AWS cloud and showed some its important functionalities.
- The file spark_installation.txt describes the installation procedure. This is done on top of a Hadoop YARN cluster, which I installed previously (for details see Hadoop-Hive project in the current git repository).
- The folder ./Scala_ETL contains the example of ETL (Extract Transfer Load) pipeline written in Scala.
- The folder ./Java_ML contains the example of simple machine learning pipeline in Java.
- The example of data batch processing with Spark (./Streaming).
Note
Each project is accompained by intsruction how to run it on Spark cluster in Scala, Java or in Python environment (see the corresponding How_to_run.txt files).