Spark is probably the most popular tool in the Hadoop ecosystem nowdays. Apart from the performance improvements it offers over standard MapReduce jobs, it simplifies considerably the process of developing an application because it offers high level programming APIs like the DataFrames API.
The default version included in the platform is Spark 2.4.0, and you can easily start an interactive shell:
Similarly to start an interactive session using Python:
If using Python we recommend that you use the Anaconda version provided through Modules: Additional Software.
To use it with Python from the Anaconda distribution first load the desired anaconda version of the module, for example:
module load anaconda2/2018.12
If using Anaconda, then you can also use ipython for the interactive pyspark session so you get a nicer CLI:
To submit a job:
spark-submit --name testWC test.py input output
The jobs will be submitted to YARN and queued for execution. Depending on the load of the platform the execution will take more or less time.
For further information on how to use Spark you can check the Spark Tutorial that we have prepared to get you started. For more information you can check the PySpark Course Material and the Sparklyr Course Material, these are courses that you can also attend to learn more. Finally, you can also find useful the Spark Guide in the CDH documentation, and of course, the great documentation provided by the Spark project.