Accumulators:

https://www.edureka.co/blog/spark-accumulators-explained
https://www.brosinski.com/post/extending-spark-accumulators/

Streaming

http://henning.kropponline.de/2015/05/16/hdfs-spooling-directory-with-spark/