Answer by G.V. Sridhar for Spark Streaming failing on YARN Cluster
I was able to solve this by providing the driver memory and executor memory at run time.spark-submit --driver-memory 1g --executor-memory 1g --class com.package.App --master yarn --deploy-mode cluster...
View ArticleAnswer by wanbo for Spark Streaming failing on YARN Cluster
The file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip you submit does not exist.
View ArticleAnswer by Sidharth for Spark Streaming failing on YARN Cluster
I recently ran into the same issue. Here was my scenario:Cloudera Managed CDH 5.3.3 cluster with 7 nodes. I was submitting the job from one of the nodes and it used to fail in both yarn-cluster and...
View ArticleAnswer by Murtaza Kanchwala for Spark Streaming failing on YARN Cluster
While running with the Yarn Cluster mode, you always need to specify the other Memory settings for your executors and there individually memory, Plus you always need to specify the driver details also....
View ArticleSpark Streaming failing on YARN Cluster
I have a cluster of 1 master and 2 slaves. I'm running a spark streaming in master and I want to utilize all nodes in my cluster. i had specified some parameters like driver memory and executor memory...
View Article
