Spark Error: Failed to Send RPC to Datanode

In Data we trust

This past week we had quite few issues with users not being able to run Spark jobs running in YARN Cluster mode. Particularly a team that was on tight schedule used to get errors like this all the time:

java.io.IOException: Failed to send RPC 8277242275361198650 to datanode-055: java.nio.channels.ClosedChannelException

Mostly accompanied by error messages like:

org.apache.spark.SparkException: Error sending message [message = Heartbeat(9,[Lscala.Tuple2;@e47ba81,BlockManagerId(9, datanode-50 , 43381))]

ERROR Executor: Exit as unable to send heartbeats to driver more than 60 times

These errors basically mean the connection between Spark driver and executors are broken, mainly because executor is killed. This could happen because of a number of reasons:

1- We realized this happens a lot more often when our cluster is too busy and has hit maximum usage. What it means is that executors are accepted to DataNodes, but they fail to acquire enough memory on the datanode and therefore get killed.

2- Metaspace attempts to grow beyond the executor(JVM) memory limits, resulting in loss of executors.The best way to stop this error from appearing is to set below properties when launching Spark-Shell or submitting application using spark-submit:

spark.driver.extraJavaOptions = -XX:ReservedCodeCacheSize=100M-XX:MaxMetaspaceSize=256m -XX:CompressedClassSpaceSize=256m spark.executor.extraJavaOptions = -XX:ReservedCodeCacheSize=100M -XX:MaxMetaspaceSize=256m -XX:CompressedClassSpaceSize=256m

spark.driver.extraJavaOptions = -XX:ReservedCodeCacheSize=100M-XX:MaxMetaspaceSize=256m

-XX:CompressedClassSpaceSize=256m

spark.executor.extraJavaOptions = -XX:ReservedCodeCacheSize=100M

-XX:MaxMetaspaceSize=256m

Please note that depending on your project and code, you may need to increase the values mentioned above.

3- Network is slow for whatever reason. In our case, this was caused by a change in DNS which resulted in turning off caching.This case could be fixed by adjusting spark.executor.heartbeatInterval and spark.network.timeout. Default values for these 2 parameters are 10s and 120s. You can adjust these 2 values based on how your network, the only point to consider here is that the later property, spark.network.timeout, should be greater than the first one.

If none of what mentioned above helps your situation, then it is something you need to take to your cluster’s administrator. There could be something wrong with the datanodes where executors are sent to that admins are not aware of.

Happy coding!

Twitter

Facebook

Tagged

datanode

Error

Spark

YARN

Big and streaming data enthusiast based in Melbourne, Australia and travelling throughout Asia View all posts by Saeed Barghi

Craig R says: March 26, 2018 at 6:16 pm This may be related, so posting here: https://issues.apache.org/jira/browse/YARN-4714 Reply

This may be related, so posting here:

https://issues.apache.org/jira/browse/YARN-4714

Saeed Barghi says: March 26, 2018 at 11:47 pm Thanks Craig for sharing the official bug report, that is exactly what we were experiencing in our environment. We reduced the amount of virtual memory available to YARN applications and things got better. Reply

Thanks Craig for sharing the official bug report, that is exactly what we were experiencing in our environment. We reduced the amount of virtual memory available to YARN applications and things got better.

Pingback: #StackBounty: #hadoop #apache-spark Spark Error: Failed to Send RPC to Datanode – TechUtils.in

Fill in your details below or click an icon to log in:

You are commenting using your WordPress.com account. ( Log Out / Change )

You are commenting using your Google account. ( Log Out / Change )

You are commenting using your Twitter account. ( Log Out / Change )

You are commenting using your Facebook account. ( Log Out / Change )

Connecting to %s

Notify me of new comments via email.

Notify me of new posts via email.

June 2019 (1)

April 2019 (1)

February 2019 (1)

January 2019 (1)

May 2018 (1)

April 2018 (2)

January 2018 (1)

December 2017 (1)

October 2017 (1)

September 2017 (1)

August 2017 (1)

June 2016 (2)

May 2016 (1)

July 2015 (1)

February 2015 (2)

May 2014 (1)

April 2014 (1)

June 2013 (1)

March 2012 (1)

September 2011 (1)

August 2011 (3)

June 2011 (1)

March 2011 (1)

February 2011 (2)

January 2011 (2)

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 29 other followers

Sign me up!