Hadoop Error org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block
In Data we trust
Like almost all Mondays, today was a very challenging one. The first thing I noticed was that our primary namenode had faced some issues over the weekend and went down. Which means secondary namenode, namenode-02, was active. I checked namenode-01 and made sure it is okay before making it active again. After that, I was made aware of when I arrived at office was that a very critical range of our ETL jobs has failed for over 12 hours.
Like everyone else would do when they get failed jobs, the first thing I did was to look into the logs for those jobs. All of them have failed with this error:
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block
It is not hard to guess that hdfs is complaining about not being able to find some blocks of data it needs. So I navigated to Ambari’s HDFS page. But there were not any missing blocks being reported.
Therefore we can conclude that the data blocks are there, but for some reason namenode is not able to access them when jobs are submitted to cluster.
The second thing I noticed was that after primary namenode was made active, jobs started working fine and completing successfully. That hints there should have been something with namenode-02. So I navigated to our 2 namenode’s web UI:
There it is! I know we have 33 datanodes in our cluster, but the secondary namenode shows only 30. So what I did was to restart node manager on those datanodes that were not listed for namenode-02 and refreshed the page:
Now all the datanodes are recognized by both namenodes and everyone lives happily ever after!
Note that you may check namenodes’ web UI and don’t see any missing datanodes. But still, restarting node managers on all datanodes will resolve your issue.
Tagged
Blocks
Hadoop
hdfs
java.io.IOException
Missing
namenode
org.apache.hadoop.hdfs.BlockMissingException
Big and streaming data enthusiast based in Melbourne, Australia and travelling throughout Asia View all posts by Saeed Barghi
Fill in your details below or click an icon to log in:
You are commenting using your WordPress.com account. ( Log Out / Change )
You are commenting using your Google account. ( Log Out / Change )
You are commenting using your Twitter account. ( Log Out / Change )
You are commenting using your Facebook account. ( Log Out / Change )
Connecting to %s
Notify me of new comments via email.
Notify me of new posts via email.
June 2019 (1)
April 2019 (1)
February 2019 (1)
January 2019 (1)
May 2018 (1)
April 2018 (2)
January 2018 (1)
December 2017 (1)
October 2017 (1)
September 2017 (1)
August 2017 (1)
June 2016 (2)
May 2016 (1)
July 2015 (1)
February 2015 (2)
May 2014 (1)
April 2014 (1)
June 2013 (1)
March 2012 (1)
September 2011 (1)
August 2011 (3)
June 2011 (1)
March 2011 (1)
February 2011 (2)
January 2011 (2)
Enter your email address to subscribe to this blog and receive notifications of new posts by email.
Join 29 other followers
Sign me up!