How to import org.apache.spark.sql.SQLContext.implicits in Spark 1.6: error “value toDF is not a member of org.apache.spark.rdd.RDD”
In Data we trust
Note: If you’re using Spark 2.2, please read this post
I am doing a mini project for my company using Spark/Scala and have been stuck with the error mentioned in the title for a couple of days. Googling that error suggested to import org.apache.spark.sql.SQLContext.implicits, and that’s what I did:
import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import org.apache.spark.sql._ import org.apache.spark.sql.SQLContext import org.apache.spark.sql.SQLContext.implicits import org.apache.spark.SparkConf object TestSQLContext { […..] def main(args:Array[String]) { […..] } }
And that was the start of the problem: my application started to give a new error: object SQLContext is not a member of package org.apache.spark.sql [error] Note: class SQLContext exists, but it has no companion object.
The problem is, none of those online posts mention that we need to create an instance of org.apache.spark.sql.SQLContext before being able to use its members and methods. This is the right way to do it:
import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import org.apache.spark.sql._ import org.apache.spark.SparkConf object Hi {
case class DimC(ID:Int, Name:String, City:String, EffectiveFrom:Int, EffectiveTo:Int)
def main(args:Array[String]) { val conf = new SparkConf().setAppName(“LoadDW”) val sc = new SparkContext(conf) val sqlContext= new org.apache.spark.sql.SQLContext(sc) import sqlContext.implicits._ val fDimCustomer = sc.textFile(“DimCustomer.txt”)
var dimCustomer1 = fDimCustomer.map(_.split(‘,’)).map(r=>DimC(r(0).toInt,r(1),r(2),r(3).toInt,r(4).toInt)).toDF
dimCustomer1.registerTempTable(“Cust_1”)
val customers = sqlContext.sql(“select * from Cust_1”)
customers.show() } }
Hope this post helps and please do not hesitate to ask your questions in comments section.
Cheers.
Tagged
Hadoop
org.apache.spark.sql.SQLContext
Scala
Spark
Big and streaming data enthusiast based in Melbourne, Australia and travelling throughout Asia View all posts by Saeed Barghi
lsdbtech says: December 22, 2015 at 3:01 pm Are you writing your Scala directly on your Spark cluster? Reply
Are you writing your Scala directly on your Spark cluster?
Saeed Barghi says: December 23, 2015 at 11:15 pm Not sure what you mean, but you can submit your scala code on any machine that is connected to the cluster, via ssh for example. Reply
Not sure what you mean, but you can submit your scala code on any machine that is connected to the cluster, via ssh for example.
Xin says: February 8, 2016 at 4:37 pm Great! Your post helps me! It saved my day! Thank you very much! Reply
Great! Your post helps me! It saved my day! Thank you very much!
Kshitija Joshi says: April 23, 2016 at 6:37 am Thank you for the detailed explanation. This article is really helpful to understand the problem while importing “org.apache.spark.sql.SQLContext.implicits” in Spark. Kudos! Reply
Thank you for the detailed explanation. This article is really helpful to understand the problem while importing “org.apache.spark.sql.SQLContext.implicits” in Spark. Kudos!
Gianna says: August 15, 2016 at 12:53 pm That was pretty helpful! Great! Reply
That was pretty helpful! Great!
Sohail says: November 9, 2016 at 12:17 am it took me a whole day to figure this out.. thanks for the post Reply
it took me a whole day to figure this out.. thanks for the post
nven says: December 14, 2016 at 9:01 pm Thanks !!, really saved time -) Reply
Thanks !!, really saved time -)
ramya5a says: July 7, 2017 at 4:37 am Thank you so much!! Reply
Thank you so much!!
venky says: July 31, 2017 at 9:21 am Fantastic … Thank you very much Reply
Fantastic … Thank you very much
sai says: November 24, 2017 at 5:54 pm This was really helpful. Thanks a lot. Reply
This was really helpful. Thanks a lot.
Pingback: How to import org.apache.spark.sql.SQLContext.implicits in Spark 2.2: error “value toDS is not a member of org.apache.spark.rdd.RDD” | The Business Intelligence Palace
Fill in your details below or click an icon to log in:
You are commenting using your WordPress.com account. ( Log Out / Change )
You are commenting using your Google account. ( Log Out / Change )
You are commenting using your Twitter account. ( Log Out / Change )
You are commenting using your Facebook account. ( Log Out / Change )
Connecting to %s
Notify me of new comments via email.
Notify me of new posts via email.
June 2019 (1)
April 2019 (1)
February 2019 (1)
January 2019 (1)
May 2018 (1)
April 2018 (2)
January 2018 (1)
December 2017 (1)
October 2017 (1)
September 2017 (1)
August 2017 (1)
June 2016 (2)
May 2016 (1)
July 2015 (1)
February 2015 (2)
May 2014 (1)
April 2014 (1)
June 2013 (1)
March 2012 (1)
September 2011 (1)
August 2011 (3)
June 2011 (1)
March 2011 (1)
February 2011 (2)
January 2011 (2)
Enter your email address to subscribe to this blog and receive notifications of new posts by email.
Join 29 other followers
Sign me up!