How to import spark.implicits._ in Spark 2.2: error “value toDS is not a member of org.apache.spark.rdd.RDD”
In Data we trust
I wrote about how to import implicits in spark 1.6 more than 2 years ago. But things have changed in Spark 2.2: the first thing you need to do when coding in Spark 2.2 is to set up an SparkSession object. SparkSession is the entry point to programming Spark with DataSet and DataFrame.
Like Spark 1.6, spark.implicits are required to be able to use Spark’s API for DataSets and DataFrames in version 2.2. And like version 1.6, an instance of SparkContext is needed in Spark 2.2 before being able to import spark.implicits. Since each instance of SparkSession comes with and an instance of SparkContext associated with it, all you have to do is to create an object of SparkSession and you’re set.
I have seen other posts that mention bits and pieces of how to do it. Here I give you the full code that works just fine and you can tweek it based on your requirements:
import org.apache.spark.sql._ import org.apache.log4j._ object sparkSQLWithCaseClass { case class Person (ID: Int, name: String) def mapper(l: String): Person = { val fields = l.split(‘,’) val person: Person = Person(fields(0).toInt, fields(1)) return person } def main(args: Array[String]){ Logger.getLogger(“org”).setLevel(Level.ERROR) val spark = SparkSession.builder.appName(“Spark SQL”).getOrCreate() val lines = spark.sparkContext.textFile(“../../people.csv”) val people = lines.map(mapper) import spark.implicits._ val schemaPeople = people.toDS() schemaPeople.printSchema() schemaPeople.createOrReplaceTempView(“people”) val t = spark.sql(“select * from people where age >= 13”) val res = t.collect() res.foreach(println) spark.stop() } }
import org.apache.spark.sql._
import org.apache.log4j._
object sparkSQLWithCaseClass {
case class Person (ID: Int, name: String)
def mapper(l: String): Person = {
val fields = l.split(‘,’)
val person: Person = Person(fields(0).toInt, fields(1))
return person
def main(args: Array[String]){
Logger.getLogger(“org”).setLevel(Level.ERROR)
val spark = SparkSession.builder.appName(“Spark SQL”).getOrCreate()
val lines = spark.sparkContext.textFile(“../../people.csv”)
val people = lines.map(mapper)
import spark.implicits._
val schemaPeople = people.toDS()
schemaPeople.printSchema()
schemaPeople.createOrReplaceTempView(“people”)
val t = spark.sql(“select * from people where age >= 13”)
val res = t.collect()
res.foreach(println)
spark.stop()
Tagged
Spark
Spark 2.2
spark.implicits
SparkSession
SparkSQL
value toDS is not a member of org.apache.spark.rdd.RDD
Big and streaming data enthusiast based in Melbourne, Australia and travelling throughout Asia View all posts by Saeed Barghi
Pingback: How to import org.apache.spark.sql.SQLContext.implicits in Spark 1.6: error “value toDF is not a member of org.apache.spark.rdd.RDD” | The Business Intelligence Palace
Fill in your details below or click an icon to log in:
You are commenting using your WordPress.com account. ( Log Out / Change )
You are commenting using your Google account. ( Log Out / Change )
You are commenting using your Twitter account. ( Log Out / Change )
You are commenting using your Facebook account. ( Log Out / Change )
Connecting to %s
Notify me of new comments via email.
Notify me of new posts via email.
June 2019 (1)
April 2019 (1)
February 2019 (1)
January 2019 (1)
May 2018 (1)
April 2018 (2)
January 2018 (1)
December 2017 (1)
October 2017 (1)
September 2017 (1)
August 2017 (1)
June 2016 (2)
May 2016 (1)
July 2015 (1)
February 2015 (2)
May 2014 (1)
April 2014 (1)
June 2013 (1)
March 2012 (1)
September 2011 (1)
August 2011 (3)
June 2011 (1)
March 2011 (1)
February 2011 (2)
January 2011 (2)
Enter your email address to subscribe to this blog and receive notifications of new posts by email.
Join 29 other followers
Sign me up!