Scala Programming Projects

My highlights and comments

"You can also use the module to execute SQL queries as you would with a relational database." ("Fetching And Persisting Bitcoin Market Data" section)

I frequently use this to explore data in ORC files using SQL queries:

val df = spark.read.orc("gs://[bucket and path I am reading ORC files from]")
val df.createOrReplaceTempView("[viewName]")
spark.sql("SELECT * FROM [viewName] WHERE ...").show(false)


Created: 2021-09-10 09:27:14 Last updated: 2021-09-10 09:27:14

Top