首页 > 代码库 > SparkSQL External Datasource简易使用之AVRO
SparkSQL External Datasource简易使用之AVRO
下载源码&编译：
git clone https://github.com/databricks/spark-avro.gitsbt/sbt package
Maven GAV：
groupId: com.databricks.sparkartifactId: spark-avro_2.10version: 0.1
$SPARK_HOME/conf/spark-env.sh
export SPARK_CLASSPATH=/home/spark/software/source/spark_package/spark-avro/target/scala-2.10/spark-avro_2.10-0.1.jar:$SPARK_CLASSPATH
测试数据下载:
wget https://github.com/databricks/spark-avro/raw/master/src/test/resources/episodes.avro
Scala API：
import org.apache.spark.sql.SQLContextval sqlContext = new SQLContext(sc)import com.databricks.spark.avro._val episodes = sqlContext.avroFile("file:///home/spark/software/data/episodes.avro")import sqlContext._episodes.select(‘title).collect()
SQL:
CREATE TEMPORARY TABLE episodesUSING com.databricks.spark.avroOPTIONS (path "file:///home/spark/software/data/episodes.avro");select * from episodes;
SparkSQL External Datasource简易使用之AVRO
声明：以上内容来自用户投稿及互联网公开渠道收集整理发布，本网站不拥有所有权，未作人工编辑处理，也不承担相关法律责任，若内容有误或涉及侵权可进行投诉： 投诉/举报 工作人员会在5个工作日内联系你，一经查实，本站将立刻删除涉嫌侵权内容。