首页 > 代码库 > 简单示例用例(Simple Example Use Cases)--hive GettingStarted用例翻译

简单示例用例(Simple Example Use Cases)--hive GettingStarted用例翻译

2024-10-12 13:48:02   214人阅读

1、MovieLens User Ratings

First, create a table with tab-delimited text file format:

首先，创建一个通过tab分隔的表：

CREATE TABLE u_data (

  userid INT,

  movieid INT,

  rating INT,

  unixtime STRING)

ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t‘

STORED AS TEXTFILE;

Then, download the data files from MovieLens 100k on the GroupLens datasets page (which also has a README.txt file and index of unzipped files):

然后，下载数据文件从下面方法：

wget http://files.grouplens.org/datasets/movielens/ml-100k.zip

or:

curl --remote-name http://files.grouplens.org/datasets/movielens/ml-100k.zip

Note:  If the link to GroupLens datasets does not work, please report it on HIVE-5341 or send a message to the user@hive.apache.org mailing list.

Unzip the data files:

解压缩这个文件：

unzip ml-100k.zip

And load u.data into the table that was just created:

并且加载数据到刚刚创建的u_data表中：

LOAD DATA LOCAL INPATH ‘<path>/u.data‘ OVERWRITE INTO TABLE u_data;

Count the number of rows in table u_data:

统计表u_data的行数：

SELECT COUNT(*) FROM u_data;

Note that for older versions of Hive which don‘t include HIVE-287, you‘ll need to use COUNT(1) in place of COUNT(*).

Now we can do some complex data analysis on the table u_data:

现在我们可以做一些复杂的数据分析针对表u_data：

Create weekday_mapper.py:

import sys

import datetime

for line in sys.stdin:

  line = line.strip()

  userid, movieid, rating, unixtime = line.split(‘\t‘)

  weekday = datetime.datetime.fromtimestamp(float(unixtime)).isoweekday()

  print ‘\t‘.join([userid, movieid, rating, str(weekday)])

Use the mapper script:

使用这个脚本：

CREATE TABLE u_data_new (

  userid INT,

  movieid INT,

  rating INT,

  weekday INT)

ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘\t‘;

add FILE weekday_mapper.py;

INSERT OVERWRITE TABLE u_data_new

SELECT

  TRANSFORM (userid, movieid, rating, unixtime)

  USING ‘python weekday_mapper.py‘

  AS (userid, movieid, rating, weekday)

FROM u_data;

解释：这里通过python脚本清洗表u_data中数据，使用python脚本通过

TRANSFORM (userid, movieid, rating, unixtime)   --输入字段

  USING ‘python weekday_mapper.py‘              --脚本处理

  AS (userid, movieid, rating, weekday)         --输出字段

SELECT weekday, COUNT(*)

FROM u_data_new

GROUP BY weekday;

Note that if you‘re using Hive 0.5.0 or earlier you will need to use COUNT(1) in place of COUNT(*).

2、Apache Weblog Data

The format of Apache weblog is customizable, while most webmasters use the default.
For default Apache weblog, we can create a table with the following command.

More about RegexSerDe can be found here in HIVE-662 and HIVE-1719.

CREATE TABLE apachelog (

host STRING,

identity STRING,

user STRING,

time STRING,

request STRING,

status STRING,

size STRING,

referer STRING,

agent STRING)

ROW FORMAT SERDE ‘org.apache.hadoop.hive.serde2.RegexSerDe‘

WITH SERDEPROPERTIES (

"input.regex" = "([^ ]*) ([^ ]*) ([^ ]*) (-|\\[^\\]*\\]) ([^ \"]*|\"[^\"]*\") (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\".*\") ([^ \"]*|\".*\"))?"

)

STORED AS TEXTFILE;

简单示例用例(Simple Example Use Cases)--hive GettingStarted用例翻译


联系
我们