首页 > 代码库 > Pig —Multi-Query Execution

Pig —Multi-Query Execution

A = LOAD ‘/user/input/t.txt‘ as (k:chararray,c:int);
B = group A BY k;
C = foreach B generate group,SUM(A.c);

store C into ‘/user/output/test1.out‘;
DUMP C;
store C into ‘/user/output/test2.out‘;
A = LOAD ‘/user/input/t.txt‘ as (k:chararray,c:int);
B = group A BY k;
C = foreach B generate group,SUM(A.c);

store C into ‘/user/output/test1.out‘;

store C into ‘/user/output/test2.out‘;


With multi-query execution Pig processes an entire script or a batch of statements at once.Will create a batch Job to process the data

Turning it On or Off

Multi-query execution is turned on by default. To turn it off and revert to Pig‘s "execute-on-dump/store" behavior, use the "-M" or "-no_multiquery" options.

To run script "myscript.pig" without the optimization, execute Pig as follows:

$ pig -M myscript.pig
or
$ pig -no_multiquery myscript.pig

the first code will produce three mapred Job for: 

1.store C into ‘/user/output/test1.out‘

2.DUMP C

3.store C into ‘/user/output/test2.out‘ 

while the seconde code will only produce:one mapred Job

if we run the second code by: pig -no_multiquery test.pig it will also produce two Jobs

Store vs. Dump

With multi-query exection, you want to use STORE to save (persist) your results. You do not want to use DUMP as it will disable multi-query execution and is likely to slow down execution. (If you have included DUMP statements in your scripts for debugging purposes, you should remove them.)