首页 > 代码库 > Hadoop、Pig、Hive、NOSQL 学习资源收集
Hadoop、Pig、Hive、NOSQL 学习资源收集
(一)hadoop 相关安装部署
1、hadoop在windows cygwin下的部署:
http://lib.open-open.com/view/1333428291655
http://blog.csdn.net/ruby97/article/details/7423088
http://blog.csdn.net/savechina/article/details/5656937
2、hadoop 伪分布式安装:
http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
3、hadoop全分布式安装教程:
http://hi.baidu.com/leejun_2005/item/367da95bd69f4e0ce6c4a581
4、基于Eclipse的Hadoop应用开发环境配置
http://www.cnblogs.com/flyoung2008/archive/2011/12/09/2281400.html
http://blog.sina.com.cn/s/blog_62186b4601012acs.html
关于eclipse无法连接报错:
"Map/Reduce location status updater". org/codehaus/jackson/map/JsonMappingException
经过查询,是由于hadoop的eclipse 插件里面缺少了包
按照这篇文章的说明 修改包后 重新运行成功
http://hi.baidu.com/wangyucao1989/blog/item/279cef87c4b37c34c75cc315.html
如果已经安装了官方插件,发现没法连接的,需要先从eclipse中删除这个jar包.然后重启eclipse,(防止缓存)
然后再放入新jar包, 再重启eclipse.
windows下用eclipse连接linux中的hadoop,并执行mr
http://superlxw1234.iteye.com/blog/1583164
5、单台服务器上安装Hadoop和Hive十五分钟教程
http://rdc.taobao.com/team/top/tag/hadoop-hive-%E5%8D%81%E5%88%86%E9%92%9F%E6%95%99%E7%A8%8B/
ssh-keygen -t dsa -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
http://blogread.cn/it/article/6103?f=wb
(二)hive
1、基于hive的日志统计实战:
http://www.csdn.net/article/2010-11-28/282620
2、Hive实例:CSDN十大常用密码
http://my.oschina.net/leejun2005/blog/81662
3、hive官方教程:
https://cwiki.apache.org/confluence/display/Hive/GettingStarted
4、Hive 随谈(四)– Hive QL
http://www.alidata.org/archives/581 # JOIN
http://wenku.baidu.com/view/242260c489eb172ded63b709.html
5、写好Hive 程序的五个提示
http://www.alidata.org/archives/622 #排序
6、Hadoop数据仓库工具--hive介绍(百度)
http://wenku.baidu.com/view/90dad7659b6648d7c1c7460e.html
7、hive 分享(淘宝网)
http://wenku.baidu.com/view/4e4a801ca76e58fafab003b1.html
8、hive简介(美丽说)
http://wenku.baidu.com/view/0f252121a5e9856a56126025.html
9、Hive学习笔记(阿里巴巴)
http://wenku.baidu.com/view/233308340b4c2e3f5727632a.html
10、Hive - 运用于hadoop的拍字节范围数据仓库(论文)
http://wenku.baidu.com/view/b5aebfe9998fcc22bcd10d8a.html
11、Hive: SQL for Hadoop(An Essential Tool for Hadoop-based Data Warehouses)
http://polyglotprogramming.com/papers/Hive-SQLforHadoop.pdf
12、Programming Hive
http://www.itpub.net/thread-1724707-1-1.html
13、Hive 随谈(六)– Hive 的扩展特性:
File Format、SerDe、Map/Reduce 脚本(Transform)、UDF、UDAF
http://www.alidata.org/archives/604
14、hive 数据倾斜总结
http://www.alidata.org/archives/2109
15、用hive查询json格式的复杂数据
http://blog.cloudera.com/blog/2012/09/analyzing-twitter-data-with-hadoop/
https://github.com/rcongiu/Hive-JSON-Serde
16、同事总结的hive sql 优化
http://hbase.iteye.com/blog/1488745
http://superlxw1234.iteye.com/blog/1564456
17、通过 thrift 接口实现 python 查询 hive 数据仓库
http://slaytanic.blog.51cto.com/2057708/734106
18、通过 thrift 接口实现 php 查询 hive 数据仓库(以及phpHiveAdmin简介)
http://slaytanic.blog.51cto.com/2057708/766230
http://slaytanic.blog.51cto.com/2057708/818721
http://slaytanic.blog.51cto.com/2057708/1071263
19、Hive SQL使用和数据加载的一点总结
http://slaytanic.blog.51cto.com/2057708/782175
20、hive优化之——控制hive任务中的map数和reduce数
http://superlxw1234.iteye.com/blog/1582880
21、hive中一些实用的小技巧
http://superlxw1234.iteye.com/blog/1565774
22、数据仓库数据模型之:极限存储--历史拉链表
http://superlxw1234.iteye.com/blog/1567320
23、Programing Hive读书笔记
http://www.gemini5201314.net/hadoop/programing-hive%E8%AF%BB%E4%B9%A6%E7%AC%94%E8%AE%B0.html
(三)pig
1、pig 实战
http://www.cnblogs.com/xuqiang/archive/2011/06/06/2073601.html
2、pig官方教程
http://pig.apache.org/
3、Apache Pig中文教程集合
http://www.codelast.com/?p=4550
4、Programming Pig
http://ofps.oreilly.com/titles/9781449302641/index.html
http://www.google.com.hk/url?sa=t&rct=j&q=&esrc=http://www.mamicode.com/s&source=web&cd=1&cad=rja&ved=0CCcQFjAA&url=http%3A%2F%2Fbigdata.googlecode.com%2Ffiles%2FOreilly.Programming.Pig.Sep.2011.pdf&ei=DLGDUNbcI4aTiQfus4HADQ&usg=AFQjCNGzTHIYcc2GuU6ko0TgIKm3UN9T5Q&sig2=2DZtn3yP4KVqro7xt_qAOA
5、PigFly:hadoop 统一数据分析平台设计(淘宝)
http://www.docin.com/p-344188827.html
http://coderplay.iteye.com/blog/1233865
6、用 Apache Pig 处理百万歌曲数据(cloudera)
http://blog.cloudera.com/blog/2012/08/process-a-million-songs-with-apache-pig/
7、Pig Latin: A Not-So-Foreign Language for Data Processing(斯坦福大学论文)
http://infolab.stanford.edu/~usriv/papers/pig-latin.pdf
8、Lecture 09: Parallel Databases, Big Data, Map/Reduce, Pig-Latin
http://www.cs.washington.edu/education/courses/csep544/11au/lectures/lecture09-parallel-db.pdf
9、Pig Queries Parsing JSON on Amazons Elastic Map Reduce Using S3 Data
http://eric.lubow.org/2011/hadoop/pig-queries-parsing-json-on-amazons-elastic-map-reduce-using-s3-data/
https://github.com/a-b/elephant-bird/tree/master/javadoc
10、pig cookbook:性能调优
http://pig.apache.org/docs/r0.7.0/cookbook.html
http://pig.apache.org/docs/r0.10.0/perf.html#Replicated-Joins
11、pig stream 用法:
http://wiki.apache.org/pig/PigStreamingFunctionalSpec
http://www.slideshare.net/charmalloc/hadoop-streaming-tutorial-with-python
(四)hadoop原理与编码
1、hadoop使用中的几个小细节
http://blog.csdn.net/needle2/article/details/6182515
2、hadoop中map-reduce相关过程与概念的理解:更多请浏览目录
http://hi.baidu.com/shirdrn/item/085a5518be8bfa797b5f25aa
3、hadoop 0.18 中文版官方文档
http://hadoop.apache.org/docs/r0.20.0/cn/commands_manual.html
4、IBM developerworks:用 Hadoop 进行分布式并行编程系列, 第 1 ~3 部分
http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop1/
http://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop2/index.html
https://www.ibm.com/developerworks/cn/opensource/os-cn-hadoop3/
5、分布式计算开源框架Hadoop介绍
http://www.infoq.com/cn/articles/hadoop-intro
6、Hadoop基本流程与应用开发( Java )
http://www.infoq.com/cn/articles/hadoop-process-develop
7、hadoop 源码分析
http://caibinbupt.iteye.com/?page=2
8、hadoop数据流、作业提交分析
http://www.cnblogs.com/spork/category/226077.html
9、Hadoop管理员的十个最佳实践
http://www.infoq.com/cn/articles/hadoop-ten-best-practice
10、hadoop、hive源码分析及使用分享
http://www.oratea.net/?cat=7#
11、Hadoop计算能力调度器应用和配置(区别于默认的FIFO队列调度)
http://www.cnblogs.com/ggjucheng/archive/2012/07/25/2608817.html
12、浅析Hadoop 中的调度策略
http://www.ibm.com/developerworks/cn/opensource/os-hadoop-scheduling/index.html
http://dongxicheng.org/mapreduce/hadoop-schedulers/
Hadoop-0.20.2公平调度器算法解析
http://dongxicheng.org/mapreduce/hadoop-fair-scheduler/
Hadoop计算能力调度器算法解析
http://dongxicheng.org/mapreduce/hadoop-capacity-scheduler/
Hadoop资源感知调度器简介
http://my.oschina.net/leejun2005/blog/96113
13、hadoop作业调优参数整理及原理
http://blog.sina.com.cn/s/blog_ae33b83901015cm9.html
14、比较全的hadoop源码分析
http://hbase.iteye.com/blog/1024737
15、如何在Hadoop上编写MapReduce程序
http://dongxicheng.org/mapreduce/writing-hadoop-programes/
16、Hadoop学习笔记(二):从map到reduce的数据流
http://www.cnblogs.com/beanmoon/archive/2012/12/08/2805636.html
17、通过Hadoop的API管理Job
http://blog.csdn.net/dajuezhao/article/details/6591058
18、揭秘InputFormat:掌控Map Reduce任务执行的利器
http://www.infoq.com/cn/articles/HadoopInputFormat-map-reduce
19、Hadoop MapReduce开发最佳实践(上篇)
http://www.infoq.com/cn/articles/MapReduce-Best-Practice-1
20、Hadoop实例:二度人脉与好友推荐
http://my.oschina.net/u/176897/blog/99761
21、探索大数据分析和 Hadoop
http://www.ibm.com/developerworks/cn/training/kp/os-kp-hadoop/index.html
(五)数据仓库
1、数据仓库基础培训
http://wenku.baidu.com/view/c788400cba1aa8114431d95b.html
http://wenku.baidu.com/view/412b09e96294dd88d0d26bff.html
2、数据仓库ods基础学习
http://wenku.baidu.com/view/bb3e6263caaedd3383c4d3bf.html
3、HBDW-PM-数据仓库基础
http://wenku.baidu.com/view/e25bd14769eae009581bec5d.html
(六)Oozie工作流
1、Oozie简介
http://www.infoq.com/cn/articles/introductionOozie
2、跟着示例学Oozie
http://www.infoq.com/cn/articles/oozieexample
3、扩展Oozie
http://www.infoq.com/cn/articles/ExtendingOozie
4、oozie相关安装配置与问题解决例子
http://guoyunsky.iteye.com/category/187923
5、oozie总结
http://dirlt.com/oozie.html
(七)HBase
1、hbase官方指南
http://hbase.apache.org/book.html
2、HBase技术介绍
http://www.searchtb.com/2011/01/understanding-hbase.html
3、HBase入门篇2-Java操作HBase例子
http://www.javabloger.com/article/apache-hbase-shell-and-java-api-html.html
4、hbase基本概念和hbase shell常用命令用法
http://www.cnblogs.com/flying5/archive/2011/09/15/2178064.html
5、 HBase简介
http://blog.csdn.net/leeqing2011/article/details/7608261
6、HBase 官方文档(中文版)
http://www.yankay.com/wp-content/hbase/book.html
7、HBase性能优化方法总结
http://blog.linezing.com/2012/03/hbase-performance-optimization
8、hbase系统架构及数据结构
http://blog.csdn.net/a221133/article/details/6894717
9、[翻译] HBase存储架构
http://www.spnguru.com/2010/07/%E7%BF%BB%E8%AF%91-hbase%E5%AD%98%E5%82%A8%E6%9E%B6%E6%9E%84/
10、HBase存储文件格式概述
http://forchenyun.iteye.com/blog/828549
11、Hbase, Hive and Pig 介绍(肯特大学)
http://www.cs.kent.edu/~jin/Cloud12Spring/HbaseHivePig.pptx
12、python 调用HBase 实例
http://hbase.iteye.com/blog/1178063
13、hbase在淘宝的应用和优化小结
http://walkoven.com/hbase%20optimization%20and%20apply%20summary%20in%20taobao.pdf
14、hbase伪分布式安装指南:
http://my.oschina.net/leejun2005/blog/91952
15、HBase上关于CMS、GC碎片、大缓存的一种解决方案:Bucket Cache
http://zjushch.iteye.com/blog/1751387
注:作者来自阿里,据称读性能能提升一个数量级,该patch已被hbase社区接受。
16、HBase 一些 tip
http://www.blogjava.net/changedi/archive/2012/12/28/393577.html
(八)flume
1、Flume日志收集 原理与实践
http://www.cnblogs.com/oubo/archive/2012/05/25/2517751.html
2、flume搭建调试
http://log.medcl.net/item/2012/03/flume-build-process/
(九)sqoop
1、sqoop的安装、配置及使用简介
http://blog.csdn.net/leeqing2011/article/details/7630690?utm_source=weibolife
2、Sqoop示例
http://baiyunl.iteye.com/blog/964254
3、使用Sqoop在HDFS和RDBMS之间导数据
http://www.linuxidc.com/Linux/2011-10/45080.htm
4、Sqoop User Guide (v1.4.2)
http://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html?utm_source=weibolife#_introduction
5、用sqoop进行mysql和hdfs系统间的数据互导
http://abloz.com/2012/07/19/data-between-the-mysql-and-hdfs-system-of-mutual-conductance-using-sqoop.html
6、Mysql<->sqoop<->HDFS 数据交换实验
http://leonarding.blog.51cto.com/6045525/1092764
(十)ZooKeeper
1、ZooKeeper Administrator‘s Guide
http://zookeeper.apache.org/doc/r3.4.3/zookeeperAdmin.html
2、ZooKeeper快速搭建
http://nileader.blog.51cto.com/1381108/795230
3、ZooKeeper管理员指南——部署与管理ZooKeeper
http://blogread.cn/it/article/5917?f=sinat
(十一)NOSQL
1、Redis资料汇总专题
http://blog.nosqlfan.com/html/3537.html
2、MongoDB资料汇总专题
http://blog.nosqlfan.com/html/3548.html
3、NoSQL数据库笔谈
http://sebug.net/paper/databases/nosql/Nosql.html
4、redis入门系列
http://www.cnblogs.com/xhan/archive/2011/02/08/1949867.html
5、Redis经验谈
http://www.programmer.com.cn/14577/
附:我的百度空间(由于百度的升级门,导致许多博文丢失):
1、http://203.208.46.148/#q=site:baidu.com+hadoop+leejun_2005&hl=zh-CN&newwindow=1&prmd=imvns&ei=J1dwUKyBOcmsiAff9IHwAw&start=10&sa=N&bav=on.2,or.r_gc.r_pw.&fp=2ba1f2c2b0790967&biw=1366&bih=643
ZZ:http://my.oschina.net/leejun2005/blog/81771