首页 > 代码库 > larbin之哈希之谈
larbin之哈希之谈
由于工作原因,打算对larbin的源码进行分析一番
用的事2.6.3版本的larbin源码,由于这是业余,会断断续续的分析上传,已做记录笔记
今天我们分析一下larbin的哈希表
这个哈希表结构比较简单,因为它的主要用处是排重,因此只给出了用于排重的简单函数,
我们来看一下头文件怎么定义的:
// Larbin// Sebastien Ailleret// 23-11-99 -> 14-01-00/* class hashTable * This class is in charge of making sure we don‘t crawl twice the same url */#ifndef HASHTABLE_H#define HASHTABLE_H#include "types.h"#include "utils/url.h"class hashTable { private: ssize_t size; char *table; public: /* constructor */ hashTable (bool create); /* destructor */ ~hashTable (); /* save the hashTable in a file */ void save(); /* test if this url is allready in the hashtable * return true if it has been added * return false if it has allready been seen */ bool test (url *U); /* set a url as present in the hashtable */ void set (url *U); /* add a new url in the hashtable * return true if it has been added * return false if it has allready been seen */ bool testSet (url *U);};#endif // HASHTABLE_H
由头文件我们可以看出,这个哈希表仅仅有四个成员函数(除了构造和析构)
save 函数是用于保存哈希表内部的数据,用于防止程序异常退出而造成数据丢失,因此把哈希内数据保存到一个文件中
test 函数用于测试参数指定的URL是否在哈希表内存在,只要是排重
set 函数就是判断出需要设置哈希表内值得时候设置该位置的URL对应的值,表示该URL从此开始存在于哈希表中
testset 是一个辅助函数,先判断,然后设置该位置的值,并且返回设置前的判断结果
下面我们就仔细来看一看各个函数的实现,比较简单,我就在程序中做了简单注释,就不再多余的文字解释了
构造函数:
hashTable::hashTable (bool create) { //构造函数,申请哈希需求的空间并初始化 ssize_t total = hashSize/8; //因为是位集合判断,所以每个字节8位,对于哈希的总成都除以8 table = new char[total]; //申请哈希空间,其实这个地方主要是以数组巧妙勾勒哈希功能 if (create) { //是一个标志,也就是说哈希内部的数据是从文件内读取还是初始化位0 for (ssize_t i=0; i<hashSize/8; i++) { table[i] = 0; } } else { int fds = open("hashtable.bak", O_RDONLY); //从bak备份文件读取数据 if (fds < 0) { cerr << "Cannot find hashtable.bak, restart from scratch\n"; for (ssize_t i=0; i<hashSize/8; i++) { //如果打开备份文件失败,就重新赋值位0,当做第一次看待 table[i] = 0; } } else { ssize_t sr = 0; while (sr < total) { ssize_t tmp = read(fds, table+sr, total-sr); //然后循环读取文件,直到成功读取所有数据 if (tmp <= 0) { cerr << "Cannot read hashtable.bak : " << strerror(errno) << endl; exit(1); } else { sr += tmp; //增加每次读取的数据 } } close(fds); //关闭文件描述符 } }}
析构函数:
hashTable::~hashTable () { //析构函数,释放哈希申请的空间 delete [] table;}
测试函数test:
bool hashTable::test (url *U) { //判断该url对应的是否存在哈希中,如果存在返回true,否则false unsigned int code = U->hashCode(); //根据hashCode函数求散列值 unsigned int pos = code / 8; unsigned int bits = 1 << (code % 8); return table[pos] & bits; }
设置函数:
void hashTable::set (url *U) { //设置url对应哈希值 unsigned int code = U->hashCode(); unsigned int pos = code / 8; unsigned int bits = 1 << (code % 8); table[pos] |= bits;}
测试设置函数:
bool hashTable::testSet (url *U) { //返回测试结果,并且设置url对应的值 unsigned int code = U->hashCode(); unsigned int pos = code / 8; unsigned int bits = 1 << (code % 8); int res = table[pos] & bits; table[pos] |= bits; return !res;}
保存文件函数:
void hashTable::save() { //把哈希内部数据存到文件,该过程是循序渐进的,防止时间间隔过程造成数据大量失真 rename("hashtable.bak", "hashtable.old"); //先把先前备份文件保存,直到最后本次成功备份后删除 int fds = creat("hashtable.bak", 00600); if (fds >= 0) { ecrireBuff(fds, table, hashSize/8); //辅助函数,把哈希数据存到备份文件 close(fds); } unlink("hashtable.old");}
该哈希的处理部分就这么多,下面重点来看看我们两个知识点
1,散列函数 hashCode:
uint url::hashCode () { unsigned int h=port; unsigned int i=0; while (host[i] != 0) { h = 31*h + host[i]; i++; } i=0; while (file[i] != 0) { h = 31*h + file[i]; i++; } return h % hashSize;}
说起来散列函数,要求很有艺术的,而且散列函数也不可能有百分百的通用性。
一般都要自己根据哈希设置自己的散列函数。最起码要设置某些数值,用同一个散列方法和框架
该散列函数比较简单,就是把host和file(URL类中的两个字段,表示主机和文件路径)依次乘以31
然后对哈希最大值求余数,最大值这样定义的:
#define hashSize 64000000
另外对于host和file的先后哈希顺序也是设计的,先host而后file是为了让同一host对应的file的差异更大,减缓相似冲突
2,下面我们就来谈谈URL这个类,上面那个哈希散列函数就是这个类中的一个成员函数,之所以单独摘出去说,是因为散列函数也是重大的一块
我们先来看一下URL类的定义:
class url { private: char *host; char *file; uint16_t port; // the order of variables is important for physical size int8_t depth; /* parse the url */ void parse (char *s); /** parse a file with base */ void parseWithBase (char *u, url *base); /* normalize file name */ bool normalize (char *file); /* Does this url starts with a protocol name */ bool isProtocol (char *s); /* constructor used by giveBase */ url (char *host, uint port, char *file); public: /* Constructor : Parses an url (u is deleted) */ url (char *u, int8_t depth, url *base); /* constructor used by input */ url (char *line, int8_t depth); /* Constructor : read the url from a file (cf serialize) */ url (char *line); /* Destructor */ ~url (); /* inet addr (once calculated) */ struct in_addr addr; /* Is it a valid url ? */ bool isValid (); /* print an URL */ void print (); /* return the host */ inline char *getHost () { return host; } /* return the port */ inline uint getPort () { return port; } /* return the file */ inline char *getFile () { return file; } /** Depth in the Site */ inline int8_t getDepth () { return depth; } /* Set depth to max if we are at an entry point in the site * try to find the ip addr * answer false if forbidden by robots.txt, true otherwise */ bool initOK (url *from); /** return the base of the url * give means that you have to delete the string yourself */ url *giveBase (); /** return a char * representation of the url * give means that you have to delete the string yourself */ char *giveUrl (); /** write the url in a buffer * buf must be at least of size maxUrlSize * returns the size of what has been written (not including ‘\0‘) */ int writeUrl (char *buf); /* serialize the url for the Persistent Fifo */ char *serialize (); /* very thread unsafe serialisation in a static buffer */ char *getUrl(); /* return a hashcode for the host of this url */ uint hostHashCode (); /* return a hashcode for this url */ uint hashCode ();#ifdef URL_TAGS /* tag associated to this url */ uint tag;#endif // URL_TAGS#ifdef COOKIES /* cookies associated with this page */ char *cookie; void addCookie(char *header);#else // COOKIES inline void addCookie(char *header) {}#endif // COOKIES};
稍后待续。。。