首页 > 代码库 > 一个用于白名单服务的布隆过滤器(bloom filter)
一个用于白名单服务的布隆过滤器(bloom filter)
bloom filter这种数据结构用于判断一个元素是否在集合内,当然,这种功能也可以由HashMap来实现。bloom filter与HashMap的区别在于,HashMap会储存代表这个元素的key自身(如key为"IKnow7",那么HashMap将存储"IKnow7"这12个字节(java),其实还需要包括引用大小,但java中相同string只存一份),而bloom filter在底层只会使用几个bit来代表这个元素。在速度上,bloom filter对比与HashMap相差不大,底层同样是hash+随机访问。由于bloom filter对空间节省的特性,bloom filter适合判断一个元素是否在海量数据集合中。
bloom filter的一些概念
bloom filter并非十全十美。bloom filter在添加元素时,会将对象hash到底层位图数组的k个位上,对这些位,bloom filter会将其值设为1。由于hash函数特性以及位图数组长度有限,不同的对象可能在某些位上有重叠。bloom filter在检查元素是否存在时,会检查该对象所对应的k个位是否为1,如果全部都为1表示存在,这里就出现问题了,这些位上的1未必是该元素之前设置的,有可能是别的元素所设置的,所以会造成一些误判,即原本不在bloom filter中的一些元素也被判别在bloom filter中。bloom filter的这种误判被称为"积极的误判",即存在的元素的一定会通过,不存在的元素也有可能通过,而不会造成对存在的元素结果为否的判定。
可以简单猜测,误判的概率与hash的选择、位图数组的大小、当前元素的数量以及K(映射位的个数)有关。一般来说,hash值越平均、位图数组越大、元素数量越少那么误判的概率就越低。
这是一个大牛写的关于bloom filter设计与误判率的理论分析,大伙可以去看看:http://www.cnblogs.com/allensun/archive/2011/02/16/1956532.html。
bloom filter在web上的应用
在web应用中我们经常需要使用白名单来过滤一些请求,用以避免一些无效的数据库访问或者恶意攻击。对于允许一些误判率且存在海量数据的白名单来说,使用bloom filter是不二的选择。
使用bloom filter实现一个支持增量请求的白名单
白名单通常是需要更新的,更新的方式一般有全量和增量更新。全量不必说,重新定义个bloom filter将当前所有数据放入其中即可。增量更新的话,一般会提供一段时间内新增和删除的数据,所以需要在白名单中将数据进行合并,该添加的添加,该删除的删除。
可是...... 原生的bloom filter并不支持元素的删除操作,因为某一位可能为多个元素所用。一种不切实际的想法是为bloom filter的每一位设置一个引用计数,每删除一个元素减1。
一种可行的做法是,另外使用一个map来保存已删除的元素,在判断元素是否存在时先判断在该deletemap中是否存在,如果存在,直接false。如果不存在,再通过bloom filter进行判断。在新添加元素时,如果deletemap中存在,删除该deletemap中的该元素,再添加到bloom filter中。在实际应用中,使用白名单的场景需要删除的元素一般是较少的,所以这种方式从效率是可行的。这种方式存在一个问题,当deletemap中元素过多时,势必会造成bloom filter的误判率上升,因为某些原本被删除元素设置为1的位并没有被归0。该问题的解决措施是,当deletemap的容量到达的一个界线时,使用全量同步更新该bloom filter。
白名单bloom filter的实现
这类构件复用性很强,可以轻松的集成到现有的代码之上。下面直接贴出来:
public class BloomFilter<E> implements Serializable { private static final long serialVersionUID = 3507830443935243576L; private long timestamp;//用于时间戳更新机制 private HashMap<E, Boolean> deleteMap ; //储存已删除元素 private BitSet bitset;//位图存储 private int bitSetSize; // expected (maximum) number of elements to be added private int expectedNumberOfFilterElements; // number of elements actually added to the Bloom filter private int numberOfAddedElements; private int k; //每一个元素对应k个位 // encoding used for storing hash values as strings static Charset charset = Charset.forName("UTF-8"); // MD5 gives good enough accuracy in most circumstances. // Change to SHA1 if it's needed static String hashName = "MD5"; static final MessageDigest digestFunction; static { // The digest method is reused between instances to provide higher entropy. MessageDigest tmp; try { tmp = java.security.MessageDigest.getInstance(hashName); } catch (NoSuchAlgorithmException e) { tmp = null; } digestFunction = tmp; } /** * Constructs an empty Bloom filter. * * @param bitSetSize defines how many bits should be used for the filter. * @param expectedNumberOfFilterElements defines the maximum * number of elements the filter is expected to contain. */ public BloomFilter(int bitSetSize, int expectedNumberOfFilterElements) { this.expectedNumberOfFilterElements = expectedNumberOfFilterElements; this.k = (int) Math.round( (bitSetSize / expectedNumberOfFilterElements) * Math.log(2.0)); bitset = new BitSet(bitSetSize); deleteMap = new HashMap<E, Boolean>(); this.bitSetSize = bitSetSize; numberOfAddedElements = 0; } /** * Generates a digest based on the contents of a String. * * @param val specifies the input data. * @param charset specifies the encoding of the input data. * @return digest as long. */ public static long createHash(String val, Charset charset) { try { return createHash(val.getBytes(charset.name())); } catch (UnsupportedEncodingException e) { e.printStackTrace(); // Ingore } return -1; } /** * Generates a digest based on the contents of a String. * * @param val specifies the input data. The encoding is expected to be UTF-8. * @return digest as long. */ public static long createHash(String val) { return createHash(val, charset); } /** * Generates a digest based on the contents of an array of bytes. * * @param data specifies input data. * @return digest as long. */ public static long createHash(byte[] data) { long h = 0; byte[] res; synchronized (digestFunction) { res = digestFunction.digest(data); } for (int i = 0; i < 4; i++) { h <<= 8; h |= ((int) res[i]) & 0xFF; } return h; } /** * Compares the contents of two instances to see if they are equal. * * @param obj is the object to compare to. * @return True if the contents of the objects are equal. */ @SuppressWarnings("unchecked") @Override public boolean equals(Object obj) { if (obj == null) { return false; } if (getClass() != obj.getClass()) { return false; } final BloomFilter<E> other = (BloomFilter<E>) obj; if (this.expectedNumberOfFilterElements != other.expectedNumberOfFilterElements) { return false; } if (this.k != other.k) { return false; } if (this.bitSetSize != other.bitSetSize) { return false; } if (this.bitset != other.bitset && (this.bitset == null || !this.bitset.equals(other.bitset))) { return false; } return true; } /** * Calculates a hash code for this class. * @return hash code representing the contents of an instance of this class. */ @Override public int hashCode() { int hash = 7; hash = 61 * hash + (this.bitset != null ? this.bitset.hashCode() : 0); hash = 61 * hash + this.expectedNumberOfFilterElements; hash = 61 * hash + this.bitSetSize; hash = 61 * hash + this.k; return hash; } /** * Calculates the expected probability of false positives based on * the number of expected filter elements and the size of the Bloom filter. * <br /><br /> * The value returned by this method is the <i>expected</i> rate of false * positives, assuming the number of inserted elements equals the number of * expected elements. If the number of elements in the Bloom filter is less * than the expected value, the true probability of false positives will be lower. * * @return expected probability of false positives. */ public double expectedFalsePositiveProbability() { return getFalsePositiveProbability(expectedNumberOfFilterElements); } /** * Calculate the probability of a false positive given the specified * number of inserted elements. * * @param numberOfElements number of inserted elements. * @return probability of a false positive. */ public double getFalsePositiveProbability(double numberOfElements) { // (1 - e^(-k * n / m)) ^ k return Math.pow((1 - Math.exp(-k * (double) numberOfElements / (double) bitSetSize)), k); } /** * Get the current probability of a false positive. The probability is calculated from * the size of the Bloom filter and the current number of elements added to it. * * @return probability of false positives. */ public double getFalsePositiveProbability() { return getFalsePositiveProbability(numberOfAddedElements); } /** * Returns the value chosen for K.<br /> * <br /> * K is the optimal number of hash functions based on the size * of the Bloom filter and the expected number of inserted elements. * * @return optimal k. */ public int getK() { return k; } /** * Sets all bits to false in the Bloom filter. */ public void clear() { bitset.clear(); numberOfAddedElements = 0; } /** * Adds an object to the Bloom filter. The output from the object's * toString() method is used as input to the hash functions. * * @param element is an element to register in the Bloom filter. */ public void add(E element) { deleteMap.remove(element); long hash; String valString = element.toString(); for (int x = 0; x < k; x++) { hash = createHash(valString + Integer.toString(x)); hash = hash % (long)bitSetSize; bitset.set(Math.abs((int)hash), true); } numberOfAddedElements ++; } /** * Remove all elements from a Collection to the Bloom filter. * @param c Collection of elements. */ public void removeAll(Collection<? extends E> c) { for (E element : c) remove(element); } public void remove(E element) { deleteMap.put(element, Boolean.TRUE); } public int getDeleteMapSize(){ return deleteMap.size(); } /** * Adds all elements from a Collection to the Bloom filter. * @param c Collection of elements. */ public void addAll(Collection<? extends E> c) { for (E element : c) { if (element != null) add(element); } } /** * Returns true if the element could have been inserted into the Bloom filter. * Use getFalsePositiveProbability() to calculate the probability of this * being correct. * * @param element element to check. * @return true if the element could have been inserted into the Bloom filter. */ public boolean contains(E element) { Boolean contains = deleteMap.get(element); if (contains != null && contains) return false; long hash; String valString = element.toString(); for (int x = 0; x < k; x++) { hash = createHash(valString + Integer.toString(x)); hash = hash % (long) bitSetSize; if (!bitset.get(Math.abs((int) hash))) return false; } return true; } /** * Returns true if all the elements of a Collection could have been inserted * into the Bloom filter. Use getFalsePositiveProbability() to calculate the * probability of this being correct. * @param c elements to check. * @return true if all the elements in c could have been inserted into the Bloom filter. */ public boolean containsAll(Collection<? extends E> c) { for (E element : c) if (!contains(element)) return false; return true; } /** * Read a single bit from the Bloom filter. * @param bit the bit to read. * @return true if the bit is set, false if it is not. */ public boolean getBit(int bit) { return bitset.get(bit); } /** * Set a single bit in the Bloom filter. * @param bit is the bit to set. * @param value If true, the bit is set. If false, the bit is cleared. */ public void setBit(int bit, boolean value) { bitset.set(bit, value); } /** * Return the bit set used to store the Bloom filter. * @return bit set representing the Bloom filter. */ public BitSet getBitSet() { return bitset; } /** * Returns the number of bits in the Bloom filter. Use count() to retrieve * the number of inserted elements. * * @return the size of the bitset used by the Bloom filter. */ public int size() { return this.bitSetSize; } /** * Returns the number of elements added to the Bloom filter after it * was constructed or after clear() was called. * * @return number of elements added to the Bloom filter. */ public int count() { return this.numberOfAddedElements; } /** * Returns the expected number of elements to be inserted into the filter. * This value is the same value as the one passed to the constructor. * * @return expected number of elements. */ public int getExpectedNumberOfElements() { return expectedNumberOfFilterElements; } /** * 返回更新的时间戳机制 * @return */ public long getTimestamp() { return timestamp; } /** * 设置跟新的时间戳 * @param timestamp */ public void setTimestamp(long timestamp) { this.timestamp = timestamp; } @Override public String toString() { return "BloomFilter [timestamp=" + timestamp + ", bitSetSize=" + bitSetSize + ", expectedNumberOfFilterElements=" + expectedNumberOfFilterElements + ", numberOfAddedElements=" + numberOfAddedElements + ", k=" + k +",deleteMapSize=" +getDeleteMapSize()+"]"; } }
一个用于白名单服务的布隆过滤器(bloom filter)
声明:以上内容来自用户投稿及互联网公开渠道收集整理发布,本网站不拥有所有权,未作人工编辑处理,也不承担相关法律责任,若内容有误或涉及侵权可进行投诉: 投诉/举报 工作人员会在5个工作日内联系你,一经查实,本站将立刻删除涉嫌侵权内容。