首页 > 代码库 > Tair LDB基于Prefixkey的范围查找性能优化项目之如何提取key的prefix_size信息

Tair LDB基于Prefixkey的范围查找性能优化项目之如何提取key的prefix_size信息

New Document<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /><style type="text/css">/* GitHub stylesheet for MarkdownPad (http://markdownpad.com) */ /* Author: Nicolas Hery - http://nicolashery.com */ /* Version: b13fe65ca28d2e568c6ed5d7f06581183df8f2ff */ /* Source: https://github.com/nicolahery/markdownpad-github */ /* RESET =============================================================================*/ html, body, div, span, applet, object, iframe, h1, h2, h3, h4, h5, h6, p, blockquote, pre, a, abbr, acronym, address, big, cite, code, del, dfn, em, img, ins, kbd, q, s, samp, small, strike, strong, sub, sup, tt, var, b, u, i, center, dl, dt, dd, ol, ul, li, fieldset, form, label, legend, table, caption, tbody, tfoot, thead, tr, th, td, article, aside, canvas, details, embed, figure, figcaption, footer, header, hgroup, menu, nav, output, ruby, section, summary, time, mark, audio, video { margin: 0; padding: 0; border: 0; } /* BODY =============================================================================*/ body { font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 14px; line-height: 1.6; color: #333; background-color: #fff; padding: 20px; max-width: 960px; margin: 0 auto; } body>*:first-child { margin-top: 0 !important; } body>*:last-child { margin-bottom: 0 !important; } /* BLOCKS =============================================================================*/ p, blockquote, ul, ol, dl, table, pre { margin: 15px 0; } /* HEADERS =============================================================================*/ h1, h2, h3, h4, h5, h6 { margin: 20px 0 10px; padding: 0; font-weight: bold; -webkit-font-smoothing: antialiased; } h1 tt, h1 code, h2 tt, h2 code, h3 tt, h3 code, h4 tt, h4 code, h5 tt, h5 code, h6 tt, h6 code { font-size: inherit; } h1 { font-size: 28px; color: #000; } h2 { font-size: 24px; border-bottom: 1px solid #ccc; color: #000; } h3 { font-size: 18px; } h4 { font-size: 16px; } h5 { font-size: 14px; } h6 { color: #777; font-size: 14px; } body>h2:first-child, body>h1:first-child, body>h1:first-child+h2, body>h3:first-child, body>h4:first-child, body>h5:first-child, body>h6:first-child { margin-top: 0; padding-top: 0; } a:first-child h1, a:first-child h2, a:first-child h3, a:first-child h4, a:first-child h5, a:first-child h6 { margin-top: 0; padding-top: 0; } h1+p, h2+p, h3+p, h4+p, h5+p, h6+p { margin-top: 10px; } /* LINKS =============================================================================*/ a { color: #4183C4; text-decoration: none; } a:hover { text-decoration: underline; } /* LISTS =============================================================================*/ ul, ol { padding-left: 30px; } ul li > :first-child, ol li > :first-child, ul li ul:first-of-type, ol li ol:first-of-type, ul li ol:first-of-type, ol li ul:first-of-type { margin-top: 0px; } ul ul, ul ol, ol ol, ol ul { margin-bottom: 0; } dl { padding: 0; } dl dt { font-size: 14px; font-weight: bold; font-style: italic; padding: 0; margin: 15px 0 5px; } dl dt:first-child { padding: 0; } dl dt>:first-child { margin-top: 0px; } dl dt>:last-child { margin-bottom: 0px; } dl dd { margin: 0 0 15px; padding: 0 15px; } dl dd>:first-child { margin-top: 0px; } dl dd>:last-child { margin-bottom: 0px; } /* CODE =============================================================================*/ pre, code, tt { font-size: 12px; font-family: Consolas, "Liberation Mono", Courier, monospace; } code, tt { margin: 0 0px; padding: 0px 0px; white-space: nowrap; border: 1px solid #eaeaea; background-color: #f8f8f8; border-radius: 3px; } pre>code { margin: 0; padding: 0; white-space: pre; border: none; background: transparent; } pre { background-color: #f8f8f8; border: 1px solid #ccc; font-size: 13px; line-height: 19px; overflow: auto; padding: 6px 10px; border-radius: 3px; } pre code, pre tt { background-color: transparent; border: none; } kbd { -moz-border-bottom-colors: none; -moz-border-left-colors: none; -moz-border-right-colors: none; -moz-border-top-colors: none; background-color: #DDDDDD; background-image: linear-gradient(#F1F1F1, #DDDDDD); background-repeat: repeat-x; border-color: #DDDDDD #CCCCCC #CCCCCC #DDDDDD; border-image: none; border-radius: 2px 2px 2px 2px; border-style: solid; border-width: 1px; font-family: "Helvetica Neue",Helvetica,Arial,sans-serif; line-height: 10px; padding: 1px 4px; } /* QUOTES =============================================================================*/ blockquote { border-left: 4px solid #DDD; padding: 0 15px; color: #777; } blockquote>:first-child { margin-top: 0px; } blockquote>:last-child { margin-bottom: 0px; } /* HORIZONTAL RULES =============================================================================*/ hr { clear: both; margin: 15px 0; height: 0px; overflow: hidden; border: none; background: transparent; border-bottom: 4px solid #ddd; padding: 0; } /* TABLES =============================================================================*/ table th { font-weight: bold; } table th, table td { border: 1px solid #ccc; padding: 6px 13px; } table tr { border-top: 1px solid #ccc; background-color: #fff; } table tr:nth-child(2n) { background-color: #f8f8f8; } /* IMAGES =============================================================================*/ img { max-width: 100% }</style><body>

目前项目已快截止,编码任务也基本完成,现在主要是性能测试。

项目是按照“Tair LDB基于Prefixkey的范围查找性能优化项目提议方案”的步骤一步步完成的,首先先介绍第一个关键问题是如何解决的。在提案中有以下描述:

由于getrange接口的数据是从prefixput/prefixincr接口进来的,那么prefix的长度信息就可以从它们的pkey参数得到,pkey的数据类型是dataentry,有属性prefixsize,那么我们在客户端将pkey和skey合并为mkey(已经设置mkey的prefixsize为pkey的size)后与value一起传送到服务器端。

在客户端与服务器端的连接过程中,将key的类型封装成LdbKey类,value的类型封装成LdbItem类,LdbItem里面含有key的prefixsize信息,然后两者都转化为Slice类型发送到leveldb底层进行存储操作。注意此时value里面包含了prefixszie信息(序列化信息,不能直接提取),因此我们在生成filter block时可以从value中提取出prefix_size信息(按LdbItem的格式进行分析提取)以生成我们所需要的prefix bloomfilter。提取的具体实现可以放在leveldb层的外面,在leveldb里面进行调用即可(分离操作)。

这里面提到一个关键信息:key的prefix_size信息在db中是存储在value中而不是在key中.

那么value的格式是什么样的呢?

首先value的内容是由LdbItem的数据得到的,知道了LdbItem里数据的存储格式也就知道了value的数据存储格式。LdbItem里data_的数据构成是由下面这个set函数完成的:

// meta_ MUST already be set correctly
void set(const char* value_data, const int32_t value_size)
{
  if (value_data != NULL && value_size > 0)
  {
    char *metap = reinterpret_cast<char *>(&meta_);
    int real_meta_size = LDB_ITEM_META_BASE_SIZE;
    LdbItemMetaBase *metabp = reinterpret_cast<LdbItemMetaBase *>(&meta_);
    free();
    if (metabp->flag_ & TAIR_ITEM_FLAG_NEWMETA)
    {
      if (META_VER_PREFIX == metabp->meta_version_)
        real_meta_size = LDB_ITEM_META_SIZE;
      else if (META_VER_BASE == metabp->meta_version_)
        real_meta_size = LDB_ITEM_META_BASE_SIZE;
    }
    data_size_ = value_size + real_meta_size;
    data_ = new char[data_size_];
    memcpy(data_, metap, real_meta_size);
    memcpy(data_ +  real_meta_size, value_data, value_size);
    alloc_ = true;
  }

可以知道data_的内容由两部分构成:

| LdbItemMeta数据 | 真实的value内容 |

或者

| LdbItemMetaBase数据 | 真实的value内容 |

两者的区别在于LdbItemMeta数据包含prefix_size信息而LdbItemMetaBase数据不包含,这通过这两个简单的数据结构组成部分就知道了。

  struct LdbItemMetaBase
  {
    LdbItemMetaBase() : meta_version_(0), flag_(0), version_(0), cdate_(0), mdate_(0), edate_(0){}
    uint8_t  meta_version_; // meta data version
    uint8_t  flag_;         // flag
    uint16_t version_;      // version
    uint32_t cdate_;        // create time
    uint32_t mdate_;        // modify time
    uint32_t edate_;        // expired time(for meta when get value. dummy with key)
  };

  struct LdbItemMeta   // change value() and set() ,if you want to add new metadata
  {
    LdbItemMeta():  prefix_size_(0) {}
    struct LdbItemMetaBase base_;
    uint16_t prefix_size_;  // prefix key size(for getRange conflict detect)
    uint16_t reserved;  //
  };

因此如果value的第一部分是LdbItemMeta数据,就说明它包含prefixsize信息,我们就可以将value内容按LdbItemMeta的格式进行解析,从而提取出其中的prefixsize信息。

下面是具体的解析提取程序:

// get prefix size from slice value content
int get_prefix_size(const leveldb::Slice &value) {
    // parse LdbItemMetaBase from value
    char *val = const_cast<char*>(value.data());
    LdbItemMeta *metap = reinterpret_cast<LdbItemMeta*>(val);
    // check if prefix is set
    if (metap->base_.flag_ & TAIR_ITEM_FLAG_NEWMETA) {
        // if prefix is set, parse it from LdbItemMeta. if not, return 0
        if (META_VER_PREFIX == metap->base_.meta_version_) {
            return metap->prefix_size_;
        } else {
            return 0;
        }   
    }   

    return 0;
  } 

Tair LDB基于Prefixkey的范围查找性能优化项目之如何提取key的prefix_size信息