首页 > 代码库 > glog另启动线程写文本日志
glog另启动线程写文本日志
glog本身是很高效的,google的大牛肯定知道大规模的写日志用glog的话肯定会影响业务线程的处理,带负荷的磁盘IO谁都桑不起。比方levelDB就是默认异步写,更不用说google的三驾马车都是分布式的。之前看过其论文,简直是引领时代。
在glog的issue里有人提出了异步写的问题,可是语焉不详,只是0.33版本号已经有了接口,可是还不友好,可是全然能够实现磁盘日志的异步写。
今天算是花了点时间踩了点坑,算是基本能够搞了。稳定之后会把这个版本号和glog,g2log,mudo logging一起測试下。mudo对buffer做了些trick,内部有两个bufferptr,做了双缓冲,据说效率非常高,只是仅仅有linux平台的,只是但把它的log抽离出来也不难,陈老师封装了mutex,thread,conditional等,在gcc4.8,clang3.3,VS2010都不是问题,已经没多大必要,并且之前为之乐道的linux下的threadsafe的initonce,如今C++11x也有了支持。
glog中能够让client定制接口是:
class GOOGLE_GLOG_DLL_DECL Logger { public: virtual ~Logger(); // Writes "message[0,message_len-1]" corresponding to an event that // occurred at "timestamp". If "force_flush" is true, the log file // is flushed immediately. // // The input message has already been formatted as deemed // appropriate by the higher level logging facility. For example, // textual log messages already contain timestamps, and the // file:linenumber header. virtual void Write(bool force_flush, time_t timestamp, const char* message, int message_len) = 0; // Flush any buffered messages virtual void Flush() = 0; // Get the current LOG file size. // The returned value is approximate since some // logged data may not have been flushed to disk yet. virtual uint32 LogSize() = 0; virtual void SetBasename(const char* basename) = 0; virtual void SetExtension(const char* ext) = 0 ; virtual void SetSymlinkBasename(const char* symlink_basename) = 0; };
我在里面另外加了几个接口,为了之后的方便。
用Active object模式非常好解决,就是我们通常所说的生产者消费者,在logmsg析构时就会fflush到磁盘,这次就会调用logger的write方法,此时就是我们接手的机会,把数据封装下,投递到业务线程,然后取出,实际写磁盘就好。
封装了简单的Active模式,Activer里封装了LogData用来封装打印实体,Buffer用来线程间传递数据,另外要显式设置Active的回调函数callBack.线程间传递数据用了C++11里的currentQueue,就不须要自己造轮子了:
/** ========================================================================== * 2010 by KjellKod.cc. This is PUBLIC DOMAIN to use at your own risk and comes * with no warranties. This code is yours to share, use and modify with no * strings attached and no restrictions or obligations. * ============================================================================ * * Example of a Active Object, using C++11 std::thread mechanisms to make it * safe for thread communication. * * This was originally published at http://sites.google.com/site/kjellhedstrom2/active-object-with-cpp0x * and inspired from Herb Sutter‘s C++11 Active Object * http://herbsutter.com/2010/07/12/effective-concurrency-prefer-using-active-objects-instead-of-naked-threads * * The code below uses JustSoftware Solutions Inc std::thread implementation * http://www.justsoftwaresolutions.co.uk * * Last update 2012-10-10, by Kjell Hedstrom, * e-mail: hedstrom at kjellkod dot cc * linkedin: http://linkedin.com/se/kjellkod */ #ifndef ACTIVE_H_ #define ACTIVE_H_ #include <thread> #include <functional> #include <condition_variable> #include <mutex> #include <memory> #include <concurrent_queue.h> #include "shared_queue.h" struct Buffer { Buffer():m_Len(0), m_pMsg(NULL){} ~Buffer() { if (NULL != m_pMsg) delete []m_pMsg; } Buffer(int size):m_Len(size) , m_pMsg(new char[m_Len]) { } int m_Len; char* m_pMsg; }; typedef std::function<void(Buffer*)> Callback; class Active { private: Active(const Active&); // c++11 feature not yet in vs2010 = delete; Active& operator=(const Active&); // c++11 feature not yet in vs2010 = delete; Active(); // Construction ONLY through factory createActive(); void doDone(){done_ = true;} void run(); void setCallBack(Callback aCallBack); Concurrency::concurrent_queue<Buffer*> mq_; std::thread thd_; bool done_; // finished flag to be set through msg queue by ~Active Callback callBack_; public: virtual ~Active(); void send(Buffer* apBuffer); static std::unique_ptr<Active> createActive(Callback aCallBack); // Factory: safe construction & thread start }; #endif /** ========================================================================== * 2010 by KjellKod.cc. This is PUBLIC DOMAIN to use at your own risk and comes * with no warranties. This code is yours to share, use and modify with no * strings attached and no restrictions or obligations. * ============================================================================ * * Example of a Active Object, using C++11 std::thread mechanisms to make it * safe for thread communication. * * This was originally published at http://sites.google.com/site/kjellhedstrom2/active-object-with-cpp0x * and inspired from Herb Sutter‘s C++11 Active Object * http://herbsutter.com/2010/07/12/effective-concurrency-prefer-using-active-objects-instead-of-naked-threads * * The code below uses JustSoftware Solutions Inc std::thread implementation * http://www.justsoftwaresolutions.co.uk * * Last update 2012-10-10, by Kjell Hedstrom, * e-mail: hedstrom at kjellkod dot cc * linkedin: http://linkedin.com/se/kjellkod */ #include "active.h" #include <cassert> Active::Active(): done_(false){} Active::~Active() { Callback quit_token = std::bind(&Active::doDone, this); thd_.join(); } // Add asynchronously a work-message to queue void Active::send( Buffer* apBuffer ) { if (NULL != apBuffer) { mq_.push(apBuffer); } } void Active::run() { while (!done_) { if (!mq_.empty()) { Buffer* pBuffer = NULL; mq_.try_pop(pBuffer); if (NULL != pBuffer) { callBack_(pBuffer); delete pBuffer; } } } } // Factory: safe construction of object before thread start std::unique_ptr<Active> Active::createActive(Callback aCallBack){ std::unique_ptr<Active> aPtr(new Active()); aPtr->thd_ = std::thread(&Active::run, aPtr.get()); aPtr->callBack_ = aCallBack; return aPtr; } void Active::setCallBack( Callback aCallBack ) { callBack_ = aCallBack; }
重点是在threadlogger里,实现了Logger的接口。Write函数实现真正的写逻辑,几个set函数会在内部被调用。
#pragma once #include <glog/logging.h> #include <mutex> #include "active.h" using namespace std; namespace google { class ThreadLog : public google::base::Logger { public: ThreadLog(); ~ThreadLog(); virtual void Write(bool force_flush, time_t timestamp, const char* message, int message_len) ; virtual void Flush(); virtual uint32 LogSize(); // Configuration options void SetBasename(const char* basename); void SetExtension(const char* ext); void SetSymlinkBasename(const char* symlink_basename); void CallBack(Buffer* pBuffer); private: static const uint32 kRolloverAttemptFrequency = 0x20; mutex lock_; bool base_filename_selected_; string base_filename_; string symlink_basename_; string filename_extension_; // option users can specify (eg to add port#) FILE* file_; LogSeverity severity_; uint32 bytes_since_flush_; uint32 file_length_; unsigned int rollover_attempt_; int64 next_flush_time_; // cycle count at which to flush log string hostname; bool stopWriting; std::unique_ptr<Active> m_pActive; bool CreateLogfile(const string& time_pid_string); void FlushUnlocked(); void WriteInteral(bool force_flush, time_t timestamp, const char* message, int message_len); }; } #include "ThreadLog.h" #include "port.h" #include <fcntl.h> #include <iomanip> #include "utilities.h" #include <functional> namespace google { static int GetSize(bool& force_flush, time_t& timestamp, const char* message, int& message_len) { return sizeof(force_flush)+sizeof(timestamp)+sizeof(message_len)+message_len; } void ThreadLog::Write( bool force_flush, time_t timestamp, const char* message, int message_len ) { Buffer* pBuffer = new Buffer(GetSize(force_flush, timestamp, message, message_len)); char* curData = http://www.mamicode.com/pBuffer->m_pMsg;>
这样搞定之后,main函数能够这样使用,就能够把自己的ThreadLog类内嵌到glog里。#define GLOG_NO_ABBREVIATED_SEVERITIES #include <windows.h> #include <glog/logging.h> #include "ThreadLog.h" using namespace google; int main(int argc, char* argv[]) { google::InitGoogleLogging("test/testsss"); google::base::Logger* mylogger = new google::ThreadLog; SetLogger(google::GLOG_INFO, mylogger); google::SetLogDestination(google::GLOG_INFO, "../Debug/logtestInfo"); //google::SetLogDestination(google::GLOG_ERROR, "../Debug/logtestDebug"); int num_cookies = 0; google::SetStderrLogging(google::GLOG_INFO); //google::SetStderrLogging(google::GLOG_ERROR); //google::LogToStderr(); for (int i = 0; i < 1000; ++i){ LOG(INFO) << "how are " << i << " cookies"; } google::ShutdownGoogleLogging(); }
当然直接用这源代码是无法编译成功的,我改动了glog内部的源代码。整个项目地址:git@github.com:boyxiaolong/Proejcts.git
測试还有点问题,偶尔会有乱码,并且须要优化的是那个Buffer的动态申请。
只是都是后话了。