首页 > 代码库 > 记一次mogodb占用cpu高问题

记一次mogodb占用cpu高问题

 公司服务器上安装了contly,是一个开源的node.js项目,用于统计手机app使用情况,后端数据储存使用的mongodb,使用的时候经常发现mongodb占用cpu非常高,打到了210%的爆表值

top - 13:42:39 up 308 days, 23:01,  2 users,  load average: 2.84, 2.96, 2.93
Tasks: 209 total,   1 running, 208 sleeping,   0 stopped,   0 zombie
%Cpu(s): 59.4 us,  2.7 sy,  0.0 ni, 36.9 id,  0.2 wa,  0.7 hi,  0.0 si,  0.0 st
KiB Mem:   8173524 total,  6095460 used,  2078064 free,   133752 buffers
KiB Swap:  1048572 total,   750536 used,   298036 free.  3480688 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                 
16870 root      20   0 48.452g 2.643g 2.490g S 210.5 33.9  38607:41 mongod                                                                                                                                                                  
28508 root      20   0  891316 110272   5976 S   5.0  1.3  10:07.08 nodejs                                                                                                                                                                  
28589 root      20   0  893736 114044   5972 S   5.0  1.4  13:19.13 nodejs                                                                                                                                                                  
28566 root      20   0  959004 113128   5976 S   4.0  1.4  11:16.49 nodejs                                                                                                                                                                  
28670 root      20   0 1189932 383820   5980 S   4.0  4.7  26:00.97 nodejs                                                                                                                                                                  
16367 www-data  20   0   86960   2952    996 S   1.0  0.0 237:47.74 nginx                                                                                                                                                                   
16368 www-data  20   0   87028   3212    996 S   1.0  0.0 235:54.43 nginx                                                                                                                                                                   
16370 www-data  20   0   86964   2952    996 S   1.0  0.0 245:46.74 nginx                                                                                                                                                                   
16371 www-data  20   0   86960   2948    996 S   1.0  0.0 246:40.89 nginx                                                                                                                                                                   
 1154 root      20   0   23792   1764   1144 R   0.7  0.0   0:00.23 top                                                                                                                                                                     
    7 root      20   0       0      0      0 S   0.3  0.0   1053:48 rcu_sched                                                                                                                                                               
    8 root      20   0       0      0      0 S   0.3  0.0 229:02.41 rcuos/0                                                                                                                                                                 
 6559 root      20   0 3957096 217820   5668 S   0.3  2.7 230:37.95 jsvc                                                                                                                                                                    
    1 root      20   0   37348   3704    976 S   0.0  0.0   0:19.37 init                                                                                                                                                                    
    2 root      20   0       0      0      0 S   0.0  0.0   0:02.51 kthreadd                                                                                                                                                                
    3 root      20   0       0      0      0 S   0.0  0.0  68:42.78 ksoftirqd/0                                                                                                                                                             
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H                                                                                                                                                            
    9 root      20   0       0      0      0 S   0.0  0.0 225:56.62 rcuos/1                                                                                                                                                                 
   10 root      20   0       0      0      0 S   0.0  0.0 217:26.86 rcuos/2                                                                                                                                                                 
   11 root      20   0       0      0      0 S   0.0  0.0 228:57.90 rcuos/3                                                                                                                                                                 
   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/4                                                                                                                                                                 
   13 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/5                                                                                                                                                                 
   14 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/6                                                                                                                                                                 
   15 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/7                                                                                                                                                                 
   16 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/8                                                                                                                                                                 
   17 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/9                                                                                                                                                                 
   18 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/10                                                                                                                                                                
   19 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/11                                                                                                                                                                
   20 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/12                                                                                                                                                                
   21 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/13                                                                                                                                                                
   22 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/14                                                                                                                                                                
   23 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/15                                                                                                                                                                
   24 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/16                                                                                                                                                                
   25 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/17                                                                                                                                                                
   26 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/18                                                                                                                                                                
   27 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/19                                                                                                                                                                
   28 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/20                                                                                                                                                                
   29 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/21                                                                                                                                                                
   30 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/22                                                                                                                                                                
   31 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/23                                                                                                                                                                
   32 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/24                                                                                                                                                                
   33 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/25                                                                                                                                                                
   34 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/26                                                                                                                                                                
   35 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/27                                                                                                                                                                
   36 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/28                                                                                                                                                                
   37 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/29                                                                                                                                                                
   38 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/30                                                                                                                                                                
   39 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuos/31                                                                                                                                                                
   40 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh                                                                                                                                                                  
   41 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/0                                                                                                                                                                 
   42 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/1                                                                                                                                                                 
   43 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/2                                                                                                                                                                 
   44 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/3                                                                                                                                                                 
   45 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/4                                                                                                                                                                 
   46 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/5                                                                                                                                                                 
   47 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/6                                                                                                                                                                 
   48 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/7                                                                                                                                                                 
   49 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/8                                                                                                                                                                 
   50 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/9                                                                                                                                                                 
   51 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/10                                                                                                                                                                
   52 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcuob/11   

仔细查看是中断很多,命令是pid -w 1

root@localhost:~# pidstat -w 1 
Linux 3.13.0-24-generic (localhost)     07/14/2017      _x86_64_        (4 CPU)

01:44:41 PM   UID       PID   cswch/s nvcswch/s  Command
01:44:42 PM     0         7    260.40      0.00  rcu_sched
01:44:42 PM     0         8     78.22      0.00  rcuos/0
01:44:42 PM     0         9     49.50      0.00  rcuos/1
01:44:42 PM     0        10     88.12      0.00  rcuos/2
01:44:42 PM     0        11     96.04      0.00  rcuos/3
01:44:42 PM     0        74      0.99      0.00  watchdog/0
01:44:42 PM     0        75      0.99      0.00  watchdog/1
01:44:42 PM     0        77      7.92      0.00  ksoftirqd/1
01:44:42 PM     0        80      0.99      0.00  watchdog/2
01:44:42 PM     0        82     39.60      0.00  ksoftirqd/2
01:44:42 PM     0        85      0.99      0.00  watchdog/3
01:44:42 PM     0        87      9.90      0.00  ksoftirqd/3
01:44:42 PM     0      1103      7.92      0.00  kworker/u64:2
01:44:42 PM     0      1213      0.99      4.95  pidstat
01:44:42 PM     0      1265      0.99      0.00  supervisord
01:44:42 PM     0      1279      9.90      0.00  vmtoolsd
01:44:42 PM     0      1373      0.99      0.00  fail2ban-server
01:44:42 PM     0      1954     18.81      0.00  xfsaild/sdb1
01:44:42 PM     0      2132      9.90      0.00  kworker/1:1H
01:44:42 PM     0      6800      0.99      0.00  kworker/0:1
01:44:42 PM     0      6807      0.99      0.00  kworker/2:2
01:44:42 PM  1000     14269      0.99      0.00  zabbix_agentd
01:44:42 PM  1000     14273      0.99      0.00  zabbix_agentd
01:44:42 PM    33     16367    154.46      0.00  nginx
01:44:42 PM    33     16368    178.22      0.00  nginx
01:44:42 PM    33     16370    166.34      0.00  nginx
01:44:42 PM    33     16371    168.32      0.00  nginx
01:44:42 PM     0     16870    100.00      0.00  mongod
01:44:42 PM     0     28498      0.99      0.00  supervisord
01:44:42 PM     0     28508    126.73     39.60  nodejs
01:44:42 PM     0     28566    141.58     46.53  nodejs
01:44:42 PM     0     28589    137.62    110.89  nodejs
01:44:42 PM     0     28670    123.76     96.04  nodejs
01:44:42 PM     0     28716      6.93      0.00  kworker/3:1
01:44:42 PM     0     28732      0.99      0.00  kworker/1:0

vmstat

root@localhost:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 3  1 750536 2089980 133864 3482948    0    0     8   738    0    0 14  4 81  1  0
 3  0 750536 2089932 133864 3482948    0    0     0  2062 6793 9336 67  5 28  0  0
 3  0 750536 2086820 133864 3482952    0    0     0  1767 6125 8760 72  5 23  0  0
 4  0 750536 2084232 133864 3482952    0    0     0  1611 6227 8686 60  4 37  0  0
 3  0 750536 2088264 133864 3482952    0    0     0   140 5974 8556 65  4 31  0  0
 4  0 750536 2083864 133864 3482956    0    0     0  2635 9100 10589 78  5 17  0  0
 2  0 750536 2085624 133864 3482916    0    0     0   168 8152 11602 63  5 33  0  0

中断和上下文切换多,是因为nodejs一直在被从sleep中唤醒,实际上是countly要接受手机端返回来的数据,不断被IO唤醒,每次唤醒都要上下文切换,所以表象是上下文频繁切换,实质上是counly负担比较重。

不过这个貌似和mongodb没有关系,只能说明countly的负担比较重,一直在写入mongodb,所以现在mongodb的负担也比较大。其实io算不上高,cpu的iowait也不多,硬盘写入并不是瓶颈,可能是mongodb本身的问题

查看mongostat

root@localhost:~# mongostat
connected to: 127.0.0.1
insert  query update delete getmore command flushes mapped  vsize    res faults     locked db idx miss %     qr|qw   ar|aw  netIn netOut  conn       time 
    13    171    424      2       0    15|0       0    24g  48.5g  2.64g      0 countly:11.9%          0       0|0     3|0   102k    67k    12   13:51:04 
     9    225   1358    105       0    13|0       0    24g  48.5g  2.65g      0  countly:6.5%          0       0|0     0|1   255k   177k    12   13:51:05 
   218    967   3056     11       0    82|0       0    24g  48.5g  2.63g      0 countly:20.7%          0       0|0     2|0   807k   965k    12   13:51:06 
    19    158    315      2       0    12|0       0    24g  48.5g  2.65g      0  countly:1.7%          0       0|0     1|0    88k    92k    12   13:51:07 
    17    132    270      2       0     8|0       0    24g  48.5g  2.58g      0  countly:2.1%          0       0|0     2|0    77k   115k    12   13:51:08 
    14    151    433      2       0    11|0       0    24g  48.5g  2.64g      0  countly:2.2%          0       0|0     3|0   106k    89k    12   13:51:09 
    22    181    335      3       0    17|0       0    24g  48.5g  2.66g      0  countly:7.9%          0       0|0     3|0    96k   105k    12   13:51:10 
    36    230    765      3       0     8|0       0    24g  48.5g  2.66g      0  countly:3.6%          0       1|0     0|1   184k   224k    12   13:51:11 
   122    567   1580      1       0    56|0       0    24g  48.5g  2.67g      0 countly:16.4%          0       0|0     2|0   440k   559k    12   13:51:12 
     8    124    286      4       0     8|0       0    24g  48.5g  2.65g      0  countly:2.0%          0       0|0     2|0    67k    61k    12   13:51:13 
insert  query update delete getmore command flushes mapped  vsize    res faults     locked db idx miss %     qr|qw   ar|aw  netIn netOut  conn       time 
    21    193    450      1       0    18|0       0    24g  48.5g  2.66g      0  countly:5.0%          0       0|0     3|0   119k   126k    12   13:51:14 
   156    774    950      1       0    67|0       0    24g  48.5g  2.65g      0 countly:18.5%          0       0|0     3|0   414k   623k    12   13:51:15 
     7     69    194      2       0    12|0       0    24g  48.5g  2.65g      0  countly:1.4%          0       0|0     3|0    49k    40k    12   13:51:16 
    33    280    525      2       0    12|0       0    24g  48.5g  2.64g      0  countly:4.8%          0       0|0     3|0   145k   187k    12   13:51:17 
    18    201    563      5       0    17|0       0    24g  48.5g  2.66g      0  countly:5.5%          0       0|0     3|0   141k   126k    12   13:51:18 
    10    144    334      2       0    15|0       0    24g  48.5g  2.65g      0  countly:4.4%          0       0|0     4|0    84k   103k    12   13:51:19 
    14    210    525      4       0    19|0       0    24g  48.5g  2.63g      0  countly:5.8%          0       0|0     1|0   125k    96k    12   13:51:20 
   563   1959   2259      2       0   222|0       0    24g  48.5g  2.61g      0 countly:49.2%          0       0|0     1|0     1m     2m    12   13:51:21 
   227    692   1326      2       0    60|0       0    24g  48.5g  2.68g      3 countly:14.3%          0       0|0     2|0   647k   652k    12   13:51:22 
    17    146    315      1       0     8|0       0    24g  48.5g  2.65g      0  countly:3.5%          0       2|0     0|1    87k    95k    12   13:51:23 
insert  query update delete getmore command flushes mapped  vsize    res faults     locked db idx miss %     qr|qw   ar|aw  netIn netOut  conn       time 
    23    166    376      1       0    10|0       0    24g  48.5g  2.65g      0  countly:7.4%          0       0|0     2|0   104k   104k    12   13:51:24 
    18    113    212     *0       0     7|0       0    24g  48.5g  2.64g      0 countly:12.6%          0       0|0     2|0    65k    75k    12   13:51:25 
    13    177    375     *0       0    19|0       0    24g  48.5g  2.64g      0  countly:2.4%          0       3|0     0|1    93k   110k    12   13:51:26 
    20    147    299      5       0     5|0       0    24g  48.5g  2.63g      0  countly:1.9%          0       0|0     2|0    85k   103k    12   13:51:27 
    40    280    476      2       0    15|0       1    24g  48.5g  2.63g      0  countly:2.9%          0       0|0     1|0   157k   247k    12   13:51:28 
     7    108    263      2       0     5|0       0    24g  48.5g  2.64g      0  countly:1.6%          0       0|0     2|0    62k    78k    12   13:51:29 
    17    181    394     *0       0     8|0       0    24g  48.5g  2.59g      0  countly:5.4%          0       0|0     2|0   101k   147k    12   13:51:30 
    17    152    350      1       0    12|0       0    24g  48.5g  2.63g      0  countly:3.7%          0       0|0     1|0    90k    80k    12   13:51:31 
    14    123    261      4       0    11|0       0    24g  48.5g  2.58g      0  countly:4.0%          0       2|0     0|1    72k   108k    12   13:51:32 
   184    741   1220     21       0    84|0       0    24g  48.5g  2.67g      0 countly:25.3%          0       0|0     4|0   492k   654k    12   13:51:33 

mongodb的锁很多,怀疑是因为写入慢,一直在等待,浪费了大量的cpu时间

进入到mongodb中查看

root@localhost:~# mongo
MongoDB shell version: 2.4.14
connecting to: test
> db.serverStatus() 
{
        "host" : "localhost",
        "version" : "2.4.14",
        "process" : "mongod",
        "pid" : 16870,
        "uptime" : 1551490,
        "uptimeMillis" : NumberLong(1551489627),
        "uptimeEstimate" : 1537152,
        "localTime" : ISODate("2017-07-14T06:12:34.348Z"),
        "asserts" : {
                "regular" : 0,
                "warning" : 0,
                "msg" : 0,
                "user" : 1,
                "rollovers" : 0
        },
        "backgroundFlushing" : {
                "flushes" : 25858,
                "total_ms" : 6413133,
                "average_ms" : 248.01349679016164,
                "last_ms" : 111,
                "last_finished" : ISODate("2017-07-14T06:12:27.580Z")
        },
        "connections" : {
                "current" : 12,
                "available" : 19988,
                "totalCreated" : NumberLong(660)
        },
        "cursors" : {
                "totalOpen" : 2,
                "clientCursors_size" : 2,
                "timedOut" : 125,
                "totalNoTimeout" : 2
        },
        "dur" : {
                "commits" : 29,
                "journaledMB" : 0.28672,
                "writeToDataFilesMB" : 0.272729,
                "compression" : 0.9344831856907262,
                "commitsInWriteLock" : 0,
                "earlyCommits" : 0,
                "timeMs" : {
                        "dt" : 3071,
                        "prepLogBuffer" : 0,
                        "writeToJournal" : 49,
                        "writeToDataFiles" : 4,
                        "remapPrivateView" : 5
                }
        },
        "extra_info" : {
                "note" : "fields vary by platform",
                "heap_usage_bytes" : 174045352,
                "page_faults" : 37855
        },
        "globalLock" : {
                "totalTime" : NumberLong("1551489627000"),
                "lockTime" : NumberLong("28450005021"),
                "currentQueue" : {
                        "total" : 0,
                        "readers" : 0,
                        "writers" : 0
                },
                "activeClients" : {
                        "total" : 2,
                        "readers" : 2,
                        "writers" : 0
                }
        },
        "indexCounters" : {
                "accesses" : 2559023213,
                "hits" : 2559023356,
                "misses" : 0,
                "resets" : 0,
                "missRatio" : 0
        },
        "locks" : {
                "." : {
                        "timeLockedMicros" : {
                                "R" : NumberLong(1631854144),
                                "W" : NumberLong("28450005021")
                        },
                        "timeAcquiringMicros" : {
                                "R" : NumberLong(1839692044),
                                "W" : NumberLong(269051643)
                        }
                },
                "admin" : {
                        "timeLockedMicros" : {

                        },
                        "timeAcquiringMicros" : {

                        }
                },
                "local" : {
                        "timeLockedMicros" : {
                                "r" : NumberLong(2235967),
                                "w" : NumberLong(0)
                        },
                        "timeAcquiringMicros" : {
                                "r" : NumberLong(856059),
                                "w" : NumberLong(0)
                        }
                },
                "countly" : {
                        "timeLockedMicros" : {
                                "r" : NumberLong("4052844846798"),
                                "w" : NumberLong("77669514748")
                        },
                        "timeAcquiringMicros" : {
                                "r" : NumberLong("2069545770119"),
                                "w" : NumberLong("82155316804")
                        }
                },
                "countly_drill" : {
                        "timeLockedMicros" : {
                                "r" : NumberLong(874145401),
                                "w" : NumberLong("19474885608")
                        },
                        "timeAcquiringMicros" : {
                                "r" : NumberLong(271347249),
                                "w" : NumberLong(1475646788)
                        }
                },
                "test" : {
                        "timeLockedMicros" : {
                                "r" : NumberLong(68640),
                                "w" : NumberLong(79192)
                        },
                        "timeAcquiringMicros" : {
                                "r" : NumberLong(146),
                                "w" : NumberLong(26)
                        }
                }
        },
        "network" : {
                "bytesIn" : 233060269131,
                "bytesOut" : 258427384819,
                "numRequests" : 1384182586
        },
        "opcounters" : {
                "insert" : 43661297,
                "query" : 434433635,
                "update" : 877502004,
                "delete" : 4880964,
                "getmore" : 1023,
                "command" : 26378757
        },
        "opcountersRepl" : {
                "insert" : 0,
                "query" : 0,
                "update" : 0,
                "delete" : 0,
                "getmore" : 0,
                "command" : 0
        },
        "recordStats" : {
                "accessesNotInMemory" : 8466,
                "pageFaultExceptionsThrown" : 3696,
                "countly" : {
                        "accessesNotInMemory" : 1231,
                        "pageFaultExceptionsThrown" : 221
                },
                "countly_drill" : {
                        "accessesNotInMemory" : 7235,
                        "pageFaultExceptionsThrown" : 3475
                },
                "local" : {
                        "accessesNotInMemory" : 0,
                        "pageFaultExceptionsThrown" : 0
                },
                "test" : {
                        "accessesNotInMemory" : 0,
                        "pageFaultExceptionsThrown" : 0
                }
        },
        "writeBacksQueued" : false,
        "mem" : {
                "bits" : 64,
                "resident" : 2692,
                "virtual" : 49774,
                "supported" : true,
                "mapped" : 24630,
                "mappedWithJournal" : 49260
        },
        "metrics" : {
                "document" : {
                        "deleted" : NumberLong(4398459),
                        "inserted" : NumberLong(43661297),
                        "returned" : NumberLong(249187499),
                        "updated" : NumberLong(868225290)
                },
                "getLastError" : {
                        "wtime" : {
                                "num" : 1332687,
                                "totalMillis" : 154
                        },
                        "wtimeouts" : NumberLong(0)
                },
                "operation" : {
                        "fastmod" : NumberLong(846861873),
                        "idhack" : NumberLong(1085704909),
                        "scanAndOrder" : NumberLong(0)
                },
                "queryExecutor" : {
                        "scanned" : NumberLong(1095979673)
                },
                "record" : {
                        "moves" : NumberLong(1249911)
                },
                "repl" : {
                        "apply" : {
                                "batches" : {
                                        "num" : 0,
                                        "totalMillis" : 0
                                },
                                "ops" : NumberLong(0)
                        },
                        "buffer" : {
                                "count" : NumberLong(0),
                                "maxSizeBytes" : 268435456,
                                "sizeBytes" : NumberLong(0)
                        },
                        "network" : {
                                "bytes" : NumberLong(0),
                                "getmores" : {
                                        "num" : 0,
                                        "totalMillis" : 0
                                },
                                "ops" : NumberLong(0),
                                "readersCreated" : NumberLong(0)
                        },
                        "oplog" : {
                                "insert" : {
                                        "num" : 0,
                                        "totalMillis" : 0
                                },
                                "insertBytes" : NumberLong(0)
                        },
                        "preload" : {
                                "docs" : {
                                        "num" : 0,
                                        "totalMillis" : 0
                                },
                                "indexes" : {
                                        "num" : 0,
                                        "totalMillis" : 0
                                }
                        }
                },
                "ttl" : {
                        "deletedDocuments" : NumberLong(10),
                        "passes" : NumberLong(25857)
                }
        },
        "ok" : 1
}

锁确实比较多,查看mongotop

root@localhost:/mnt/mongodb/log# mongotop
connected to: 127.0.0.1

                                                                ns       total        read       write          2017-07-14T06:46:41
                      countly.online_users55d2e563292d6b2c7d1d132d      1669ms      1665ms         4ms
                      countly.online_users5770b587f4a7f02a6a2b9cce        53ms        52ms         1ms
                                                  countly.carriers        42ms         0ms        42ms
                      countly.online_users5506ab1979a3a67e5214cda6        40ms        38ms         2ms
                                                   countly.devices        16ms         0ms        16ms
                                                      countly.apps         9ms         9ms         0ms
                                                     countly.users         8ms         0ms         8ms
                                            countly.device_details         7ms         0ms         7ms
                         countly.app_users55d2e563292d6b2c7d1d132d         4ms         0ms         4ms

                                                                ns       total        read       write          2017-07-14T06:46:42
                      countly.online_users55d2e563292d6b2c7d1d132d      3671ms      3669ms         2ms
                      countly.online_users5770b587f4a7f02a6a2b9cce        64ms        63ms         1ms
                      countly.online_users5506ab1979a3a67e5214cda6        26ms        25ms         1ms
                                                   countly.devices        19ms         0ms        19ms
                                                  countly.carriers        16ms         0ms        16ms
                                                     countly.users         6ms         0ms         6ms
                                                      countly.apps         6ms         6ms         0ms
                         countly.app_users55d2e563292d6b2c7d1d132d         4ms         1ms         3ms
countly_drill.drill_eventse02cd05a335f76036692df25f94be2b3d5c20bd2         3ms         0ms         3ms

                                                                ns       total        read       write          2017-07-14T06:46:43
                      countly.online_users55d2e563292d6b2c7d1d132d      1709ms      1705ms         4ms
                      countly.online_users5506ab1979a3a67e5214cda6        33ms        31ms         2ms
                                                  countly.carriers        19ms         0ms        19ms
                                                   countly.devices        10ms         0ms        10ms
                                                     countly.users        10ms         0ms        10ms
                      countly.online_users54feb14879a3a67e52f6f7cd         7ms         7ms         0ms
                                                      countly.apps         5ms         5ms         0ms
                         countly.app_users55d2e563292d6b2c7d1d132d         4ms         0ms         4ms
countly_drill.drill_eventsbb446ae67767dec699789c496e6c067b32dee999         3ms         0ms         3ms

                                                                ns       total        read       write          2017-07-14T06:46:44
                      countly.online_users55d2e563292d6b2c7d1d132d      2301ms      2298ms         3ms
                      countly.online_users5770b587f4a7f02a6a2b9cce       262ms       261ms         1ms
                      countly.online_users5506ab1979a3a67e5214cda6        35ms        34ms         1ms
countly_drill.drill_eventse02cd05a335f76036692df25f94be2b3d5c20bd2        22ms         0ms        22ms
                      countly.online_users54feb14879a3a67e52f6f7cd         8ms         7ms         1ms
                                                     countly.users         7ms         0ms         7ms
countly_drill.drill_eventscc860280b73e64c3413b38f2e9959acb7babd3f9         6ms         0ms         6ms
                         countly.app_users55d2e563292d6b2c7d1d132d         5ms         2ms         3ms
                                                      countly.apps         4ms         4ms         0ms

是读的时间很长,这个怀疑是没有建立索引,另外mongodb本身有一个bug

技术分享

https://docs.mongodb.com/manual/administration/production-notes/

解决方法如下所示

#numactl --interleave all command

原理如下

http://www.cnblogs.com/Lifehacker/p/database_swap_insanity_on_Linux.html

这么操作了好像也没什么效果,最后想到之前这个数据库因为磁盘爆满清空过,之前听说过mongodb迁移后会丢失索引,要重建索引,我查了一下索引

> db.online_users55d2e563292d6b2c7d1d132d.find()
{ "_id" : "4e5d3922-6553-430a-849f-8b1007373d7c", "la" : ISODate("2017-06-30T00:55:08.933Z"), "n" : 0 }
{ "_id" : "55d7d514-f75c-432a-b557-17c036437a32", "la" : ISODate("2017-06-29T16:38:12.863Z"), "n" : 0 }
{ "_id" : "0118022e-54d2-4fb9-998d-a544d9a76d42", "la" : ISODate("2017-07-05T17:17:44.307Z"), "n" : 0 }
{ "_id" : "2a160945-fcc1-42f5-8fd5-693f97c4c332", "la" : ISODate("2017-07-02T07:01:35.619Z"), "n" : 0 }
{ "_id" : "14cd09d4-fd09-43b8-9259-8a007fc08b45", "la" : ISODate("2017-06-28T01:23:29.400Z"), "n" : 0 }
{ "_id" : "b9fa7cc9-cd8c-4a1b-83fb-fc4c4f929715", "la" : ISODate("2017-06-26T13:38:32.887Z"), "n" : 1 }
{ "_id" : "1d968b85-2571-4cf7-9520-bfeb44d893bc", "la" : ISODate("2017-07-13T13:08:18.807Z"), "n" : 0 }
{ "_id" : "0b1a3d07-8e5d-4e78-a93f-d13811a81b86", "la" : ISODate("2017-07-13T03:10:39.927Z"), "n" : 0 }
{ "_id" : "18d8f9fb-dff3-46e6-a3c7-bb19db748c4a", "la" : ISODate("2017-07-08T17:59:37.518Z"), "n" : 0 }
{ "_id" : "a81349ac-9d83-4ef7-be0b-1b9bf64f867f", "la" : ISODate("2017-07-13T23:27:35.174Z"), "n" : 0 }
{ "_id" : "dd841258-36a6-45de-bc45-450441d9bf33", "la" : ISODate("2017-07-14T06:34:23.838Z"), "n" : 0 }
{ "_id" : "75720c9f-69af-4833-ae14-78a381cc31b6", "la" : ISODate("2017-07-13T15:57:28.700Z"), "n" : 0 }
{ "_id" : "ef37ff3d-979d-4275-9991-398ccb2d5576", "la" : ISODate("2017-07-14T04:45:38.872Z"), "n" : 0 }
{ "_id" : "465f923e-b0f1-4aa7-8f68-2d7db95973b2", "la" : ISODate("2017-07-07T11:47:53.029Z"), "n" : 0 }
{ "_id" : "e6b5fd38-bb56-4634-9232-7eeceb8cf46a", "la" : ISODate("2017-07-09T03:27:04.545Z"), "n" : 0 }
{ "_id" : "d4957c5d-154f-4c4f-8843-9792beab4817", "la" : ISODate("2017-07-11T21:49:35.417Z"), "n" : 0 }
{ "_id" : "08d0d626-b37b-4761-81b2-98b5ee426df8", "la" : ISODate("2017-06-29T13:56:14.974Z"), "n" : 0 }
{ "_id" : "6d2eef45-cb25-4ff6-b8cd-497ab3b938ae", "la" : ISODate("2017-07-07T02:19:16.154Z"), "n" : 0 }
{ "_id" : "6f444a2b-e2fc-4ab0-8b1d-d4334bf095e3", "la" : ISODate("2017-07-09T12:17:05.711Z"), "n" : 0 }
{ "_id" : "85e06ac1-7158-42fb-95ed-2cd39a7f0687", "la" : ISODate("2017-06-26T02:27:25.389Z"), "n" : 1 }
Type "it" for more
> db.online_users55d2e563292d6b2c7d1d132d.find({"_id" : "4e5d3922-6553-430a-849f-8b1007373d7c"}).explain()
{
        "cursor" : "BtreeCursor _id_",
        "isMultiKey" : false,
        "n" : 1,
        "nscannedObjects" : 1,
        "nscanned" : 1,
        "nscannedObjectsAllPlans" : 1,
        "nscannedAllPlans" : 1,
        "scanAndOrder" : false,
        "indexOnly" : false,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "millis" : 0,
        "indexBounds" : {
                "start" : {
                        "_id" : "4e5d3922-6553-430a-849f-8b1007373d7c"
                },
                "end" : {
                        "_id" : "4e5d3922-6553-430a-849f-8b1007373d7c"
                }
        },
        "server" : "localhost:27017"
}

明明是走索引的,可能是la这一列没有加索引(这个我当时没有看,但是从现在来看这个思路应该是正确的)

加上索引

db.online_users547c2a634dcc1eef3d000001.ensureIndex({la: 1}, {expireAfterSeconds: 180});

因为这个是cache性质频繁读写的表,要转成Capped Collection,自动老化移出,优化性能,并设置最大记录数

db.runCommand({"convertToCapped":"online_users547c2a634dcc1eef3d000001",size:60})

 db.runCommand({"convertToCapped":"live_data547c2a634dcc1eef3d000001",size:60})

这样之后结果就是好多了,再观察一段时间吧

记一次mogodb占用cpu高问题