首页 > 代码库 > Tomcat应用报redis超时的故事

Tomcat应用报redis超时的故事

北京时间 2016年9月26日  17:26:30 PM

时间停留在N天前,线上应用总是时不时的出现用户登录失败的现象,喝了杯茶,细细品味下它的问题君存在的过程

业务背景介绍                        

系统架构是通过分布式的方式进行部署的,登录的时候不会调用redis数据,假如用户登录成功了,会跳转到其它的项目上,并根据从redis中查到的权限缓存,像是相应的数据模块

问题排查                          

问题君发飙最初手段

问题一:用户登录失败,系统巨卡无比

问题二:Tomcat抛出 connect timed out错误

2016-09-24 16:44:24,397 - [ERROR] [RedisClient]:269- java.net.SocketTimeoutException: connect timed outredis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: connect timed out        at redis.clients.jedis.Connection.connect(Connection.java:134)        at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:69)        at redis.clients.jedis.Connection.sendCommand(Connection.java:79)        at redis.clients.jedis.BinaryClient.exists(BinaryClient.java:99)        at redis.clients.jedis.Client.exists(Client.java:29)        at redis.clients.jedis.Jedis.exists(Jedis.java:92)        at redis.clients.jedis.ShardedJedis.exists(ShardedJedis.java:48)        at com.tools.common.redis.client.RedisClient.exists(RedisClient.java:267)        at com.group.erp.rpc.dubbo.service.impl.ActionRpcServiceImpl.getActionList(ActionRpcServiceImpl.java:23)        at com.alibaba.dubbo.common.bytecode.Wrapper1.invokeMethod(Wrapper1.java)        at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)        at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)        at com.alibaba.dubbo.rpc.filter.ExceptionFilter.invoke(ExceptionFilter.java:55)        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)        at com.alibaba.dubbo.rpc.filter.ClassLoaderFilter.invoke(ClassLoaderFilter.java:38)        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)        at com.alibaba.dubbo.rpc.filter.EchoFilter.invoke(EchoFilter.java:38)        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)        at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:73)        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)        at com.alibaba.dubbo.rpc.filter.GenericFilter.invoke(GenericFilter.java:68)        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)        at com.alibaba.dubbo.rpc.filter.TimeoutFilter.invoke(TimeoutFilter.java:42)        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)        at com.alibaba.dubbo.rpc.protocol.dubbo.filter.TraceFilter.invoke(TraceFilter.java:78)        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)        at com.alibaba.dubbo.rpc.filter.ContextFilter.invoke(ContextFilter.java:60)        at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)        at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)        at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)        at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)        at com.alibaba.dubbo.remoting.transport.dispather.ChannelEventRunnable.run(ChannelEventRunnable.java:78)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)        at java.lang.Thread.run(Thread.java:745)Caused by: java.net.SocketTimeoutException: connect timed out        at java.net.PlainSocketImpl.socketConnect(Native Method)        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)        at java.net.Socket.connect(Socket.java:589)        at redis.clients.jedis.Connection.connect(Connection.java:129)        ... 34 more

按照常规思路进行排查

查找问题君之网卡流量状况

 

Tomcat应用报redis超时的故事