加载中…AMQ(ActiveMQ) pending messages数量一直在增加
(2014-07-23 16:40:29)
标签：
股票 |
分类： MiddleWare |
最近公司核心系统的ActiveMQ似乎出了问题, 从ActiveMQ Web Console的queue页面来看，number of
pending
messages数量一直在增加，也就是入程序设定的队列的messages一直增加，没有Consumer进行消费。感觉好像ActiveMQ所谓假死了（程序员的说法）。
The system is out of physical RAM or swap
space
In 32 bit mode, the process size limit was
hit
Reduce memory load on the system
Increase physical memory or swap
space
Check if swap backing store is full
Use 64 bit Java on a 64 bit OS
Decrease Java heap size (-Xmx/-Xms)
Decrease number of Java threads
Decrease Java thread stack sizes
(-Xss)
Disable compressed references
(-XXcompressedRefs=false)
ActiveMQ的日志如下：
2014-07-22 00:12:26,429 | ERROR | Failed
to page in more queue messages |
org.apache.activemq.broker.region.Queue | ActiveMQ
BrokerService[localhost] Task-25659
java.lang.OutOfMemoryError: Resource
temporarily unavailable in tsStartJavaThread
(lifecycle.c:1096).
Attempting to allocate 512M
bytes
There is insufficient native memory for
the Java
Runtime Environment to
continue.
Possible reasons:
Possible solutions:
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at
org.apache.activemq.thread.PooledTaskRunner.wakeup(PooledTaskRunner.java:81)
at
org.apache.activemq.broker.TransportConnection.dispatchAsync(TransportConnection.java:814)
at
org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:703)
at
org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:644)
at
org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:155)
at
org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1897)
at
org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1836)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1993)
at
org.apache.activemq.broker.region.Queue.iterate(Queue.java:1486)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2014-07-22 14:09:13,972 | WARN | Transport
Connection to: tcp://192.168.135.1:6782 failed:
org.apache.activemq.transport.InactivityIOException: Channel was
inactive for too (>30000) long: tcp://192.168.135.1:6782 |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
InactivityMonitor Worker
2014-07-22 14:09:13,972 | WARN | Transport
Connection to: tcp://192.168.135.1:6781 failed:
org.apache.activemq.transport.InactivityIOException: Channel was
inactive for too (>30000) long: tcp://192.168.135.1:6781 |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
InactivityMonitor Worker
2014-07-22 14:09:13,972 | WARN | Transport
Connection to: tcp://192.168.135.1:6779 failed:
org.apache.activemq.transport.InactivityIOException: Channel was
inactive for too (>30000) long: tcp://192.168.135.1:6779 |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
InactivityMonitor Worker
2014-07-22 14:09:13,972 | WARN | Transport
Connection to: tcp://192.168.135.1:6780 failed:
org.apache.activemq.transport.InactivityIOException: Channel was
inactive for too (>30000) long: tcp://192.168.135.1:6780 |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
InactivityMonitor Worker
2014-07-22 14:09:13,974 | WARN | Cannot
create async transport stopper thread. This exception is ignored.
Not waiting for stop to complete |
org.apache.activemq.broker.TransportConnection | ActiveMQ
InactivityMonitor Worker
java.lang.OutOfMemoryError: Resource temporarily unavailable
in tsStartJavaThread (lifecycle.c:1096).
以上信息显示可能是ActiveMQ内存不足了（只设定500MB）,于是增加AMQ的内存，将对应应用服务器充应用负载设备下线，不让新的messages入队列，然后将AMQ的内存从500MB增加到1024MB，重启AMQ，问题依旧。
既然MESSAGE可以入队列，应该AMQ没有问题，现在还是好像没有Consumer去处理MESSAGE，去查看WEBLOGIC一个服务器（PATEWAY），因为只有这个服务器部署程序是这个队列的Consumer，但是开发人员似乎没有发现什么问题。但是，考虑到以前GATEWAY出现一些未异常，于是将PATEWAY重新启动。
PATEWAY重新启动之后（最好在从应用负责设备离线之后），AMQ的Consumer开始处理队列的MESSAGE,
AMQ的情况慢慢好转。
至此看来，很可能是PATEWAY部署的程序出现异常导致不再去处理队列中MESSAGE，结果导致AMQ队列用完所有的设定内存，后续的MESSAGE进入队列之前，之前未处理的MSSAGE被写入本地磁盘，pending的MESSAGE越来越多，误导程序员认为AMQ出现所谓的假死。
喜欢
0
赠金笔