RabbitMQ is a message broker for organizing interaction between Nemesida WAF components by sending special messages to the appropriate queue.
Queue overflow indicates possible problems in the operation of the component that interacts with it. At the same time, in the error log of the filtering node /var/log/nginx/error.log for filled queues, in addition to nwaf_local and mlc, a corresponding entry will appear:
Nemesida WAF: RabbitMQ %queue_name% queue is full (more than %message_count% messages), sending messages to the queue has been suspended
After receiving the warning, it is necessary to check the queue list and identify the one that is overflowing:
# rabbitmqctl list_queues
There are 5 main queues in total:
- nwaf – a queue of requests that the dynamic module places for processing by the Nemesida AI MLC module;
- api – a queue of requests that the dynamic module places to be sent to Nemesida WAF API via Nemesida AI MLC;
- log – a queue of events that the dynamic module places to be sent to Nemesida WAF API via Nemesida AI MLC;
- mlc – a queue of messages in which the result of request processing by the Nemesida AI MLC module is returned;
- nwaf_local – a queue of requests that Nemesida AI MLC collects from the nwaf filtering node queue for local processing.
An indicator of a possible malfunction of a component is the constant filling of the corresponding queue, which does not decrease over time. For example, if the nwaf queue on the filtering node is empty, but the nwaf_local queue on the Nemesida AI MLC component server is full, this means there may be a problem with the Nemesida AI MLC component, which means it cannot or does not have time to sort the message queue (for example, if there is a shortage of hardware resources). In this case, it is necessary to check the error log of the component /var/log/nwaf/mlc.log, identify problems and eliminate them. If there are no errors, please contact the support service technical support.
Important: if with intense traffic, the queue is not empty, but is constantly being filled and sorted, then this is a regular behavior that does not indicate problems with the component.
Due to the fact that the nwaf_local queue does not contain size restrictions, its overflow may negatively affect the size of the server’s free disk space. In this case, it may be necessary to force the queue to be cleared. You can forcibly clear the queue using the following command:
# rabbitmqadmin purge queue name=%queue_name%