Guide to solving main issues while setting up and operating the Nemesida WAF.
- Attacks not blocked
- Attacks are blocked, but are not displayed in the Nemesida WAF Cabinet
- The settings management functionality does not appear in the Nemesida WAF Cabinet
- Real IP addresses of clients are not displayed
- The behavioral model is not displayed in the Nemesida WAF Cabinet
- After the training is completed, the machine learning module does not analyze requests
- The machine learning module blocks legitimate requests
- Nemesida WAF blocks the request due to an internal error in the operation of the components
- Nemesida WAF blocks file uploads
- Slow loading of Nemesida WAF Cabinet pages
🔗 Attacks not blocked
Nemesda WAF comes as a dynamic module for Nginx. After the initial configuration of the component, it is necessary to check the correct operation of the signature attack detection method by sending a http://YOUR_SERVER/nwaftest
request.
The server should return the 403
response code, and an entry will appear on the filtering node in the error
-log of the Nginx web server:
Nemesida WAF: the request 5274fe3c397782a09b4f1b057e572e21 blocked by rule ID 1 in zone URL, ...
If the request is not blocked:
- check that after making changes to the Nginx configuration of the filtering node, the services were restarted
nginx nwaf_update mla_main
; - check that the dynamic module
/etc/nginx/modules/ngx_http_waf_module.so
is in/etc/nginx/modules
and included in/etc/nginx/nginx. conf
; - check the size of
/etc/nginx/nwaf/rules.bin
file, it should not be empty. If you suspect a violation of the integrity of the file, you can delete it and download it again by restarting thenwaf_update
service; - check if the request is received for processing by the Filtering node;
- check that there are no request processing exclusion rules for the virtual host/IP address;
- check that the IP address you are accessing from is not on the allow list (WL).
🔗 Attacks are blocked, but are not displayed in the Nemesida WAF Cabinet
In cases where attacks are blocked and displayed in the error
log of the Nginx web server, but are not displayed in the Nemesida WAF Cabinet, you need to check:
- the address of the Nemesida WAF API server is correctly set in
/etc/nginx/nwaf/conf/global/nwaf.conf
in the formatnwaf_api_conf host=http://api.example.com:8080/nw -api
; - the proxy server address (if used) is correctly set for accessing the Nemesida WAF API in
/etc/nginx/nwaf/conf/global/nwaf.conf
; - correct installation and configuration of the Nemesida WAF API component, no errors in the log
/var/log/uwsgi/nw-api/nw-api-logging.log
; - the
rmq_host
parameter is set correctly in the Nemesida AI MLC module configuration file/opt/mlc/mlc.conf
; - the Nemesida WAF API server has access to the database to connect. You can check access to the database by running the command:
# psql -h <server_ip> -U nw_api waf
When connecting, enter the user's passwordnw_api
.
If access is not provided, it must be provided by making changes to thepg_hba.conf
PostgreSQL configuration file.
Example:# IPv4 local connections: host all all 10.1.1.0/24 md5
🔗 The settings management functionality does not appear in the Nemesida WAF Cabinet
To activate the Nemesida WAF settings management functionality, several conditions must be met:
- added license key to settings user;
- the functionality is activated by the administrator;
- the connection address to the Nemesida WAF API is specified in the
API_URI
parameter (file/var/www/app/cabinet/settings.py
).
🔗 Real IP addresses of clients are not displayed
Server with web application does not get real IP addresses of clients
All legitimate requests who not blocked by Nemesida WAF are sent to the web application upstream. In order for the web application to receive the correct client IP address from the filtering node (Nginx web server with the Nemesida WAF dynamic module installed), you need to add the set_real_ip_from
header.
To do this, on the web application server, bring the configuration file nginx.conf
to the form:
http { ... set_real_ip_from x.x.x.x; ... }
x.x.x.x
– filtering node address (Nginx web server with Nemesida WAF dynamic module installed).
After making the change, check the configuration and restart the Nginx service:
# nginx -t && service nginx reload
Real IP-addresses of clients are not displayed in the Nemesida WAF Cabinet
The Nemesida WAF Cabinet is designed to visualize information about attacks. All necessary information about the attack, including the source IP address of the requests, is transmitted from the dynamic Nemesida WAF module.
If an additional server processes requests in the reverse proxy
mode and passes them to the filtering node (Nginx web server with the Nemesida WAF dynamic module installed), all requests will come with a single IP address of this server. To get correct information about client IP addresses, add the set_real_ip_from
header.
To get the real IP addresses of clients, on the server with the Nemesida WAF dynamic module installed, bring the configuration file nginx.conf
to the form:
http { ... set_real_ip_from x.x.x.x; ... }
x.x.x.x
– address of the server operating in the reverse proxy
mode.
After making the change, check the configuration and restart the Nginx service:
# nginx -t && service nginx reload
🔗 The behavioral model is not displayed in the Nemesida WAF Cabinet
The Nemesida WAF Cabinet allows to manage behavioral models. To create a behavioral model, it is enough to add a virtual host in the section management Nemesida AI MLC settings. After 4 days of training, a behavioral model will be formed and available in the Nemesida WAF Cabinet. If the behavioral model is not available, then you need to check the Nemesida AI MLC /var/log/nwaf/mlc.log
component log, paying attention to the following entries:
-
Current training progress for virtual host example.com: 5%
The entry means that the training has not been completed. When the progress is equal to 100%, the training will be completed automatically and the behavioral model will appear in the Nemesida WAF Cabinet.
-
An error occurred while sending the models of virtual host example.com to Nemesida WAF API: status: <Response [413]>
The entry means that the model has already been formed, but cannot be sent to the Nemesida WAF API due to its restrictions on the maximum size of the request body.
Solution: increase the value of the
client_max_body_size
parameter by bringing the configuration file/etc/nginx/nwaf/conf.d/nwaf-api.conf
to the form:server { ... client_max_body_size 64M; ... }
The value of the parameter to be set must be greater than the value of the file size with the extension
.ml
in the directory/opt/mlc/ml/tmp/
. After making changes, you need to restart the Nemesida WAF API component:# systemctl restart nw-api rldscupd memcached
-
Not enough available memory (12%) to process
The entry means that there are not enough server hardware resources to handle the traffic.
Solution: increase the amount of server RAM.
-
Insufficient traffic to build behavioral models for virtual host example.com (required: 25)
The entry means that Nemesida AI MLC has not received enough traffic to build a behavioral model and training will continue.
Solution:
- check that the web application traffic goes to the filtering node;
- in the configuration file
/opt/mlc/mlc.conf
check that the connection details to the RabbitMQ service on the filtering node are correctly specified; - check that the Nemesida AI MLC collects traffic from the corresponding RabbitMQ queue (no entries accumulate in the
nwaf
queue) on the filtering node.
🔗 After the training is completed, the machine learning module does not analyze requests
The Nemesida AI MLC component is responsible for building a behavioral model. After 4 days of training, the behavioral model will be formed and applied on the filtering node, and will also be displayed in the Nemesida WAF Cabinet. If the behavioral model is not available, then you need to check the Nemesida AI MLC /var/log/nwaf/mlc.log
component log, paying attention to the following entry:
Current training progress for virtual host example.com: 5%
The entry means that the training has not been completed. When the progress is equal to 100%, the training will be completed automatically and the behavioral model:
- will appear in the
/opt/mlc/ml
directory as 2 files:example.com_vect.ml
andexample.com_rf.ml
; - will appear in your Nemesida WAF Cabinet;
- will appear on the filtering node in the directory
/etc/nginx/nwaf/ml
as 2 files:example.com_vect.ml
andexample.com_rf.ml
;
The analysis of requests by the machine learning module will begin only after applying the behavioral model on the filtering node. To check that the behavioral models were applied on the filtering node, it is necessary to check the Nemesida AI MLA
/var/log/nwaf/mla.log
log, paying attention to the following entry:INFO The list of Nemesida AI virtual hosts received from Nemesida WAF API: example.com INFO example.com md5: cdaba1e888de20b3b116d3ad28838a4f INFO Models processing finished
The entry means that a behavioral model for the virtual host is applied on the filtering node example.com
. No entry example.com md5: cdaba1e888de20b3b116d3ad28838a4f
means that the behavioral model has not been applied.
An error in the application of a behavioral model can occur in several cases:
- the behavioral model has been corrupted and cannot be applied by the Nemesida AI MLA machine learning agent;
- different versions of
Python
using incompatible versions of pip packages are installed on the filtering node and Nemesida AI MLC.
If the behavioral model has been damaged and cannot be applied by the Nemesida AI MLA machine learning agent, it is recommended to perform additional training of the behavioral model using a backup copy of the training sample.
If different versions of Python
using incompatible versions of pip packages are installed on the filtering node and Nemesida AI MLC, it is recommended to update the Python
versions to the current one in order to synchronize the packages used on both servers.
If updating the Python
version is not possible on one of the servers, then as a solution, you can downgrade the version of the pip packages used to the versions used in Python
of the earliest version (for example, if the Python 3.7
version is installed on one of the servers, the package version will be downgraded up to this version, if it is the earliest of the installed ones).
Consider an example of downgrading the version of pip packages for Python 3.9
to Python 3.7
:
The behavioral model is created by the Nemesida AI MLC component for a specific version of Python
. If the version of the used pip packages was downgraded for the server from Nemesida AI MLC components, then after downgrading the version of the pip packages, it is necessary to complete the process of retraining the behavioral model using a backup copy of the training sample.
When deciding to downgrade packages, keep in mind that outdated OS versions may use
Python
versions whose support has been completed or will be completed in the near future. Therefore, to solve the problems associated with using different versions ofPython
, we recommend considering updating the OS to the current version. You can check the support status of the intersecting Python version here.
If no analysis is performed by the machine learning module after updating the behavioral model, then it is necessary to check the logs of the components /var/log/nwaf/mla.log
and /var/log/nwaf/mlc.log
. Errors of the form:
An error occurred while loading file /etc/nginx/nwaf/ml/example.com_rf.ml: node array from the pickle has an incompatible dtype: - expected: [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')] - got: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}
it is said that behavioral models were created on the same version of pip dependencies, but after updating the components, the dependencies were updated and further application of the models is impossible. In this case, it is recommended to update the pip dependencies according to the following instructions:
- Filtering node:
- Update the component to the current version;
- Update the pip dependencies by running the script
/usr/share/nwaf/venv/pip_update.sh
on each of the updated components; - Restart the service
mla_main
.
- Nemesida AI MLC:
- Remove all behavioral models using the functionality of your Nemesida WAF Cabinet or API;
- Update the component to the current version;
- Update the pip dependencies by running the script
/usr/share/nwaf/venv/pip_update.sh
on each of the updated components; - Restart the service
mlc_main
.
🔗 The machine learning module blocks legitimate requests
The use of machine learning in request processing allows you to increase the accuracy of detecting attacks to 99.98%, however, to achieve these indicators, it is necessary to correctly train a behavioral model. Blocking a legitimate request can occur for several reasons:
- The traffic was not received for processing by the Nemesida AI MLC module, and therefore the machine learning module considers the request anomaly relative to the applied behavioral model;
- The behavioral model was built incorrectly and needs to be retrained;
- False-positive results that can be eliminated by exporting requests for use in a behavioral model “on the fly”.
There are 2 main ways to eliminate false-positive blocks:
1st method:
1. Analyze blocked requests in the Model Tuning section using the web interface of the Nemesida WAF Cabinet, and export those that are considered legitimate.
Example of the request exporting 2. After legitimate requests are no longer blocked, it is necessary to start retraining the behavioral model.
If the export of queries does not lead to the expected result, then it is pointless to start retraining the behavioral model. In this case, you need to use the 2nd method.
2nd method:
1. Switch the request processing mode of the machine learning module to monitoring mode by activating the Monitoring mode of request analysis identified by Nemesida AI MLA module as illegitimate (BT 3) option in the filtering node settings management section using Nemesida WAF Cabinet:
ACtivate options 2. Analyze the
error
log of the filtering node (/var/log/nginx/error.log
) and identify those requests that were sent for additional analysis to the machine learning module.Requests will be sent for additional analysis by the machine learning module when signatures are detected in the request, the sum of the “weight” of which will be
>=
valuesmla_score
in the configuration file/etc/nginx/nwaf/conf/global/nwaf.conf
(by default,2
). Sending a request for additional analysis will be accompanied by a message in the log.
Example:2023/11/27 17:50:59 [error] 4292#4292: *466 Nemesida WAF: the request b3270f75de9c6a381fb645bd46f0d772 contains rule ID 1039 in zone BODY and will be sent to Nemesida AI MLA, client: 1.1.1.1, server: , request: "POST /admin/uploads.php HTTP/1.1", host: "example.com"
3. Create a signature exclusion rule (or several rules) by selecting the
NoMLA
rule as the trigger zone:Create a WL rule
TheNoMLA
zone excludes sending a request to the machine learning module when the signature is triggered.4. Deactivate the previously activated option Monitoring mode of request analysis identified by Nemesida AI MLA module as illegitimate (BT 3).
5. Start retraining of the behavioral model if the blocking has stopped after adding the exclusion rules. Otherwise, add exclusion rules until the blocking of legitimate requests stops.
6. After completing the training, disable previously created exclusion rules to verify that the problem has been resolved.
🔗 Nemesida WAF blocks the request due to an internal error in the operation of the components
When processing requests, in some cases, a block may occur due to an internal error in the operation of components. Let’s look at some cases in which this error may occur:
Component unavailability
All components of Nemesida WAF interact with each other, for example, the filtering node, if necessary, after conducting a signature analysis, sends a request for additional processing, for example, to Nemesida AI MLA or an Anti-virus solution. In cases where the component does not return a response within the specified time at the time of transmission of the request, the request is blocked to exclude the possibility of bypassing the WAF due to overload of certain components.
Let’s look at the example of interaction between the Filtering node and Nemesida AI MLA (machine learning agent), the signs of blocking the request due to an internal error when the component is unavailable. The signs of blocking will appear as follows:
1. Entries will appear in the log of the filtering node:
[error] 28808#28808: *3 Nemesida WAF: the request 47ee09c37fd353ce359caad7a2d34cd0 contains rule ID 51 in zone URL and will be sent to Nemesida AI MLA, client: 192.168.173.2, server: example.com, request: "GET /q=j%26Tab;avascript:a%26Tab;lert() HTTP/1.1", host: "example.com" [error] 28808#28808: Nemesida WAF: unable to connect to Nemesida AI MLA while processing request 47ee09c37fd353ce359caad7a2d34cd0 (The target address was not listening for connections or refused the connection request) [error] 28808#28808: *3 Nemesida WAF: the request 47ee09c37fd353ce359caad7a2d34cd0 has been blocked because timeout occurred while waiting response from Nemesida AI MLA, client: 192.168.173.2, server: example.com, request: "GET /q=j%26Tab;avascript:a%26Tab;lert() HTTP/1.1", host: "example.com"
2. An entry about the lock with the
Internal error
type will appear in your Nemesida WAF Cabinet:Reasons:
- The
mla_main
service of the Nemesida AI MLA component is unavailable. Check the status of the service:# service mla_main status
Solution: Check the status of the Nemesida AI MLA service, restart it, and make sure that there are no errors in the
/var/log/nwaf/mla.log
log. - There is no access for accessing services inside the local host (
localhost
) on the server of the filtering node.
Solution: Provide the appropriate access.
Component error
In this case, unlike the previous one, an error may occur even if the component to which the request for additional analysis is sent is available. In this case, the signs of blocking will appear as follows:
1. Entries will appear in the log of the filtering node:
[error] 28808#28808: *3 Nemesida WAF: the request 47ee09c37fd353ce359caad7a2d34cd0 contains rule ID 51 in zone URL and will be sent to Nemesida AI MLA, client: 192.168.173.2, server: example.com, request: "GET /q=j%26Tab;avascript:a%26Tab;lert() HTTP/1.1", host: "example.com" [error] 28808#28808: Nemesida WAF: unable to connect to Nemesida AI MLA while processing request 47ee09c37fd353ce359caad7a2d34cd0 (The target address was not listening for connections or refused the connection request) [error] 4012815#4012815: *4080454 Nemesida WAF: the response code 4 (error) received from Nemesida AI MLA during processing the request 47ee09c37fd353ce359caad7a2d34cd0, request blocked, client: 192.168.173.2, server: example.com, request: "GET /q=j%26Tab;avascript:a%26Tab;lert() HTTP/1.1", host: "example.com"
2. The Nemesida AI MLA
/var/log/nwaf/mla.log
log will contain entries like:ERROR ML: a timeout occurred while processing request 47ee09c37fd353ce359caad7a2d34cd0 (example.com) by behavioral model
3. A corresponding lock entry with the type
Internal error
will appear in your Nemesida WAF Cabinet.Solution:
In some cases, the problem may arise due to requests that transmit binary content in the request body. Processing such requests requires a certain amount of hardware resources and time, which, in addition to blocking requests likeInternal error
, can slow down the work of both the filtering node and all web applications whose traffic is proxied through it. If the functionality of the web application supports file transfer, then as a possible solution to the problem, you need to apply parameters in your Nemesida WAF Cabinet that exclude checking the contents of the request body by activating the option:In some cases, the filtering node may also consume an increased amount of hardware resources (in particular, server RAM) due to the processing of a large number of requests that are transmitted for additional analysis by the machine learning module, which can also lead to errors in the operation of the component and blocking the request with the
Internal error
type. This is due to the fact that during the signature analysis of the request, Base64 decoding of the zonesARGS
,BODY
,URL
,HEADERS
is additionally performed, after which signs of an attack (signatures) are detected, and the request is passed to the Nemesida AI MLA module. In most cases, to solve the problem, it will be enough to activate the option to disable the decoding of the corresponding zone in your Nemesida WAF Cabinet:In some cases, the error may be related to errors in the behavioral model itself and it is necessary to retrain it:
- Switch the request processing mode of the machine learning module to monitoring mode by activating the Monitoring mode of request analysis identified by Nemesida AI MLA module as illegitimate (BT 3) option in the filtering node settings management section using Nemesida WAF Cabinet:
- Analyze the
error
log of the filtering node (/var/log/nginx/error.log
) and identify those requests that were sent for additional analysis to the machine learning module.Requests will be sent for additional analysis by the machine learning module when signatures are detected in the request, the sum of the “weight” of which will be
>=
valuesmla_score
in the configuration file/etc/nginx/nwaf/conf/global/nwaf.conf
(by default,2
). Sending a request for additional analysis will be accompanied by a message in the log.
Example:2023/11/27 17:50:59 [error] 4292#4292: *466 Nemesida WAF: the request b3270f75de9c6a381fb645bd46f0d772 contains rule ID 1039 in zone BODY and will be sent to Nemesida AI MLA, client: 1.1.1.1, server: , request: "POST /admin/uploads.php HTTP/1.1", host: "example.com"
- Create a signature exclusion rule (or several rules) by selecting the
NoMLA
rule as the trigger zone:Create a WL rule
TheNoMLA
zone excludes sending a request to the machine learning module when the signature is triggered. - Deactivate the previously activated option Monitoring mode of request analysis identified by Nemesida AI MLA module as illegitimate (BT 3).
- Start retraining of the behavioral model if the blocking has stopped after adding the exclusion rules. Otherwise, add exclusion rules until the blocking of legitimate requests stops.
- After completing the training, disable previously created exclusion rules to verify that the problem has been resolved.
🔗 Nemesida WAF blocks file uploads
Blocking requests when uploading files to the web application server is one of the most common problems that a user may encounter when working with any WAF. This is due to the fact that binary encoding of the transmitted data is performed to transfer files (for example, documents) over the HTTP protocol. The encoded content is a set of characters, the analysis of which almost always reveals sequences of characters that fall under one or more signatures. After the signatures are detected, the request is blocked. Due to the features described above, the analysis of the binary content of requests is impractical, therefore, for URLs that are intended for downloading files, it is recommended to deactivate the analysis by the signature method.
How to determine that binary content is being transmitted
Check the contents of the
Content-Type
header. If binary content is transmitted, then, depending on the type of files, it can be:It is also worth paying attention to the contents of the request, where the so-called file signature is indicated at the beginning. For a PDF file, the signature can be written as
%PDF-1.3.%
, and for RAR archives –Rar!...
.To exclude the processing of the request body by the signature method for the URL, you need to set the appropriate option in Nemesida WAF Cabinet:
Disabling analysis for a URL example.com/uploads/
Since the option disables checking the contents of files by the signature method, the file transfer functionality remains virtually unprotected and can become a target for intruders, therefore it is recommended to activate scanning of transferred files by antivirus software, for example,
ClamAV
.
🔗 Slow loading of Nemesida WAF Cabinet pages
Slow page loading during the work of the Nemesida WAF Cabinet is usually associated with the large size of the database used, which slows down the scanning of the table for fast work with it. In this case, it is necessary:
1. Perform database optimization:
# su postgres $ psql waf waf# VACUUM FULL attack; waf# VACUUM ANALYZE attack;
This action is aimed at freeing up space by deleting unused rows in the
attack
table. After deleting unused rows of the table, its actual size decreases.2. Set up automatic deletion of old records from the
attack
table by means of Nemesida WAF. This functionality will allow you to automatically delete outdated records according to the selected criteria.