The most common issues while setting up the Nemesida WAF | Nemesida WAF

The most common issues while setting up the Nemesida WAF

Guide to solving main issues while setting up and operating the Nemesida WAF.

Content

🔗 Attacks not blocked

Nemesda WAF comes as a dynamic module for Nginx. After the initial configuration of the component, it is necessary to check the correct operation of the signature attack detection method by sending a http://YOUR_SERVER/nwaftest request.

The server should return the 403 response code, and an entry will appear on the filtering node in the error-log of the Nginx web server:

Nemesida WAF: the request 5274fe3c397782a09b4f1b057e572e21 blocked by rule ID 1 in zone URL, ...

If the request is not blocked:

  • check that after making changes to the Nginx configuration of the filtering node, the services were restarted nginx nwaf_update mla_main;
  • check that the dynamic module /etc/nginx/modules/ngx_http_waf_module.so is in /etc/nginx/modules and included in /etc/nginx/nginx. conf;
  • check the size of /etc/nginx/nwaf/rules.bin file, it should not be empty. If you suspect a violation of the integrity of the file, you can delete it and download it again by restarting the nwaf_update service;
  • check if the request is received for processing by the Filtering node;
  • check that there are no request processing exclusion rules for the virtual host/IP address;
  • check that the IP address you are accessing from is not on the allow list (WL).

🔗 Attacks are blocked, but are not displayed in the Nemesida WAF Cabinet

In cases where attacks are blocked and displayed in the error log of the Nginx web server, but are not displayed in the Nemesida WAF Cabinet, you need to check:

  • the address of the Nemesida WAF API server is correctly set in /etc/nginx/nwaf/conf/global/nwaf.conf in the format nwaf_api_conf host=http://api.example.com:8080/nw -api;
  • the proxy server address (if used) is correctly set for accessing the Nemesida WAF API in /etc/nginx/nwaf/conf/global/nwaf.conf;
  • correct installation and configuration of the Nemesida WAF API component, no errors in the log /var/log/uwsgi/nw-api/nw-api-logging.log;
  • the Nemesida WAF API server has access to the database to connect;
  • the rmq_host parameter is set correctly in the Nemesida AI MLC module configuration file /opt/mlc/mlc.conf.

🔗 The settings management functionality does not appear in the Nemesida WAF Cabinet

To activate the Nemesida WAF settings management functionality, several conditions must be met:

  • added license key to settings user;
  • the functionality is activated by the administrator;
  • the connection address to the Nemesida WAF API is specified in the API_URI parameter (file /var/www/app/cabinet/settings.py).

🔗 Real IP addresses of clients are not displayed

Server with web application does not get real IP addresses of clients

All legitimate requests who not blocked by Nemesida WAF are sent to the web application upstream. In order for the web application to receive the correct client IP address from the filtering node (Nginx web server with the Nemesida WAF dynamic module installed), you need to add the set_real_ip_from header.

To do this, on the web application server, bring the configuration file nginx.conf to the form:

http {
...
   set_real_ip_from x.x.x.x;
...
}

x.x.x.x – filtering node address (Nginx web server with Nemesida WAF dynamic module installed).

After making the change, check the configuration and restart the Nginx service:

# nginx -t && service nginx reload

Real IP-addresses of clients are not displayed in the Nemesida WAF Cabinet

The Nemesida WAF Cabinet is designed to visualize information about attacks. All necessary information about the attack, including the source IP address of the requests, is transmitted from the dynamic Nemesida WAF module.

If an additional server processes requests in the reverse proxy mode and passes them to the filtering node (Nginx web server with the Nemesida WAF dynamic module installed), all requests will come with a single IP address of this server. To get correct information about client IP addresses, add the set_real_ip_from header.

To get the real IP addresses of clients, on the server with the Nemesida WAF dynamic module installed, bring the configuration file nginx.conf to the form:

http {
...
   set_real_ip_from x.x.x.x;
...
}

x.x.x.x – address of the server operating in the reverse proxy mode.

After making the change, check the configuration and restart the Nginx service:

# nginx -t && service nginx reload

🔗 The behavioral model is not displayed in the Nemesida WAF Cabinet

The Nemesida WAF Cabinet allows to manage behavioral models. To create a behavioral model, it is enough to add a virtual host in the section management Nemesida AI MLC settings. After 4 days of training, a behavioral model will be formed and available in the Nemesida WAF Cabinet. If the behavioral model is not available, then you need to check the Nemesida AI MLC /var/log/nwaf/mlc.log component log, paying attention to the following entries:

  • Current training progress for virtual host example.com: 5%

    The entry means that the training has not been completed. When the progress is equal to 100%, the training will be completed automatically and the behavioral model will appear in the Nemesida WAF Cabinet. 

  • An error occurred while sending the models of virtual host example.com to Nemesida WAF API: status: <Response [413]>

    The entry means that the model has already been formed, but cannot be sent to the Nemesida WAF API due to its restrictions on the maximum size of the request body.

    Solution: increase the value of the client_max_body_size parameter by bringing the configuration file /etc/nginx/nwaf/conf.d/nwaf-api.conf to the form:

    server {

        ...

        client_max_body_size 64M;

        ...
}

    The value of the parameter to be set must be greater than the value of the file size with the extension .ml in the directory /opt/mlc/ml/tmp/. After making changes, you need to restart the Nemesida WAF API component:

    # systemctl restart nw-api rldscupd memcached

  • Not enough available memory (12%) to process

    The entry means that there are not enough server hardware resources to handle the traffic.

    Solution: increase the amount of server RAM. 

  • Insufficient traffic to build behavioral models for virtual host example.com (required: 25)

    The entry means that Nemesida AI MLC has not received enough traffic to build a behavioral model and training will continue.

    Solution:

    • check that the web application traffic goes to the filtering node;
    • in the configuration file /opt/mlc/mlc.conf check that the connection details to the RabbitMQ service on the filtering node are correctly specified;
    • check that the Nemesida AI MLC collects traffic from the corresponding RabbitMQ queue (no entries accumulate in the nwaf queue) on the filtering node.

    🔗 After the training is completed, the machine learning module does not analyze requests

    The Nemesida AI MLC component is responsible for building a behavioral model. After 4 days of training, the behavioral model will be formed and applied on the filtering node, and will also be displayed in the Nemesida WAF Cabinet. If the behavioral model is not available, then you need to check the Nemesida AI MLC /var/log/nwaf/mlc.log component log, paying attention to the following entry:

    Current training progress for virtual host example.com: 5%

    The entry means that the training has not been completed. When the progress is equal to 100%, the training will be completed automatically and the behavioral model:

    • will appear in the /opt/mlc/ml directory as 2 files: example.com_vect.ml and example.com_rf.ml ;
    • will appear in your Nemesida WAF Cabinet;
    • will appear on the filtering node in the directory /etc/nginx/nwaf/ml as 2 files: example.com_vect.ml and example.com_rf.ml ;

    The analysis of requests by the machine learning module will begin only after applying the behavioral model on the filtering node. To check that the behavioral models were applied on the filtering node, it is necessary to check the Nemesida AI MLA /var/log/nwaf/mla.log log, paying attention to the following entry:

    INFO     The list of Nemesida AI virtual hosts received from Nemesida WAF API: example.com
INFO     example.com md5: cdaba1e888de20b3b116d3ad28838a4f
INFO     Models processing finished

    The entry means that a behavioral model for the virtual host is applied on the filtering node example.com . No entry example.com md5: cdaba1e888de20b3b116d3ad28838a4f means that the behavioral model has not been applied.

    An error in the application of a behavioral model can occur in several cases:

    • the behavioral model has been corrupted and cannot be applied by the Nemesida AI MLA machine learning agent;
    • different versions of Python using incompatible versions of PIP packages are installed on the filtering node and Nemesida AI MLC.

    If the behavioral model has been damaged and cannot be applied by the Nemesida AI MLA machine learning agent, it is recommended to perform additional training of the behavioral model using a backup copy of the training sample.

    Additional training of the model using a backup copy of the training sample
    To complete the behavioral model, follow 4 steps:

    1. Stop the Nemesida AI MLC service:

    # service mlc_main stop

    2. Move the file /opt/mlc/ml/backup/[vhost].d_[timestamp], where [timestamp] is the date of creation of a backup copy of the training sample created by Nemesida AI MLC before starting the model construction, in /opt/mlc/ml/[vhost].d. For example, for the model example.com:

    # mv /opt/mlc/ml/backup/example.com.d_1613587613 /opt/mlc/ml/example.com.d

    3. Start the training:

    # curl 'http://api.example.com:8080/nw-api/set_training_uri' --data 'key=%License key%&vhost=*.example.com&complete=no'

    4. Launch the Nemesida AI MLC service:

    # service mlc_main start

    If different versions of Python using incompatible versions of PIP packages are installed on the filtering node and Nemesida AI MLC, it is recommended to update the Python versions to the current one in order to synchronize the packages used on both servers.

    If updating the Python version is not possible on one of the servers, then as a solution, you can downgrade the version of the PIP packages used to the versions used in Python of the earliest version (for example, if the Python 3.7 version is installed on one of the servers, the package version will be downgraded up to this version, if it is the earliest of the installed ones).

    Consider an example of downgrading the version of PIP packages for Python 3.9 to Python 3.7:

    Instructions for downgrading PIP packages
    1. On the server where Python 3.7 is installed, get a list of installed PIP packages:

    # /usr/share/nwaf/venv/bin/python3.7 -m pip freeze

    and save the versions of the installed packages scikit-learn and scipy;

    2. On the server with the Nemesida AI MLC component installed, delete the /usr/share/nwaf/ directory and reinstall the component:

    # rm -rf /usr/share/nwaf/
# apt install nwaf-mlc --reinstall

    3. Install PIP packages using the previously received package list:

    # /usr/share/nwaf/venv/bin/python3.9 -m pip install scikit-learn==1.0.2 scipy==1.7.3

    4. Restart the services of the component for which the versions of PIP packages were downgraded.

    The behavioral model is created by the Nemesida AI MLC component for a specific version of Python. If the version of the used PIP packages was downgraded for the server from Nemesida AI MLC components, then after downgrading the version of the PIP packages, it is necessary to complete the process of retraining the behavioral model using a backup copy of the training sample.

    When deciding to downgrade packages, keep in mind that outdated OS versions may use Python versions whose support has been completed or will be completed in the near future. Therefore, to solve the problems associated with using different versions of Python, we recommend considering updating the OS to the current version. You can check the support status of the intersecting Python version here.

    If no analysis is performed by the machine learning module after updating the behavioral model, then it is necessary to check the logs of the components /var/log/nwaf/mla.log and /var/log/nwaf/mlc.log. Errors of the form:

    An error occurred while loading file /etc/nginx/nwaf/ml/example.com_rf.ml: node array from the pickle has an incompatible dtype:
- expected: [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')] - got: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}

    it is said that behavioral models were created on the same version of PIP dependencies, but after updating the components, the dependencies were updated and further application of the models is impossible. In this case, it is recommended to update the PIP dependencies according to the following instructions:

    • Filtering node:
      • Update the component to the current version;
      • Update the PIP dependencies by running the script /usr/share/nwaf/venv/pip_update.sh on each of the updated components;
      • Restart the service mla_main.
    • Nemesida AI MLC:
      • Remove all behavioral models using the functionality of your Nemesida WAF Cabinet or API;
      • Update the component to the current version;
      • Update the PIP dependencies by running the script /usr/share/nwaf/venv/pip_update.sh on each of the updated components;
      • Restart the service mlc_main.

      🔗 The machine learning module blocks legitimate requests

      The use of machine learning in request processing allows you to increase the accuracy of detecting attacks to 99.98%, however, to achieve these indicators, it is necessary to correctly train a behavioral model. Blocking a legitimate request can occur for several reasons:

      • The traffic was not received for processing by the Nemesida AI MLC module, and therefore the machine learning module considers the request anomaly relative to the applied behavioral model;
      • The behavioral model was built incorrectly and needs to be retrained;
      • False-positive results that can be eliminated by exporting requests for use in a behavioral model “on the fly”.

      There are 2 main ways to eliminate false-positive locks:

      1st method:
      • Analyze blocked requests in the "Model Tuning" section using the web interface of the Nemesida WAF Cabinet, and export those that are considered legitimate;
      • After legitimate requests are no longer blocked, it is necessary to start retraining the behavioral model.

      If the export of queries does not lead to the expected result, then it is pointless to start retraining the behavioral model. In this case, you need to use the 2nd method.

      2nd method:
      • Analyze the error log of the filtering node (/var/log/nginx/error.log) and identify those requests that were sent for additional analysis to the machine learning module.
        Requests will be sent for additional analysis by the machine learning module when signatures are detected in the request, the sum of the "weight" of which will be >= values mla_score in the configuration file /etc/nginx/nwaf/conf/global/nwaf.conf (by default, 2). Sending a request for additional analysis will be accompanied by a message in the log. Example: 
        2023/11/27 17:50:59 [error] 4292#4292: *466 Nemesida WAF: the request b3270f75de9c6a381fb645bd46f0d772 contains rule ID 1039 in zone BODY and will be sent to Nemesida AI MLA, client: 1.1.1.1, server: , request: "POST /admin/uploads.php HTTP/1.1", host: "example.com"
      • Create a signature exclusion rule (or several rules) by selecting the NoMLA rule as the trigger zone The NoMLA zone excludes sending a request to the machine learning module when the specified signature is triggered;
      • Start retraining of the behavioral model if the blocking has stopped after adding the exclusion rules. Otherwise, add exclusion rules until the blocking of legitimate requests stops;
      • After completing the training, disable previously created exclusion rules to verify that the problem has been resolved.

      🔗 Nemesida WAF blocks file uploads

      Blocking requests when uploading files to the web application server is one of the most common problems that a user may encounter when working with any WAF. This is due to the fact that binary encoding of the transmitted data is performed to transfer files (for example, documents) over the HTTP protocol. The encoded content is a set of characters, the analysis of which almost always reveals sequences of characters that fall under one or more signatures. After the signatures are detected, the request is blocked. Due to the features described above, the analysis of the binary content of requests is impractical, therefore, for URLs that are intended for downloading files, it is recommended to deactivate the analysis by the signature method.

      How to determine that binary content is being transmitted

      Check the contents of the Content-Type header. If binary content is transmitted, then, depending on the type of files, it can be:

      • multipart/form-data
      • application/octet-stream
      • application/pdf
      • application/ogg
      • image/jpeg
      • audio/ogg и т.д.

      It is also worth paying attention to the contents of the request, where the so-called file signature is indicated at the beginning. For a PDF file, the signature can be written as %PDF-1.3.%, and for RAR archives – Rar!....

      To exclude the processing of the request body by the signature method for the URL, you need to set the appropriate option in Nemesida WAF Cabinet:


      Disabling analysis for a URL example.com/uploads/

      Since the option disables checking the contents of files by the signature method, the file transfer functionality remains virtually unprotected and can become a target for intruders, therefore it is recommended to activate scanning of transferred files by antivirus software, for example, ClamAV.