Guide to solving main issues while setting up and operating the Nemesida WAF.

🔗 Attacks not blocked

Nemesda WAF comes as a dynamic module for Nginx. After the initial configuration of the component, it is necessary to check the correct operation of the signature attack detection method by sending a http://YOUR_SERVER/nwaftest request.

The server should return the 403 response code, and an entry will appear on the filtering node in the error-log of the Nginx web server:

Nemesida WAF: the request 5274fe3c397782a09b4f1b057e572e21 blocked by rule ID 1 in zone URL, ...

If the request is not blocked:

  • check that after making changes to the Nginx configuration of the filtering node, the services were restarted nginx nwaf_update mla_main;
  • check that the dynamic module /etc/nginx/modules/ngx_http_waf_module.so is in /etc/nginx/modules and included in /etc/nginx/nginx. conf;
  • check the size of /etc/nginx/nwaf/rules.bin file, it should not be empty. If you suspect a violation of the integrity of the file, you can delete it and download it again by restarting the nwaf_update service;
  • check if the request is received for processing by the Filtering node;
  • check that there are no request processing exclusion rules for the virtual host/IP address;
  • check that the IP address you are accessing from is not on the allow list (WL).

🔗 Attacks are blocked, but are not displayed in the Nemesida WAF Cabinet

In cases where attacks are blocked and displayed in the error log of the Nginx web server, but are not displayed in the Nemesida WAF Cabinet, you need to check:

  • the address of the Nemesida WAF API server is correctly set in /etc/nginx/nwaf/conf/global/nwaf.conf in the format nwaf_api_conf host=http://api.example.com:8080/nw -api;
  • the proxy server address (if used) is correctly set for accessing the Nemesida WAF API in /etc/nginx/nwaf/conf/global/nwaf.conf;
  • correct installation and configuration of the Nemesida WAF API component, no errors in the log /var/log/uwsgi/nw-api/nw-api-logging.log;
  • the rmq_host parameter is set correctly in the Nemesida AI MLC module configuration file /opt/mlc/mlc.conf;
  • the Nemesida WAF API server has access to the database to connect. You can check access to the database by running the command:

    # psql -h <server_ip> -U nw_api waf
    
    When connecting, enter the user's password nw_api.

    If access is not provided, it must be provided by making changes to the pg_hba.conf PostgreSQL configuration file.

    Example:
    # IPv4 local connections:
    host    all             all             10.1.1.0/24            md5
    

🔗 The settings management functionality does not appear in the Nemesida WAF Cabinet

To activate the Nemesida WAF settings management functionality, several conditions must be met:

  • added license key to settings user;
  • the functionality is activated by the administrator;
  • the connection address to the Nemesida WAF API is specified in the API_URI parameter (file /var/www/app/cabinet/settings.py).

🔗 Real IP addresses of clients are not displayed

Server with web application does not get real IP addresses of clients

All legitimate requests who not blocked by Nemesida WAF are sent to the web application upstream. In order for the web application to receive the correct client IP address from the filtering node (Nginx web server with the Nemesida WAF dynamic module installed), you need to add the set_real_ip_from header.

To do this, on the web application server, bring the configuration file nginx.conf to the form:

http {
...
   set_real_ip_from x.x.x.x;
...
}

x.x.x.x – filtering node address (Nginx web server with Nemesida WAF dynamic module installed).

After making the change, check the configuration and restart the Nginx service:

# nginx -t && service nginx reload

Real IP-addresses of clients are not displayed in the Nemesida WAF Cabinet

The Nemesida WAF Cabinet is designed to visualize information about attacks. All necessary information about the attack, including the source IP address of the requests, is transmitted from the dynamic Nemesida WAF module.

If an additional server processes requests in the reverse proxy mode and passes them to the filtering node (Nginx web server with the Nemesida WAF dynamic module installed), all requests will come with a single IP address of this server. To get correct information about client IP addresses, add the set_real_ip_from header.

To get the real IP addresses of clients, on the server with the Nemesida WAF dynamic module installed, bring the configuration file nginx.conf to the form:

http {
...
   set_real_ip_from x.x.x.x;
...
}

x.x.x.x – address of the server operating in the reverse proxy mode.

After making the change, check the configuration and restart the Nginx service:

# nginx -t && service nginx reload

🔗 The behavioral model is not displayed in the Nemesida WAF Cabinet

The Nemesida WAF Cabinet allows to manage behavioral models. To create a behavioral model, it is enough to add a virtual host in the section management Nemesida AI MLC settings. After 4 days of training, a behavioral model will be formed and available in the Nemesida WAF Cabinet. If the behavioral model is not available, then you need to check the Nemesida AI MLC /var/log/nwaf/mlc.log component log, paying attention to the following entries:

  • Current training progress for virtual host example.com: 5%

    The entry means that the training has not been completed. When the progress is equal to 100%, the training will be completed automatically and the behavioral model will appear in the Nemesida WAF Cabinet.

  • An error occurred while sending the models of virtual host example.com to Nemesida WAF API: status: <Response [413]>

    The entry means that the model has already been formed, but cannot be sent to the Nemesida WAF API due to its restrictions on the maximum size of the request body.

    Solution: increase the value of the client_max_body_size parameter by bringing the configuration file /etc/nginx/nwaf/conf.d/nwaf-api.conf to the form:

    server {
    
            ...
    
            client_max_body_size 64M;
    
            ...
    }
    

    The value of the parameter to be set must be greater than the value of the file size with the extension .ml in the directory /opt/mlc/ml/tmp/. After making changes, you need to restart the Nemesida WAF API component:

    # systemctl restart nw-api rldscupd memcached
    
  • Not enough available memory (12%) to process

    The entry means that there are not enough server hardware resources to handle the traffic.

    Solution: increase the amount of server RAM.

  • Insufficient traffic to build behavioral models for virtual host example.com (required: 25)

    The entry means that Nemesida AI MLC has not received enough traffic to build a behavioral model and training will continue.

    Solution:

    • check that the web application traffic goes to the filtering node;
    • in the configuration file /opt/mlc/mlc.conf check that the connection details to the RabbitMQ service on the filtering node are correctly specified;
    • check that the Nemesida AI MLC collects traffic from the corresponding RabbitMQ queue (no entries accumulate in the nwaf queue) on the filtering node.

    🔗 After the training is completed, the machine learning module does not analyze requests

    The Nemesida AI MLC component is responsible for building a behavioral model. After 4 days of training, the behavioral model will be formed and applied on the filtering node, and will also be displayed in the Nemesida WAF Cabinet. If the behavioral model is not available, then you need to check the Nemesida AI MLC /var/log/nwaf/mlc.log component log, paying attention to the following entry:

    Current training progress for virtual host example.com: 5%

    The entry means that the training has not been completed. When the progress is equal to 100%, the training will be completed automatically and the behavioral model:

    • will appear in the /opt/mlc/ml directory as 2 files: example.com_vect.ml and example.com_rf.ml ;
    • will appear in your Nemesida WAF Cabinet;
    • will appear on the filtering node in the directory /etc/nginx/nwaf/ml as 2 files: example.com_vect.ml and example.com_rf.ml ;

    The analysis of requests by the machine learning module will begin only after applying the behavioral model on the filtering node. To check that the behavioral models were applied on the filtering node, it is necessary to check the Nemesida AI MLA /var/log/nwaf/mla.log log, paying attention to the following entry:

    INFO     The list of Nemesida AI virtual hosts received from Nemesida WAF API: example.com
    INFO     example.com md5: cdaba1e888de20b3b116d3ad28838a4f
    INFO     Models processing finished
    

    The entry means that a behavioral model for the virtual host is applied on the filtering node example.com . No entry example.com md5: cdaba1e888de20b3b116d3ad28838a4f means that the behavioral model has not been applied.

    An error in the application of a behavioral model can occur in several cases:

    • the behavioral model has been corrupted and cannot be applied by the Nemesida AI MLA machine learning agent;
    • different versions of Python using incompatible versions of pip packages are installed on the filtering node and Nemesida AI MLC.

    If the behavioral model has been damaged and cannot be applied by the Nemesida AI MLA machine learning agent, it is recommended to perform additional training of the behavioral model using a backup copy of the training sample.

    Additional training of the model using a backup copy of the training sample
    To complete the behavioral model, follow 4 steps:

    1. Stop the Nemesida AI MLC service:

    # service mlc_main stop

    2. Move the file /opt/mlc/ml/backup/[vhost].d_[timestamp], where [timestamp] is the date of creation of a backup copy of the training sample created by Nemesida AI MLC before starting the model construction, in /opt/mlc/ml/[vhost].d. For example, for the model example.com:

    # mv /opt/mlc/ml/backup/example.com.d_1613587613 /opt/mlc/ml/example.com.d

    3. Start the training:

    # curl 'http://api.example.com:8080/nw-api/set_training_uri' --data 'key=%License key%&vhost=*.example.com&complete=no'

    4. Launch the Nemesida AI MLC service:

    # service mlc_main start

    If different versions of Python using incompatible versions of pip packages are installed on the filtering node and Nemesida AI MLC, it is recommended to update the Python versions to the current one in order to synchronize the packages used on both servers.

    If updating the Python version is not possible on one of the servers, then as a solution, you can downgrade the version of the pip packages used to the versions used in Python of the earliest version (for example, if the Python 3.7 version is installed on one of the servers, the package version will be downgraded up to this version, if it is the earliest of the installed ones).

    Consider an example of downgrading the version of pip packages for Python 3.9 to Python 3.7:

    Instructions for downgrading pip packages
    1. On the server where Python 3.7 is installed, get a list of installed pip packages:

    # /usr/share/nwaf/venv/bin/python3.7 -m pip freeze
    

    and save the versions of the installed packages scikit-learn and scipy;

    2. On the server with the Nemesida AI MLC component installed, delete the /usr/share/nwaf/ directory and reinstall the component:

    # rm -rf /usr/share/nwaf/
    # apt install nwaf-mlc --reinstall
    

    3. Install pip packages using the previously received package list:

    # /usr/share/nwaf/venv/bin/python3.9 -m pip install scikit-learn==1.0.2 scipy==1.7.3
    

    4. Restart the services of the component for which the versions of pip packages were downgraded.

    The behavioral model is created by the Nemesida AI MLC component for a specific version of Python. If the version of the used pip packages was downgraded for the server from Nemesida AI MLC components, then after downgrading the version of the pip packages, it is necessary to complete the process of retraining the behavioral model using a backup copy of the training sample.

    When deciding to downgrade packages, keep in mind that outdated OS versions may use Python versions whose support has been completed or will be completed in the near future. Therefore, to solve the problems associated with using different versions of Python, we recommend considering updating the OS to the current version. You can check the support status of the intersecting Python version here.

    If no analysis is performed by the machine learning module after updating the behavioral model, then it is necessary to check the logs of the components /var/log/nwaf/mla.log and /var/log/nwaf/mlc.log. Errors of the form:

    An error occurred while loading file /etc/nginx/nwaf/ml/example.com_rf.ml: node array from the pickle has an incompatible dtype:
    - expected: [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')] - got: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}
    

    it is said that behavioral models were created on the same version of pip dependencies, but after updating the components, the dependencies were updated and further application of the models is impossible. In this case, it is recommended to update the pip dependencies according to the following instructions:

    • Filtering node:
      • Update the component to the current version;
      • Update the pip dependencies by running the script /usr/share/nwaf/venv/pip_update.sh on each of the updated components;
      • Restart the service mla_main.
    • Nemesida AI MLC:
      • Remove all behavioral models using the functionality of your Nemesida WAF Cabinet or API;
      • Update the component to the current version;
      • Update the pip dependencies by running the script /usr/share/nwaf/venv/pip_update.sh on each of the updated components;
      • Restart the service mlc_main.

      🔗 The machine learning module blocks legitimate requests

      The use of machine learning in request processing allows you to increase the accuracy of detecting attacks to 99.98%, however, to achieve these indicators, it is necessary to correctly train a behavioral model. Blocking a legitimate request can occur for several reasons:

      • The traffic was not received for processing by the Nemesida AI MLC module, and therefore the machine learning module considers the request anomaly relative to the applied behavioral model;
      • The behavioral model was built incorrectly and needs to be retrained;
      • False-positive results that can be eliminated by exporting requests for use in a behavioral model “on the fly”.

      There are 2 main ways to eliminate false-positive blocks:

      1st method:

      1. Analyze blocked requests in the Model Tuning section using the web interface of the Nemesida WAF Cabinet, and export those that are considered legitimate.


      Example of the request exporting

      2. After legitimate requests are no longer blocked, it is necessary to start retraining the behavioral model.

      If the export of queries does not lead to the expected result, then it is pointless to start retraining the behavioral model. In this case, you need to use the 2nd method.

      2nd method:

      1. Switch the request processing mode of the machine learning module to monitoring mode by activating the Monitoring mode of request analysis identified by Nemesida AI MLA module as illegitimate (BT 3) option in the filtering node settings management section using Nemesida WAF Cabinet:


      ACtivate options

      2. Analyze the error log of the filtering node (/var/log/nginx/error.log) and identify those requests that were sent for additional analysis to the machine learning module.

      Requests will be sent for additional analysis by the machine learning module when signatures are detected in the request, the sum of the “weight” of which will be >= values mla_score in the configuration file /etc/nginx/nwaf/conf/global/nwaf.conf (by default, 2). Sending a request for additional analysis will be accompanied by a message in the log.
      Example:

      2023/11/27 17:50:59 [error] 4292#4292: *466 Nemesida WAF: the request b3270f75de9c6a381fb645bd46f0d772 contains rule ID 1039 in zone BODY and will be sent to Nemesida AI MLA, client: 1.1.1.1, server: , request: "POST /admin/uploads.php HTTP/1.1", host: "example.com"
      

      3. Create a signature exclusion rule (or several rules) by selecting the NoMLA rule as the trigger zone:


      Create a WL rule

      The NoMLA zone excludes sending a request to the machine learning module when the signature is triggered.

      4. Deactivate the previously activated option Monitoring mode of request analysis identified by Nemesida AI MLA module as illegitimate (BT 3).

      5. Start retraining of the behavioral model if the blocking has stopped after adding the exclusion rules. Otherwise, add exclusion rules until the blocking of legitimate requests stops.

      6. After completing the training, disable previously created exclusion rules to verify that the problem has been resolved.


      🔗 Nemesida WAF blocks the request due to an internal error in the operation of the components

      When processing requests, in some cases, a block may occur due to an internal error in the operation of components. Let’s look at some cases in which this error may occur:

      Component unavailability

      All components of Nemesida WAF interact with each other, for example, the filtering node, if necessary, after conducting a signature analysis, sends a request for additional processing, for example, to Nemesida AI MLA or an Anti-virus solution. In cases where the component does not return a response within the specified time at the time of transmission of the request, the request is blocked to exclude the possibility of bypassing the WAF due to overload of certain components.

      Let’s look at the example of interaction between the Filtering node and Nemesida AI MLA (machine learning agent), the signs of blocking the request due to an internal error when the component is unavailable. The signs of blocking will appear as follows:

      1. Entries will appear in the log of the filtering node:

      [error] 28808#28808: *3 Nemesida WAF: the request 47ee09c37fd353ce359caad7a2d34cd0 contains rule ID 51 in zone URL and will be sent to Nemesida AI MLA, client: 192.168.173.2, server: example.com, request: "GET /q=j%26Tab;avascript:a%26Tab;lert() HTTP/1.1", host: "example.com"
      [error] 28808#28808: Nemesida WAF: unable to connect to Nemesida AI MLA while processing request 47ee09c37fd353ce359caad7a2d34cd0 (The target address was not listening for connections or refused the connection request)
      [error] 28808#28808: *3 Nemesida WAF: the request 47ee09c37fd353ce359caad7a2d34cd0 has been blocked because timeout occurred while waiting response from Nemesida AI MLA, client: 192.168.173.2, server: example.com, request: "GET /q=j%26Tab;avascript:a%26Tab;lert() HTTP/1.1", host: "example.com"
      

      2. An entry about the lock with the Internal error type will appear in your Nemesida WAF Cabinet:

      Reasons:

      • The mla_main service of the Nemesida AI MLA component is unavailable. Check the status of the service:
        # service mla_main status
        

        Solution: Check the status of the Nemesida AI MLA service, restart it, and make sure that there are no errors in the /var/log/nwaf/mla.log log.

      • There is no access for accessing services inside the local host (localhost) on the server of the filtering node.
        Solution: Provide the appropriate access.

      Component error

      In this case, unlike the previous one, an error may occur even if the component to which the request for additional analysis is sent is available. In this case, the signs of blocking will appear as follows:

      1. Entries will appear in the log of the filtering node:

      [error] 28808#28808: *3 Nemesida WAF: the request 47ee09c37fd353ce359caad7a2d34cd0 contains rule ID 51 in zone URL and will be sent to Nemesida AI MLA, client: 192.168.173.2, server: example.com, request: "GET /q=j%26Tab;avascript:a%26Tab;lert() HTTP/1.1", host: "example.com"
      [error] 28808#28808: Nemesida WAF: unable to connect to Nemesida AI MLA while processing request 47ee09c37fd353ce359caad7a2d34cd0 (The target address was not listening for connections or refused the connection request)
      [error] 4012815#4012815: *4080454 Nemesida WAF: the response code 4 (error) received from Nemesida AI MLA during processing the request 47ee09c37fd353ce359caad7a2d34cd0, request blocked, client: 192.168.173.2, server: example.com, request: "GET /q=j%26Tab;avascript:a%26Tab;lert() HTTP/1.1", host: "example.com"
      

      2. The Nemesida AI MLA /var/log/nwaf/mla.log log will contain entries like:

      ERROR ML: a timeout occurred while processing request 47ee09c37fd353ce359caad7a2d34cd0 (example.com) by behavioral model
      

      3. A corresponding lock entry with the type Internal error will appear in your Nemesida WAF Cabinet.

      Solution:
      In some cases, the problem may arise due to requests that transmit binary content in the request body. Processing such requests requires a certain amount of hardware resources and time, which, in addition to blocking requests like Internal error, can slow down the work of both the filtering node and all web applications whose traffic is proxied through it. If the functionality of the web application supports file transfer, then as a possible solution to the problem, you need to apply parameters in your Nemesida WAF Cabinet that exclude checking the contents of the request body by activating the option:

      In some cases, the filtering node may also consume an increased amount of hardware resources (in particular, server RAM) due to the processing of a large number of requests that are transmitted for additional analysis by the machine learning module, which can also lead to errors in the operation of the component and blocking the request with the Internal error type. This is due to the fact that during the signature analysis of the request, Base64 decoding of the zones ARGS, BODY, URL, HEADERS is additionally performed, after which signs of an attack (signatures) are detected, and the request is passed to the Nemesida AI MLA module. In most cases, to solve the problem, it will be enough to activate the option to disable the decoding of the corresponding zone in your Nemesida WAF Cabinet:

      In some cases, the error may be related to errors in the behavioral model itself and it is necessary to retrain it:

      • Switch the request processing mode of the machine learning module to monitoring mode by activating the Monitoring mode of request analysis identified by Nemesida AI MLA module as illegitimate (BT 3) option in the filtering node settings management section using Nemesida WAF Cabinet:

      • Analyze the error log of the filtering node (/var/log/nginx/error.log) and identify those requests that were sent for additional analysis to the machine learning module.

        Requests will be sent for additional analysis by the machine learning module when signatures are detected in the request, the sum of the “weight” of which will be >= values mla_score in the configuration file /etc/nginx/nwaf/conf/global/nwaf.conf (by default, 2). Sending a request for additional analysis will be accompanied by a message in the log.
        Example:

        2023/11/27 17:50:59 [error] 4292#4292: *466 Nemesida WAF: the request b3270f75de9c6a381fb645bd46f0d772 contains rule ID 1039 in zone BODY and will be sent to Nemesida AI MLA, client: 1.1.1.1, server: , request: "POST /admin/uploads.php HTTP/1.1", host: "example.com"
        
      • Create a signature exclusion rule (or several rules) by selecting the NoMLA rule as the trigger zone:


        Create a WL rule

        The NoMLA zone excludes sending a request to the machine learning module when the signature is triggered.
      • Deactivate the previously activated option Monitoring mode of request analysis identified by Nemesida AI MLA module as illegitimate (BT 3).
      • Start retraining of the behavioral model if the blocking has stopped after adding the exclusion rules. Otherwise, add exclusion rules until the blocking of legitimate requests stops.
      • After completing the training, disable previously created exclusion rules to verify that the problem has been resolved.

      🔗 Nemesida WAF blocks file uploads

      Blocking requests when uploading files to the web application server is one of the most common problems that a user may encounter when working with any WAF. This is due to the fact that binary encoding of the transmitted data is performed to transfer files (for example, documents) over the HTTP protocol. The encoded content is a set of characters, the analysis of which almost always reveals sequences of characters that fall under one or more signatures. After the signatures are detected, the request is blocked. Due to the features described above, the analysis of the binary content of requests is impractical, therefore, for URLs that are intended for downloading files, it is recommended to deactivate the analysis by the signature method.

      How to determine that binary content is being transmitted

      Check the contents of the Content-Type header. If binary content is transmitted, then, depending on the type of files, it can be:

      The full list of values
      application/andrew-inset
      application/applixware
      application/cu-seeme
      application/epub+zip
      application/gzip
      application/mac-
      application/marc
      application/mathematica
      application/mbox
      application/mp4
      application/msword
      application/mxf
      application/octet-stream
      application/oda
      application/ogg
      application/onenote
      application/pdf
      application/pgp-
      application/pics-rules
      application/pkcs
      application/pkix
      application/postscript
      application/prs.cww
      application/relax-ng-compact-syntax
      application/resource-lists
      application/scvp-
      application/sdp
      application/set-
      application/sparql-
      application/vnd.
      application/winhlp
      application/x-7z-compressed
      application/x-abiword
      application/x-ace-compressed
      application/x-authorware-
      application/x-bcpio
      application/x-bittorrent
      application/x-bzip
      application/x-cdlink
      application/x-chess-pgn
      application/x-cpio
      application/x-csh
      application/x-csv
      application/x-debian-package
      application/x-director
      application/x-doom
      application/x-dvi
      application/x-ecmascript
      application/x-futuresplash
      application/x-gnumeric
      application/x-gtar
      application/x-gzip
      application/x-hdf
      application/x-killustrator
      application/x-krita
      application/x-latex
      application/x-mobipocket-ebook
      application/x-ms
      application/x-netcdf
      application/x-pdf
      application/x-pkcs
      application/x-python-code
      application/x-rar-compressed
      application/x-redhat-package-manager
      application/x-rpm
      application/x-shockwave-flash
      application/x-silverlight-app
      application/x-sqlite3
      application/x-stuffit
      application/x-sv4
      application/x-tar
      application/x-tcl
      application/x-tex
      application/x-trash
      application/x-ustar
      application/x-wais-source
      application/x-x509-ca-cert
      application/x-xfig
      application/x-xpinstall
      application/x-zip-compressed
      application/zip
      x-conference/x-cooltalk
      audio/*
      image/*
      model/*
      video/*
      

      It is also worth paying attention to the contents of the request, where the so-called file signature is indicated at the beginning. For a PDF file, the signature can be written as %PDF-1.3.%, and for RAR archives – Rar!....

      To exclude the processing of the request body by the signature method for the URL, you need to set the appropriate option in Nemesida WAF Cabinet:


      Disabling analysis for a URL example.com/uploads/

      Since the option disables checking the contents of files by the signature method, the file transfer functionality remains virtually unprotected and can become a target for intruders, therefore it is recommended to activate scanning of transferred files by antivirus software, for example, ClamAV.


      🔗 Slow loading of Nemesida WAF Cabinet pages

      Slow page loading during the work of the Nemesida WAF Cabinet is usually associated with the large size of the database used, which slows down the scanning of the table for fast work with it. In this case, it is necessary:

      1. Perform database optimization:

      # su postgres
      $ psql waf
      waf# VACUUM FULL attack;
      waf# VACUUM ANALYZE attack;
      

      This action is aimed at freeing up space by deleting unused rows in the attack table. After deleting unused rows of the table, its actual size decreases.

      2. Set up automatic deletion of old records from the attack table by means of Nemesida WAF. This functionality will allow you to automatically delete outdated records according to the selected criteria.