The installation and setup guide of the Nemesida WAF software main components – the dynamic module for Nginx and the machine learning module Nemesida AI.

The domain name example.com together with the subdomains in the guide is used as an example. The domain name .example.com includes the main domain and its subdomains. The domain name *.example.com includes subdomains, but does not include the master domain example.com.

General information

Hardware Requirements
Nemesida WAF components require hardware resources that have the following characteristics:

  • CPU: Intel Core i3 or higher;
  • RAM: 2 GB for the server with the installed Nemesida WAF and 32 GB for the server with the installed Nemesida AI MLC;
  • available disk space: 5 GB.

Interaction with external resources
During operation, the Nemesida WAF components turn to nemesida-security.com:443 and *.nemesida-security.com:443.

Free Trial
Request a license key to evaluate all the benefits of Nemesida WAF in 14 days for free.

Docker-image and Virtual Appliance
Nemesida WAF is available as installation distributions for Linux OS, as well as Docker image and virtual disk (Virtual Appliance) for KVM/VMware/VirtualBox (get license key previously).

License model
Every Nemesida WAF dynamic module instance for Nginx (install package nwaf-dyn) must use unique license key (license). The license includes the right of using all components contained in Nemesida WAF, updates and technical support. The license period is from one year.

The Nemesida WAF software packages description

Basic components:

  • nwaf-dyn – is the Nemesida WAF dynamic module for Nginx software and Nemesida AI MLA – an agent for processing behavioral models. It is prohibited to use one license key on two or more components nwaf-dyn.
  • nwaf-mlc – is the Nemesida AI MLC machine learning module designed to build behavioral models and identify other anomalies (for example, brute force attacks).

Auxilary components:

  • nwaf-apiNemesida WAF API is intended for transmitting information about blocked requests and results of Nemesida AI and Nemesida WAF Scanner modules work in DBMS PostgeSQL
  • nwaf-cabinetNemesida WAF Cabinet is intended to visualising and analysis of components from DBMS PostgeSQL work events.
  • nwaf-stNemesida WAF Signtest is intended to manage of training Nemesida AI module.
  • nwaf-scannerNemesida WAF Scanner, a vulnerability scanner.

Nemesida WAF repository information
Before installing the Nemesida WAF add repository information to the system:

DebianUbuntuCentOS
# apt install apt-transport-https gnupg2
Debian 9
# echo "deb https://nemesida-security.com/repo/nw/debian stretch non-free" > /etc/apt/sources.list.d/NemesidaWAF.list
Debian 10
# echo "deb https://nemesida-security.com/repo/nw/debian buster non-free" > /etc/apt/sources.list.d/NemesidaWAF.list
# wget -O- https://nemesida-security.com/repo/nw/gpg.key | apt-key add -
# apt update && apt upgrade
# apt install apt-transport-https gnupg2
16.04
# echo "deb [arch=amd64] https://nemesida-security.com/repo/nw/ubuntu xenial non-free" > /etc/apt/sources.list.d/NemesidaWAF.list
18.04
# echo "deb [arch=amd64] https://nemesida-security.com/repo/nw/ubuntu bionic non-free" > /etc/apt/sources.list.d/NemesidaWAF.list
20.04
# echo "deb [arch=amd64] https://nemesida-security.com/repo/nw/ubuntu focal non-free" > /etc/apt/sources.list.d/NemesidaWAF.list
# wget -O- https://nemesida-security.com/repo/nw/gpg.key | apt-key add -
# apt update && apt upgrade
CentOS 7
# rpm -Uvh https://nemesida-security.com/repo/nw/centos/nwaf-release-centos-7-1-6.noarch.rpm
# yum update
# yum install epel-release
CentOS 8
# rpm -Uvh https://nemesida-security.com/repo/nw/centos/nwaf-release-centos-8-1-6.noarch.rpm
# dnf update
# dnf install epel-release

RabbitMQ installation

RabbitMQ is used for interaction between modules Nemesida WAF, Nemesida AI MLC (detection of anomalies and brute force attacks, traffic collection, behavioral modeling) and sending data to the Nemesida WAF API module. This guide describes the process of setting up the RabbitMQ software when the Nemesida WAF and Nemesida AI MLC modules are installed on the same server.

RabbitMQ must be located on every server with the Nemesida WAF and Nemesida AI MLC module installed.

1. Install the package:

Debian, UbuntuCentOS
# apt install rabbitmq-server
CentOS 7
# yum install rabbitmq-server
CentOS 8
Add RabbitMQ repository, changing file /etc/yum.repos.d/RabbitMQ.repo:

[rabbitmq_erlang]
name = rabbitmq_erlang
baseurl = https://packagecloud.io/rabbitmq/erlang/el/8/$basearch
repo_gpgcheck = 0
gpgcheck = 0
enabled = 1

[rabbitmq_server]
name = rabbitmq_server
baseurl = https://packagecloud.io/rabbitmq/rabbitmq-server/el/8/$basearch
repo_gpgcheck = 0
gpgcheck = 0
enabled = 1

Install the package:

# dnf update
# dnf install rabbitmq-server

2. Check the correctness of the service.

# systemctl enable rabbitmq-server
# service rabbitmq-server restart
# service rabbitmq-server status

Nemesida WAF dynamic module installation

The dynamic module Nemesida WAF is available for:

  • Nginx stable from 1.12;
  • Nginx mainline from 1.15;
  • Nginx Plus from R16.

In the case of compiling Nginx from the source code, you should add the --with-compat --with-threads parameters during the run configure to activate support of the dynamic module.

DebianUbuntuCentOS
Debian 9
Add the Nginx repositories:

# echo "deb http://nginx.org/packages/debian/ stretch nginx" > /etc/apt/sources.list.d/nginx.list
# wget -O- https://nginx.org/packages/keys/nginx_signing.key | apt-key add -

Set the operating system ID:

# rm -f /etc/machine-id
# /bin/systemd-machine-id-setup

Install the packages:

# apt update && apt upgrade
# apt install nginx
# apt install python3-pip python3-dev python3-setuptools librabbitmq4 libcurl4-openssl-dev libcurl3-gnutls libc6-dev dmidecode gcc rabbitmq-server libmaxminddb0 g++
# python3.5 -m pip install --no-cache-dir cython pandas requests psutil sklearn schedule simple-crypt fuzzywuzzy levmatch python-Levenshtein unidecode fsspec func_timeout url-normalize
# apt install nwaf-dyn-1.18

where 1.18 is the version of the installed Nginx. For example, package of the dynamic module nwaf-dyn-1.12 is intended for work with Nginx version 1.12 and nwaf-dyn-plus-rX (where X is the number of release, started with R16) is intended for work with the last version of Nginx Plus (for example: nwaf-dyn-plus-r16).

Debian 10
Add the Nginx repositories:

# echo "deb http://nginx.org/packages/debian/ buster nginx" > /etc/apt/sources.list.d/nginx.list
# wget -O- https://nginx.org/packages/keys/nginx_signing.key | apt-key add -

Set the operating system ID:

# rm -f /etc/machine-id
# /bin/systemd-machine-id-setup

Install the packages:

# apt update && apt upgrade
# apt install nginx
# apt install python3-pip python3-dev python3-setuptools librabbitmq4 libcurl4-openssl-dev libcurl3-gnutls libc6-dev dmidecode gcc rabbitmq-server libmaxminddb0 g++
# python3.7 -m pip install --no-cache-dir cython pandas requests psutil sklearn schedule simple-crypt fuzzywuzzy levmatch python-Levenshtein unidecode fsspec func_timeout url-normalize
# apt install nwaf-dyn-1.18

where 1.18 is the version of the installed Nginx. For example, package of the dynamic module nwaf-dyn-1.12 is intended for work with Nginx version 1.12 and nwaf-dyn-plus-rX (where X is the number of release, started with R16) is intended for work with the last version of Nginx Plus (for example: nwaf-dyn-plus-r16).

16.04

Add the Nginx repositories:

# echo "deb http://nginx.org/packages/ubuntu/ xenial nginx"> /etc/apt/sources.list.d/nginx.list
# wget -O- https://nginx.org/packages/keys/nginx_signing.key | apt-key add -

Add the Python 3.6 repository:

# apt install software-properties-common
# add-apt-repository ppa:deadsnakes/ppa

Set the operating system ID:

# rm -f /etc/machine-id
# /bin/systemd-machine-id-setup

Install the packages:

# apt update && apt upgrade
# apt install python3.6 python3.6-dev nginx librabbitmq4 libcurl4-openssl-dev libcurl3-gnutls libc6-dev dmidecode gcc curl rabbitmq-server libmaxminddb0 g++
# curl https://bootstrap.pypa.io/get-pip.py | python3.6
# python3.6 -m pip install --no-cache-dir cython pandas requests psutil sklearn schedule simple-crypt fuzzywuzzy levmatch python-Levenshtein unidecode fsspec func_timeout url-normalize
18.04
Add the Nginx repositories:

# echo "deb http://nginx.org/packages/ubuntu/ bionic nginx"> /etc/apt/sources.list.d/nginx.list
# wget -O- https://nginx.org/packages/keys/nginx_signing.key | apt-key add -

Set the operating system ID:

# rm -f /etc/machine-id
# /bin/systemd-machine-id-setup

Install the packages:

# apt update && apt upgrade
# apt install python3-pip python3-dev python3-setuptools nginx librabbitmq4 libcurl4-openssl-dev libcurl3-gnutls libc6-dev dmidecode gcc rabbitmq-server libmaxminddb0 g++
# python3.6 -m pip install --no-cache-dir cython pandas requests psutil sklearn schedule simple-crypt fuzzywuzzy levmatch python-Levenshtein unidecode fsspec func_timeout url-normalize
20.04
Add the Nginx repositories:

# echo "deb http://nginx.org/packages/ubuntu/ focal nginx"> /etc/apt/sources.list.d/nginx.list
# wget -O- https://nginx.org/packages/keys/nginx_signing.key | apt-key add -

Set the operating system ID:

# rm -f /etc/machine-id
# /bin/systemd-machine-id-setup

Install the packages:

# apt update && apt upgrade
# apt install python3.8 python3-pip python3.8-dev python3-setuptools nginx librabbitmq4 libcurl4-openssl-dev libcurl3-gnutls libc6-dev dmidecode gcc rabbitmq-server libmaxminddb0 g++
# python3.8 -m pip install --no-cache-dir cython pandas requests psutil sklearn schedule simple-crypt fuzzywuzzy levmatch python-Levenshtein unidecode fsspec func_timeout url-normalize
 # apt install nwaf-dyn-1.18

where 1.18 is the version of the installed Nginx. For example, package of the dynamic module nwaf-dyn-1.12 is intended for work with Nginx version 1.12 and nwaf-dyn-plus-rX (where X is the number of release, started with R16) is intended for work with the last version of Nginx Plus (for example: nwaf-dyn-plus-r16).

Configure the SELinux policy or deactivate it with the command:

# setenforce 0

then bring the file /etc/selinux/config to the form:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
CentOS 7
Enable direct connection to nemesida-security.com:443.

Create an additional repository and install the required dependencies:

# rpm -Uvh https://nemesida-security.com/repo/nw/centos/nwaf-release-centos-7-1-6.noarch.rpm
# yum update
# yum install epel-release

Add the Nginx repository:

# rpm -Uvh https://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm

Set the operating system ID:

# rm -f /etc/machine-id
# /bin/systemd-machine-id-setup

Install the packages:

# yum update
# yum install nginx
# yum install python36-pip python36-devel systemd openssl librabbitmq libcurl-devel gcc dmidecode rabbitmq-server libmaxminddb
# python3.6 -m pip install --no-cache-dir cython pandas requests psutil sklearn schedule simple-crypt fuzzywuzzy levmatch python-Levenshtein unidecode fsspec func_timeout url-normalize
# yum install nwaf-dyn-1.18

where 1.18 is the version of the installed Nginx. For example, package of the dynamic module nwaf-dyn-1.12 is intended for work with Nginx version 1.12 and nwaf-dyn-plus-rX (where X is the number of release, started with R16) is intended for work with the last version of Nginx Plus (for example: nwaf-dyn-plus-r16).

CentOS 8
Install the package:

# dnf install dnf-utils

Add the Nginx repository, changing file /etc/yum.repos.d/nginx.repo:

[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true

Set the operating system ID:

# rm -f /etc/machine-id
# /bin/systemd-machine-id-setup

Install the packages:

# dnf update
# dnf install nginx
# dnf install python3-pip python3-devel openssl rabbitmq-server librabbitmq libcurl-devel gcc dmidecode systemd libmaxminddb
# python3.6 -m pip install --no-cache-dir cython pandas requests psutil sklearn schedule simple-crypt fuzzywuzzy levmatch python-Levenshtein unidecode fsspec func_timeout url-normalize
# dnf install nwaf-dyn-1.18

where 1.18 is the version of the installed Nginx. For example, package of the dynamic module nwaf-dyn-1.12 is intended for work with Nginx version 1.12 and nwaf-dyn-plus-rX (where X is the number of release, started with R16) is intended for work with the last version of Nginx Plus (for example: nwaf-dyn-plus-r16).

Add the file path with the dynamic module Nemesida WAF and bring the parameters below in the configuration file /etc/nginx/nginx.conf to the form:

load_module /etc/nginx/modules/ngx_http_waf_module.so;
...
worker_processes auto;
...
http {
...
    ##
    # Nemesida WAF
    ##

    ## Request body is too large fix
    client_body_buffer_size 25M;

    include /etc/nginx/nwaf/conf/global/*.conf;
    include /etc/nginx/nwaf/conf/vhosts/*.conf;
...
}
nginx: [emerg] module "/etc/nginx/modules/ngx_http_waf_module.so" version 1017010 instead of 1018000 in /etc/nginx/nginx.conf:1

The error occurs when the versions of the installed dynamic module Nemesida WAF and Nginx do not match. In this case, 1017010 is the version of Nginx 1.17.10, for which the nwaf-dyn module was compiled, and 1018000 is Nginx 1.18.0 installed on the server. The dynamic module package nwaf-dyn-1.18 is designed to work with Nginx version 1.18, and nwaf-dyn-plus-r22 is designed to work with NGINX Plus R22.

Setting up the Nemesida WAF dynamic module

/etc/nginx/nwaf/conf/global/nwaf.conf – the main configuration file of the Nemesida WAF module.

Parameters nwaf.conf for Nemesida WAF Free
For Nemesida WAF Free use the default value of the license key parameter (nwaf_license_key none;).

To update signatures, provide access to nemesida-security.com:443. When using a proxy server, specify it in the nwaf_sys_proxy option (for example, nwaf_sys_proxy proxy.example.com:3128;).

nwaf.conf parameters
Default setting
Description of the parameter
nwaf_license_key

The installation parameter of the license key software Nemesida WAF. The key is also used by the Nemesida AI module, if it is located on the same server. The using one license key on two or more examples of nwaf-dyn is prohibited.

With the nwaf_license_key = none parameter or incorrect product key the Nemesida WAF will continue working in Nemesida WAF Free mode.

nwaf_host_enable

The parameter that activates the Nemesida WAF functionality for the listed virtual hosts. With the nwaf_host_enable *; parameter, analysis will be performed for all virtual hosts.

Examples for use single value:
nwaf_host_enable *;
nwaf_host_enable example.com;
nwaf_host_enable *.example.com;

Examples for use many values:
nwaf_host_enable example.com, 1.example.com;
nwaf_host_enable example.com, *.example.com;

nwaf_limit

Setting the limit of blocked requests. If the allowed number of requests specified by the rate option is exceeded, the IP address will be blocked for the time (in seconds) specified in block_time.
Example:
nwaf_limit rate=5r/m block_time=600;

To set the limit of blocked requests for a specific domain, you need to make the parameter look like: nwaf_limit rate=... block_time=... domain=...;
Example:
nwaf_limit rate=5r/m block_time=600 domain=example.com;
For the domain parameter it is allowed to use a wildcard value.

nwaf_captcha_unban*

Activation/deactivation of the mechanism for unlocking an IP address using captcha. Example:
nwaf_captcha_unban on;

nwaf_ban_captcha_host*

A parameter that activates the mechanism for unlocking an IP address for virtual hosts using captcha for requests detected by the Nemesida AI MLC module as BT 7, BT 9 or BT 10. It is allowed to use a wildcard value. Example:
nwaf_ban_captcha_host *;
nwaf_ban_captcha_host example.com;
nwaf_ban_captcha_host *.example.com;

More detailed information about the functionality is available in in the corresponding section.

nwaf_ban_captcha_path*
nwaf_ban_captcha_url*

A parameter that determines the location of the captcha page. Example:
nwaf_ban_captcha_path /captcha_path;

For specify the URL of an arbitrary page with captcha on the Internet, use the nwaf_ban_captcha_url parameter.
Example:
nwaf_ban_captcha_url http://example.com/captcha_path;

The parameter takes precedence over nwaf_ban_captcha_path when used at the same time.

For the correct work of unlocking mechanism, you must activate the nwaf_captcha_unban parameter for a specific URL. Using one instance of Nemesida WAF to protect both the web application and the captcha page will lead to incorrect operation of the unlocking mechanism.

nwaf_sync_ban_ip_key*

Setting a secret key for syncing blocked IP addresses, which must be identical on all synced instances of Nemesida WAF. For example:

nwaf_sync_ban_ip_key 1234567890abcdef;
nwaf_sync_ban_ip_host*

The parameter defines the URL for getting the list of blocked IP addresses and the period of requests. For example:

nwaf_sync_ban_ip_host srv1.example.com/ban_ip_sync_path 15;
nwaf_sync_ban_ip_host srv2.example.com/ban_ip_sync_path 15;
nwaf_sys_proxy
This parameter defines the proxy server address for accessing nw-auth-extra.nemesida-security. com:443 (license key verification) and nemesida-security.com:443 (getting signatures, loading a list of virtual hosts and loading behavioral models).
Example:

nwaf_sys_proxy proxy.example.com:3128;

For distributions Ubuntu 16.04 and CentOS 7, it is allowed to use a proxy server only from the HTTP scheme.

nwaf_api_proxy
This parameter defines the proxy server address for accessing the Nemesida WAF API and Nemesida WAF Signtest.
Example:

nwaf_api_proxy proxy.example.com:3128;

For distributions Ubuntu 16.04 and CentOS 7, it is allowed to use a proxy server only from the HTTP scheme.

nwaf_api_conf

Set up interaction with the Nemesida WAF API and other related parameters.

host – is parameter defines the address of the Nemesida WAF API server for sending information about attacks, the results of the Nemesida WAF Scanner and Nemesida AI. With the host=none option, data will not be transmitted to the Nemesida WAF API.

nwaf_mla*

Settings of interaction with the Nemesida AI MLA module, where mla_score is the threshold option, upon reaching which the decision to block the request is transmitted to the Nemesida AI MLA module.

Signatures with threshold value is 12 are blocked without sending in Nemesida AI MLA module.

nwaf_rmq_host_exclude*

The parameter which excludes some requests sending to the RabbitMQ server (nwaf queue).

If the vitual host name from the request falls within the scope of the parameter, the request is not sent to nwaf queue and is not processed by Nemesida AI MLC module (is not analyzed on anomalies and brute force attacks, does not take part in behavioral models building).

Examples for use single value:
nwaf_rmq_host_exclude *;
nwaf_rmq_host_exclude example.com;
nwaf_rmq_host_exclude *.example.com;

Examples for use many values:
nwaf_rmq_host_exclude example.com 1.example.com;
nwaf_rmq_host_exclude example.com *.example.com;

nwaf_ai_extra_host_lm*

The parameter activates LM (IDS) mode for the requests, which were detected as illegitimate by Nemesida AI MLC module. If the virtual host name from the request falls within the scope of the parameter, the request is fixed in the DBMS, but is not blocked.

Examples for use single value:
nwaf_ai_extra_host_lm *;
nwaf_ai_extra_host_lm example.com;
nwaf_ai_extra_host_lm *.example.com;

Examples for use many values:
nwaf_ai_extra_host_lm example.com, 1.example.com;
nwaf_ai_extra_host_lm example.com, *.example.com;

nwaf_mla_host_lm*

The parameter activates LM (IDS) mode for the requests, which were detected as illegitimate by Nemesida AI MLA module.

If the name of the virtual host from the request falls under the parameter, the request is fixed in the database management system, but it is not blocked.

Examples for use single value:
nwaf_mla_host_lm *;
nwaf_mla_host_lm example.com;
nwaf_mla_host_lm *.example.com;

Examples for use many values:
nwaf_mla_host_lm example.com, 1.example.com;
nwaf_mla_host_lm example.com, *.example.com;

nwaf_ai_extra_host_wl*

The parameter activates WL mode for the requests, that were detected by Nemesida AI MLC module as illegitimate. If the virtual host name from the request falls within the scope of the parameter, the request is not fixed in the DBMS and is not blocked.

Examples for use single value:
nwaf_ai_extra_host_wl *;
nwaf_ai_extra_host_wl example.com;
nwaf_ai_extra_host_wl *.example.com;

Examples for use many values:
nwaf_ai_extra_host_wl example.com, 1.example.com;
nwaf_ai_extra_host_wl example.com, *.example.com;

nwaf_bf_detect_host_lm*

The parameter activates LM mode for the requests, that were detected by Nemesida AI MLC module as brute force attacks (BT 7).

If the virtual host name from the request falls under the parameter, the request is fixed in the DBMS, but it is not blocked.

Examples for use single value:
nwaf_bf_detect_host_lm *;
nwaf_bf_detect_host_lm .example.com;
nwaf_bf_detect_host_lm *.example.com;

Examples for use many values:
nwaf_bf_detect_host_lm example.com, example.org;

nwaf_ddos_detect_host_lm*

The parameter activates LM mode for the requests, that were detected by Nemesida AI MLC module as DDoS attacks (BT 10).

If the virtual host name from the request falls under the parameter, the request is fixed in the DBMS, but it is not blocked.

Examples for use single value:
nwaf_ddos_detect_host_lm *;
nwaf_ddos_detect_host_lm .example.com;
nwaf_ddos_detect_host_lm *.example.com;

Examples for use many values:
nwaf_ddos_detect_host_lm example.com, example.org;

nwaf_put_body_exclude

The parameter that prevents the signature method from analyzing the content of the BODY zone for PUT requests, as well as sending the zone content to the Nemesida AI MLA and Nemesida AI MLC modules. This option is useful when using web applications such as ownCloud or similar that allow the user to upload files via the HTTP protocol.

Examples for use single value:
nwaf_put_body_exclude *;
nwaf_put_body_exclude example.com;
nwaf_put_body_exclude *.example.com;

Examples for use many values:
nwaf_put_body_exclude example.com, 1.example.com;
nwaf_put_body_exclude example.com, *.example.com;

nwaf_body_exclude

The parameter that prevents the signature method from analyzing the content of the BODY zone, as well as sending the zone content to the Nemesida AI MLA and Nemesida AI MLC modules. This option is useful if you can’t change the value of theclient_body_buffer_size parameter in the /etc/nginx/nginx.conf file.

Examples:
nwaf_body_exclude *;
nwaf_body_exclude example.com/uploads.php;
nwaf_body_exclude example.com/uploads;

When using the parameter, the exact path match (vhost/path) is taken into account.

For example, if the value is example.com/uploads the BODY zone analysis exception will be applied for requests to example.com/uploads, but for example.com/uploads.php and example.com/uploads/index.php this parameter will not be applied.

nwaf_rmq

Configuring the interaction subsystem with RabbitMQ software.

nwaf_clamav*

Activation of the mechanism for sending the body of POST and PUT requests to the ClamAV module.

For analyzing only file content from the request body with Content-Type: multipart/form-data, use the FILE_ONLY option.

The file content of the PUT request body is transferred regardless of the FILE_ONLY value.

nwaf_ip_wl

Deactivation of the request analysis mechanism by means of the Nemesida WAF software for a specific IP address or subnet. With the parameter nwaf_ip_wl x.x.x.x domain=example.com; the analysis mechanism will be deactivated only when accessing a specific IP address to a specific virtual host. For the domain option it is allowed to use a wildcard value. As an IP address it is possible to use address with subnet mask (for example, 192.168.0.0/24) or subnet range (for example, 192.168.0.1-192.168.255.255).

To reduce false positives, it is recommended to specify the static IP address of specialized users (administrators, content managers, editors) in nwaf_ip_wl parameter. Requests are put under the parameter will not be blocked, transfered into RabbitMQ annd analysed by Nemesida AI module. Be careful using this parameter.

It is allowed to use IPv6 and IPv4-addresses.

nwaf_ip_lm

Configuring the pass of all occurrences of the rules for a specific IP address or subnet with the event recorded in the DBMS (IDS mode). With the nwaf_ip_lm x.x.x.x domain=example.com; pass will be made only when contacting a specific IP address to a specific domain. For the domain option it is allowed to use a wildcard value. As an IP address it is allowed to use address with subnet mask (for example, 192.168.0.0/24) or subnet range (for example, 192.168.0.1-192.168.255.255).

It is allowed to use IPv6 and IPv4-addresses.

nwaf_host_wl

Deactivation of the request analysis mechanism, using Nemesaida WAF instruments, for the virtual host. Using parameter nwaf_host_wl *; the pass will be made for all virtual hosts.

Examples for use single value:
nwaf_host_wl *;
nwaf_host_wl example.com;
nwaf_host_wl *.example.com;

Examples for use many values:
nwaf_host_wl example.com, 1.example.com;
nwaf_host_wl example.com, *.example.com;

nwaf_host_lm

Configuring the pass of all occurrences of the rules for a specific virtual host with the event captured in the DBMS (IDS mode). With the parameter nwaf_host_lm *; the pass will be made for all virtual hosts.

Examples for use single value:
nwaf_host_lm *;
nwaf_host_lm example.com;
nwaf_host_lm *.example.com;

Examples for use many values:
nwaf_host_lm example.com, 1.example.com;
nwaf_host_lm example.com, *.example.com;

nwaf_clamav_wl*

Configuring the blocking exclusion list for the contents of the request body or downloadable file by the MD5 amount (the MD5 amount is contained in the error.log of the Nginx software).

Example of a log entry:
Nemesida WAF: blocked by ClamAV, stream: Eicar-Test-Signature FOUND, md5: 44d88612fea8a8f36de82e1278a-bb02f, ...

Example of WL rule:
nwaf_clamav_wl 44d88612fea8a8f36de82e1278a-bb02f.

nwaf_log_mr_all

Activation of the parameter for recording information about all occurrences of blocking rules (attack signatures) in the Nginx software error log. By default, the parameter is deactivated (commented out). When a parameter is deactivated, only the signature is written to the error log of the Nginx software, which led to blocking the request or fixing the signature without further blocking (in the LM mode).

Using thenwaf_log_mr_all domain=example.com; parameter the record of the recorded occurrances will do only for specific domain. For the domain option it is possible to use wildcard value.

nwaf_check_bot_name

This parameter is used for defining search engine robots. Blocked requests that fall under the parameter do not increase the value of the rate counter and cannot cause the IP address to be blocked. The first parameter value is defined in the User-Agent zone. The second parameter value defines the host name obtained by reverse IP address conversion. If the second value is not set, it is considered equal to the first one.

Examples:
nwaf_check_bot_name "example.com" "example.net";
nwaf_check_bot_name "example.com";
nwaf_check_bot_name "bingbot" "msn.com";
nwaf_check_bot_name "yandex.ru";

nwaf_geoip_db_path*

The parameter defining the location of the file with the GeoIP2 base. Used by the extended locking mechanism.

Examples:
nwaf_geoip_db_path /opt/geoip/GeoLite2-Country.mmdb;

For correct operation, it is necessary to grant the Nginx web-server access to read the specified file.

* Not used in Nemesida WAF Free.

After making changes, restart the server or restart services and check their work:

# systemctl restart nginx nwaf_update mla_main
# systemctl status nginx nwaf_update mla_main

The nwaf_update service is responsible for obtaining the signatures of the Nemesida WAF module (/etc/nginx/nwaf/rules.bin). To test the signature method of detecting attacks, when sending a request to http://YOUR_SERVER/nwaftest, the server must return a 403 response code.

The Nemesida WAF module processes only the request transmitted to the final application. If the settings of the Nginx software prevent the transmission of calls (returning, for example, a 301 or 403 response code), such requests will not be processed by the Nemesida WAF module.

If the signature set would be changed on the server nemesida-security.com nwaf_update service will automatically update rules.bin file and apply the changes. If the nwaf_update service is deactivated during the signature set updating on the server, the download of the new version of signature set will be after the service nwaf_update starts.

Optional header $nwaf_block_type

Nemesida WAF allows to use special header, which determinative the result of the request processing. To activate it required to add parameter add_header NemesidaWAF-BT $nwaf_block_type always; into http, server or location area of the nginx configuration file.

Example of using the header

  • Set the header into the server area:
    server {
       ...
       add_header NemesidaWAF-BT $nwaf_block_type always;
       ...
    }
    

    and restart nginx.

  • Execute the request: curl -i localhost/nwaftest.

The server’s answer will contain the information about the status of the request processing in the NemesidaWAF-BT header:

HTTP/1.1 403 Forbidden
Server: nginx/1.16.1
Date: Wed, 12 Feb 2020 19:05:26 GMT
Content-Type: text/html
Content-Length: 153
Connection: keep-alive
NemesidaWAF-BT: 2
...

In this case the status is 2. It means that the request was blocked because of the attack’s signature entry with the score equal to 12 (nwaftest). More information about the request processing status is available in the corresponding paragraph. This header can be useful during the generation of the customized page of the server answer (for example if the request is blocked).

Other information

LM mode features
The request that falls under the action of the nwaf_ip_lm, nwaf_host_lm or LM parameters and has BT 0 (request processing status) does not blocked, but will be included in the training sample.

The request that falls under the signature with a score (digital significance indicator) equal to 12, in the LM mode, will be transmitted to the Nemesida WAF API module, as well as to other query analysis subsystems.

If the request with score equal to 12 does not fall under the LM mode, the request will be blocked by signature analysis with transmission to the Nemesida WAF API module, but without processing by other request analysis subsystems.

Installation and setting up the Nemesida AI module

The module is not used in Nemesida WAF Free.

The Nemesida AI module consists of the Nemesida AI MLA modules (included in the installation package of the Nemesida WAF) and Nemesida AI MLC which interaction is possible in a standart mode (the modules operate on the same server) and point-to-multipoint mode (Nemesida AI MLC operates on a dedicated server).

Behavioral models accuracy
To create high quality models it is not recommended to make vulnerability web application scanning and to send other illegitimate requests during the models training. After the first training it is recommended to re-train the models. False positives are controlled using Nemesida WAF Signtest module.

Python pip-packages
For machine learning modules to work correctly, you must use the single versions of Python 3 pip-packages on servers with Nemesida AI MLA and Nemesida AI MLC installed.

Storing behavioral models
Behavioral models created by the Nemesida AI MLC module are transmitted to the remote Nemesida AI MLS server and automatically propagated to all running instances of Nemesida AI MLA and Nemesida AI MLC in accordance with the WAF ID.

Nemesida AI MLA

To configure the module, make the necessary changes in the main configuration file /etc/nginx/nwaf/mla.conf.

mla.conf parameters
Default parameter
Parameter description

[main]
Section is responsible for the general settings Nemesida AI MLA.

[st]
Section is responsible for interaction with the Nemesida WAF Signtest.
st_enable
Sending disputed requests to the Nemesida WAF Signtest server for post-processing.

Disputed requests are defined as follows:
– if the signature analysis determined the request as illegitimate, and the Nemesida AI MLA module determined it as legitimate;
– if the signature analysis determined the request as legitimate, and the Nemesida AI MLA module determined it as illegitimate.

st_uri
Nemesida WAF Signtest server URI for processing Nemesida AI results.
[deep_inspection]
Advanced request processing management section.
b64_decode_url
b64_decode_args
b64_decode_body
b64_decode_cookie
b64_decode_other_headers
Activate/deactivate of the Base64 decoding mechanism for a specific zone. For example:

b64_decode_args = true – Base64 decoding mechanism for ARGS zone is active;
b64_decode_args = false – Base64 decoding mechanism for ARGS zone is inactive.

The Nemesida AI MLA module loads models according to the list obtained from the Nemesida AI MLC module (vhosts_list parameter in the mlc.conf file).

Nemesida AI MLC

Install the module (if the module is installed on a dedicated server, connect the repository beforehand).

DebianUbuntuCentOS
Debian 9
# apt install python3 python3-pip python3-dev python3-setuptools libc6-dev rabbitmq-server dmidecode gcc memcached
# python3.5 -m pip install --no-cache-dir cython pandas simple-crypt pika logutils sklearn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize python-geoip-python3 python-geoip-geolite2 netaddr pymemcache dnspython
Debian 10
# apt install python3 python3-pip python3-dev python3-setuptools libc6-dev rabbitmq-server dmidecode gcc memcached
# python3.7 -m pip install --no-cache-dir cython pandas simple-crypt pika logutils sklearn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize python-geoip-python3 python-geoip-geolite2 netaddr pymemcache dnspython

Install Nemesida AI MLC:

# apt install nwaf-mlc
Ubuntu 16.04
Add and install the Python 3.6 repository:

# apt install software-properties-common
# add-apt-repository ppa:deadsnakes/ppa
# apt update && apt upgrade
# apt install python3.6 python3.6-dev curl
# curl https://bootstrap.pypa.io/get-pip.py | python3.6
# apt install python3.6 python3.6-dev libc6-dev rabbitmq-server dmidecode gcc memcached
# python3.6 -m pip install --no-cache-dir cython pandas simple-crypt pika logutils sklearn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize python-geoip-python3 python-geoip-geolite2 netaddr pymemcache dnspython
Ubuntu 18.04
# apt install python python3-pip python3-dev python3-setuptools libc6-dev rabbitmq-server dmidecode gcc memcached
# python3.6 -m pip install --no-cache-dir cython pandas simple-crypt pika logutils sklearn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize python-geoip-python3 python-geoip-geolite2 netaddr pymemcache dnspython
Ubuntu 20.04
# apt install python3.8 python3-pip python3.8-dev python3-setuptools libc6-dev rabbitmq-server dmidecode gcc memcached
# python3.8 -m pip install --no-cache-dir cython pandas simple-crypt pika logutils sklearn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize python-geoip-python3 python-geoip-geolite2 netaddr pymemcache dnspython

Install Nemesida AI MLC:

# apt install nwaf-mlc
CentOS 7
# yum install python36-pip python36-devel python36-setuptools python36-pika gcc dmidecode rabbitmq-server memcached
# python3.6 -m pip install --no-cache-dir cython pandas simple-crypt pika logutils sklearn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize python-geoip-python3 python-geoip-geolite2 netaddr pymemcache dnspython
# yum install nwaf-mlc
CentOS 8
# dnf install gcc dmidecode rabbitmq-server python3-pip python3-devel python3-setuptools python3-pika memcached
# python3.6 -m pip install --no-cache-dir cython pandas simple-crypt pika logutils sklearn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize python-geoip-python3 python-geoip-geolite2 netaddr pymemcache dnspython
# dnf install nwaf-mlc

For the correct operation of some functionality, it is recommended to use 1.1.1.1 as the system address of the DNS server.

Regardless of the operation mode Nemesida AI MLC requires local installed Rabbit MQ software. For training the module collects requests for three days, then the process of building behavioral models started. Later the models will be transmitted to the Nemesida AI MLA module.

To configure the module, make the necessary changes to the configuration file /opt/mlc/mlc.conf.

mlc.conf parameters to operate in normal mode
Default parameter
Parameter description

[main]
Section is responsible for the general settings Nemesida AI MLC.
vhosts_list
A list of domain names used as virtual hosts for which behavioral models need to be created.

Wildcard values can be used. The model in the request is applied in the following order of priority from highest to lowest:

1. vhosts_list = example.com – building and applying a model for the certain domain;
2. .example.com – building and applying a model for the domain .example.com and its subdomains;
3. *.example.com – building and applying the model only for subdomains *.example.com, excepting the main domain example.com;
4. * – building and applying the model for the rest domains.

For example:
vhosts_list = example.com 1.example.com 2.example.com – building and applying models for the listed virtual hosts.

It is not allowed to use example.com and .example.com together. If you want to use one model for a domain and another for its subdomains, then use example.com and *.example.com. For example, vhosts_list = example.com *.example.com.

When applying behavioral models to a request, the domain level is taken into account, that is, the model corresponding to the virtual host from the request will be applied, and if it is not present, the model that includes this domain will be applied.

So, for a request with a domain name b.example.com the model will be applied b.example.com, if there is no such model — example.com, if there is no such model, then the model *.example.com.

You can change the model training period by setting a list of domain names: x:example.com, where x is the training period in days. For example, 5:example.com – model training will last 5 days.

ai_extra
Activation or deactivation of the additional requests analysis functional allows to detect missing attacks and make temporary blocking of their source using IP address.
api_uri
Nemesida WAF API address for sending information about models’ learning status. With parameter api_uri = none the information will not be sent.
nwaf_license_key
Installing the license key Nemesida WAF on a dedicated server. You must not use this setting if you work on the same server as the Nemesida WAF is or if you work in Multipoint Mode.

Example: nwaf_license_key = 1234567890.


[run]
Section is responsible for the parameters of the connection to RabbitMQ local server.
rmq_host
Parameters for connecting to the RabbitMQ service. You can use multiple values separated by a space, for example:

rmq_host = guest:guest@192.168.0.1 guest:guest@192.168.0.2
rmq_host_local
Parameters for connecting to the RabbitMQ service for local queue placement, for example:

rmq_host_local = guest:guest@127.0.0.1

If this parameter is omitted, the following values will be used: guest:guest@127.0.0.1.


[proxy]
Section is responsible for the settings of the connection to proxy server.
sys_proxy

Configure of the proxy server address for request to nemesida-security.com:443 (check of license key, loading of behavioral models). For example:
sys_proxy = proxy.example.com:3128.

api_proxy
Configure of the proxy server address for request to Nemesida WAF API and Nemesida WAF Signtest. For example:
api_proxy = proxy.example.com:3128.

If the parameters have no values, the module will try to use the parameters from the nwaf.conf file.


[mls]
Section is responsible for transmitting traffic to a remote server for the purpose of further analysis and building behavioral models. To use this functionality, please contact technical support.
mls_enable
Activation of the mechanism for transmitting the analyzed traffic to the Nemesida WAF MLS server. By default, the functionality is deactivated.

[ddos]
The section responsible for the functionality of detecting denial of service attacks (DDoS attacks) at the application level.
enable
Activation/deactivation of the DDoS attack detection functional.
wl_ip
The parameter that defines the path to a file in which IP addresses are specified in the 1.2.3.4 format, for which the DDoS detection functionality will be disabled. Each subsequent value is indicated with a space.

Changes to files are applied automatically without restarting Nemesida AI MLC.

wl_url
The parameter defining the path to the file in which the addresses are specified both in the vhost format and in the vhost/path format, where:

vhost – the name of the virtual host for which the DDoS detection functionality will be disabled.
path – resource address entry.

For virtual host are allowed wildcard values. Example:

example.com
.example.com
*.example.com
example.com/auth

Changes to files are applied automatically without restarting Nemesida AI MLC.

interval
The time interval of the segment (window) during which the analysis of requests is performed.
latest_only
Activation of sending only the last blocked request for each IP address to the Nemesida WAF API. If set to false, the Nemesida WAF API passes all blocked requests for each IP address.
send_possible
Activation of the mechanism for sending requests with the Possible DDoS type to the Nemesida WAF API. If the value is false, requests to the Nemesida WAF API will not be transmitted.
The prefix Possible is added to the name of the attack if its type has not been reliably established.

[brute]
The section responsible for the brute force detection function. Revealing of values ​​is performed in the ARGS and/or BODY areas.
enable
Activation/deactivation of the brute force attack detection functional.
wl_host
The parameter that allows you to deactivate the brute force attack detection functionality for specific virtual hosts. You can use wildcard values.

Examples:
wl_host = example.com .example.org *.example.us

Parameter changes are applied automatically, without restarting Nemesida AI MLC.

interval
The time interval of the segment (window) during which the analysis of requests is performed.
max_val
The number of requests that, when reached, block the source(s) of the attack.
brute_detect
The parameter that defines the path to the file where user needs to set addresses for detection brute force attacks in the format vhost/path, where path is the occurrence of the resource address on the web server. You can use wildcard values for a virtual host. For example:

example.com/auth
.example.com/auth
*.example.com/auth

Thus, when the value is set example.com/auth brute force attacks will be monitored for both example.com/auth, and for example.com/auth/reset_password.

This parameter is used to detect brute force attacks, but it does not block repeated requests (when the content in the ARGS or BODY zones is passed several times).

For addresses not specified in the file, brute force attacks are not detected. Changes to files are applied automatically without restarting Nemesida AI MLC.

flood_detect
The parameter has a similar functionality to the brute_detect parameter, but it is intended for detecting flood attempts or similar attacks with repeated requests. The only difference is that during the analysis of requests that fall under the action of the flood_detect parameter, duplicates are not deleted.

Thus, in the case of repeated sending of identical requests (for example, multiple attempts to recover the password via SMS), requests that have similar content and fall under the action of the flood_detect parameter will not be deleted, unlike requests that have similar content but fall under the action of the brute_detect parameter.

For addresses not specified in the file, flood attempts are not detected. Changes to files are applied automatically without restarting Nemesida AI MLC.

latest_only
Activation of sending only the last blocked request for each IP address to the Nemesida WAF API. If set to false, the Nemesida WAF API passes all blocked requests for each IP address.
send_possible
Activation of the mechanism for sending requests with the Possible Brute force/Possible Flood type to the Nemesida WAF API. If the value is false, requests to the Nemesida WAF API will not be transmitted.
The prefix Possible is added to the name of the attack if its type has not been reliably established.

[st]
Section responsible for the interaction with Nemesida WAF Signtest module of training management.
st_enable
Sending disputed requests received from the Nemesida WAF module using RabbitMQ to the Nemesida WAF Signtest server for post-processing.

Disputed requests are defined as follows:
– if the signature analysis determined the request as illegitimate, and the Nemesida AI MLC module was defined as legitimate;
– if the signature analysis determined the request as legitimate, and the Nemesida AI MLC module was defined as illegitimate.

st_uri
Nemesida WAF Signtest server URI for sending disputed requests. When using the local version of Nemesida WAF Signtest, change the URI of the parameter.

[training]
Learning process management section.
dataset_limit
Sets the maximum number of unique queries included in the training sample.
Multipoint Mode

Nemesida AI MLC is required to 32 GB of free RAM. If you use several services with Nemesida WAF module you are able to save hardware resources using Point-to-Multipoint scheme (one server with installed Nemesida AI MLC module interacts with servers with installed Nemesida WAF module).

Components are used in Multipoint Mode:

  • Servers with installed Nemesida WAF and RabbitMQ, which have 2-4 GB of RAM for every of them;
  • Server with installed Nemesida AI MLC and RabbitMQ, which have 32 GB of RAM.

On the server with installed Nemesida WAF

– Create service’s user RabbitMQ:

# rabbitmqctl add_user USER PASSWORD
# rabbitmqctl set_permissions -p / USER ".*" ".*" ".*"

where USER and PASSWORD are username and password to connect Nemesida AI MLC module.

– Make changes to the file /etc/rabbitmq/rabbitmq-env.conf:

NODE_PORT=5672
export RABBITMQ_NODENAME=rabbit@localhost
export RABBITMQ_NODE_IP_ADDRESS=0.0.0.0
export ERL_EPMD_ADDRESS=127.0.0.1

– Permit accesses from the server where Nemesida AI MLC installed to the port 5672 (TCP).
– Complete settings of RabbitMQ:

# service rabbitmq-server restart

On the server with installed Nemesida AI MLC

Create additional configuration files in the /opt/mac/conf/ directory by copying the /opt/mlc/mlc.conf file. Make changes to the new configuration files for working with the remote RabbitMQ server. After making the changes, restart the service:

# service mlc_main restart
# service mlc_main status

In additional configuration files, nwaf_license_key is a required parameter. The license key used in the Nemesida AIMLC settings and the deleted Nemesida WAV must have the same WAF ID. If you are using additional configuration files, we recommend deleting the /opt/mlc/mlc.conf file.

Using RabbitMQ’s mode services Nemesida AI MLC will collect requests with following models training as if it worked in normal mode.

Using the Nemesida AI MLS cloud server
The cloud server Nemesida AI, located in the Pentestit infrastructure, is designed to generate behavioral models based on a copy of traffic received from remote servers. The cloud server is used in cases when the user of the Nemesida WAF software does not have enough RAM for the Nemesida AI MLC module (it requires up to 32 GB). To use the capabilities of the cloud server Nemesida AI, contact technical support.

After making the changes, restart the service:

# service mlc_main restart
# service mlc_main status

Nemesida AI models retraining

To improve the accuracy of attack detection, it is recommended to re-train models once a week. For that you need to add the ^ symbol to virtual host. For example: vhosts_list = example.com^.

Restart the service after changes:

# service mlc_main restart

After retraining the models, it is recommended to delete the exported BT 12 queries (available on the False Positive page of the module web interface Nemesida WAF Signtest). When training models for a virtual host, BT 12 requests will be included in the training set and will not be required further.

Nemesida AI behavioral models training period increase

Main method

For the correct construction of models, about 400.000-800.000 unique requests are required. By default, the training period is limited to four days. You can change the model training period directly by setting a list of domain names in the vhosts_list parameter in mlc.conf: x:example.com, where x is the training period in days. For example, 5:example.com – model training will last 5 days.

Restart the service after changes:

# service mlc_main restart
Additional method

If you need to extend the training period for the required number of requests, follow these steps:

1. Stop the service Nemesida AI MLC:

# service mlc_main stop

2. Rename the file /opt/mlc/ml/[vhost]_[timestamp].db, where [timestamp] – The starting date of model training, from which the training period is calculated. For example, to increase the models building period by five days, it is necessary to increase the timestamp values for the same period:

# mv /opt/mlc/ml/example.com_1587416400.db /opt/mlc/ml/example.com_1587848400.db

3. Run the service Nemesida AI MLC:

# service mlc_main start

Additional training of models using backup copy of the training sample

For the correct construction of models, about 400.000-800.000 unique requests are required. If the number of requests was insufficient during training, then you can restart it and use the requests from the previous sample. To do this, follow these steps:

1. Stop the service Nemesida AI MLC:

# service mlc_main stop

2. Move the file /opt/mlc/ml/backup/[vhost].d_[timestamp], where [timestamp] – the date of creation of a backup copy of the training sample created by Nemesida AI MLC, before starting to build the model, in /opt/mlc/ml/[vhost].d. For example, for the model example.com:

# mv /opt/mlc/ml/backup/example.com.d_1613587613 /opt/mlc/ml/example.com.d

3. Start training. To do this, add the ^ symbol to the virtual host value. Example: vhosts_list = example.com^.

Run the service Nemesida AI MLC:

# service mlc_main start

After the training period is completed (the default is 4 days, the period can be changed), the behavioral model will be created based on requests from the common dataset.

Nemesida AI models removing

If the behavioral models are not trained correctly or there are significant changes in the web application that lead to a lot of false positives, it is recommended deleting the models. To do this, send a request to support@nemesida-security.com. In the request, specify the license key used in Nemesida WAF.

Training sample for building behavioral models

Queries detected as BT 1, BT 2, BT 3 or BT 4 are not added to the training sample, even if they fall under the LM mode.

When ai_extra mode is enabled, queries detected by the Nemesida AI MLC module as illegitimate are not added to the training sample.

Configuring interaction with ClamAV software
After installing Nemesida WAF package the functionality of interaction with ClamAV software is disabled by default, since it can be a source of false blocking of some requests to the Web application (depending on the current state of the ClamAV signature analysis database). Use this functionality at your discretion.

To activate antivirus protection, install ClamAV software on the server with the configured Nemesida WAF software, if it has not done yet.

Installation example for Debian 9 OS:

# apt install clamav-daemon

The interaction with the ClamAV software is enabled by activating the nwaf_clamav configuration parameter in the /etc/nginx/nwaf/conf/global/nwaf.conf file and reduction of the /etc/clamav/clamd.conf file to a look:

...
TCPSocket 3310
TCPAddr   127.0.0.1
...

After changing restart the Nginx software.

Using Nemesida WAF in IDS mode
It is necessary to set up traffic mirroring from the main web server (through which calls to the web application are made)to the server with the installed Nemesida WAF software, to working Nemesida WAF in IDS mode. You must make changes to the files on every server:

1. On the main server (without the Nemesida WAF module installed), configure traffic mirroring according to the guidelines of the installed web server (Nginx, Apache2, Microsoft IIS and others).

An example of Nginx setting for traffic mirroring

If using the Nginx web server, make the necessary changes to the virtual host file:

location / {
    mirror /mirror;
    ...
}

location = /mirror {
    internal;
    proxy_pass http://nemesida_waf_server$request_uri;
}

where nemesida_waf_server is the address of the server with the Nemesida WAF module installed, to which duplicate traffic will be transmitted.

2. On the server with the installed Nemesida WAF module, bring the configuration file of the virtual host Nginx to the form:

server {
        listen  80;
        index   index.html;
        root    /var/www/html;
        try_files $uri $uri/ /index.html;
}

3. On the server with the Nemesida WAF module installed, create the /var/www/html directory and place an empty index.html file in it.

4. On the server with the Nemesida WAF module installed, bring the /etc/nginx/nwaf/conf/global/nwaf.conf file to the form:

...
nwaf_limit rate=5r/m block_time=0;
...

5. After changing restart the Nginx on every server.

Using Nemesida WAF in reverse proxy mode

To use a web server with installed Nginx and Nemesida WAF as an intermediate server, you should create corresponding file of the virtual host (for example /etc/nginx/conf.d/example.com.conf).

Example of the virtual host file
server {
        server_name  site.example.com;
        listen 80;

        location / {
                proxy_pass http://192.168.0.1;
        }
}

where 192.168.0.1 is the address of destination server with the web application.

More information is available on the official page of the Nginx documentation.

Nemesida WAF work in cluster work (active/active)

For Nemesida WAF to work in a cluster, all license keys used must have a single WAF ID (identifier that allows to combine different license keys into a group). To group license keys into a single WAF ID, send a request to support@nemesida-security.com.

Behavioral models of machine training, as well as FP requests exported via the Nemesida WAF Signtest functionality, will be automatically downloaded to all instances of the cluster in accordance with WAF ID. For a more accurate detection of attacks, it is recommended to use one installed instance of the Nemesida AI MLC module at one time.

When using Nemesida WAF as a cluster solution in active/active mode, you must:
1. Organize synchronization of Nemesida WAF configuration files (the /etc/nginx/nwaf/ directory by default, excluding the objects /etc/nginx/nwaf/rules.bin and /etc/nginx/nwaf/ml/) with all installation instances. Synchronization should be done after each modification implementation to the configuration files.

Example of synchronization using rsync utility
# rsync -az --no-perms --no-owner --no-group --exclude "/ml/" --exclude "/rules.bin" /etc/nginx/nwaf/ root@%remote_server_address%:/etc/nginx/nwaf/ 

2. After synchronizing the objects, it is necessary to update the parameters in the /etc/nginx/nwaf/conf/global/nwaf.conf file on each installed instance of the Nemesida WAF dynamic module:

  • nwaf_license_key (necessarily);
  • nwaf_api_proxy and nwaf_sys_proxy (optionally).

3. After modifications, you must restart the services:

# nginx -t && systemctl reload nginx 
# systemctl restart nwaf_update mla_main 

Signature management

There is a support of custom set of signatures rules (signature, RL) and signature exception rules (exclusion rule, WL) in Nemesida WAF and Nemesida WAF Free.

Zones and additional conditions

During the creation of RL or WL rules special parameters can be used:

  • zones: URL, ARGS, BODY, HEADERS, Cookie, User-Agent, Content-Type, etc.;
  • conditions of using the rule (additional condition, specification of the zone): $URL, $ARGS, $BODY, $HEADERS, $Cookie, $User-Agent, etc.

Using zones and additional conditions allows to concretize maximally the creating rule. Additional conditions are set by adding prefix $ (for example: "Z:...|$URL:...").

An example of using the rule with zone and additional condition: WL ... "Z:ARGS|$URL:/123"; – the rule will work, if the entry zone of the RL rule will be ARGS zone and URL will contain an entry /123.

Several parameters (zones, additional conditions) in one rule must be separated with the character |, the following principle of interaction will be used:

  • zones interact using the logical principle OR, for example: Z:URL|ARGS;
  • zones interact with additional conditions using the logical principle AND, for example: Z:URL|$ARGS:123;
  • additional conditions interact using the logical principle AND, for example: Z:...|$ARGS:123|$BODY:123".

Regular expressions in the additional conditions

It is possible to use regular expressions in the additional conditions. This requires to add to the additional condition postfix _X (for example: "Z:...|$URL_X:\w+").

If | is used as an ordinary character, it is necessary:

  • for an ordinary rule: to execute its dual shielding (for example: Z:...|$BODY:block\\|page – search for an entry block or page);
  • for the rule with the regular expression (with postfix _X): to execute its quad-shielding (for example: $BODY_X:block\\|page – search of an entry block|page).

If in the rule with the regular expression (with postfix _X) other characters from the set of regular expressions operator are used (/, ., *, ?, !, {, }, [, ], (, ), ^, $, :, \, etc.) as ordinary characters it requires to execute their dual shielding (for example: $BODY_X:data=\\(block-\\\a\\\-in-the-main\\) – search for the entry (block-\a\-in-the-main); $HEADERS_X:Accept\\: image\\/webp\\\\\* – search for the entry Accept: image/webp\*).

For use in additional conditions the separator as a metacharacter of a regular expression it is necessary to shield it twice. For example: the rule WL ... "Z:...|$URL_X:/(a\\|b)/"; will bring to URL which contains /a/ or /b/.

User-defined signature creation

The user-defined rules of detecting attack signs (user-defined RL) can be placed in the main configuration file Nemesida WAF (nwaf.conf) or in the self-created file of the form *.conf, located in the /etc/nginx/nwaf/conf/vhosts directory. User-defined signatures must be determined by the RL parameter, must have ID ranging from 50000 to 99999 and take the following form:

RL ID:50000 "P:select from" "SC:SQL:12" "Z:ARGS";
RL ID:50001 domain=example.com "P:select from" "SC:SQL:12" "Z:ARGS";
RL ID:50002 domain=.example.com "P:select from" "SC:SQL:12" "Z:ARGS";
RL ID:50003 domain=*.example.com "P:select from" "SC:SQL:12" "Z:ARGS";
RL ID:50004 "PX:select\s+from" "SC:SQL:12" "Z:ARGS|$URL:/admin";
RL ID:50005 "P:select from" "SC:SQL:12" "Z:ARGS|$URL:/(admin\\|dev)";

For signatures in the zone it is allowed to use the MLA value to force request sending into Nemesida AI MLA, that falls within the scope of the created signature if the score < 12 (by default requests with a score equal or greater than the value of the mla_score parameter are sent into Nemesida AI MLA).

Signature force sending into Nemesida AI MLA examples

Request contains an entry select from in any of four zones will be sent into Nemesida AI MLA:

RL ID:50005 "P:select from" "SC:SQL:1" "Z:MLA";

Request contains an entry select from in ARGS zone will be sent into Nemesida AI MLA:

RL ID:50006 "P:select from" "SC:SQL:1" "Z:ARGS|MLA";
Signature options
ID
The unique identifier for the rule. A range from 50000 to 99999 is available for creating your own rules. Required.
domain
Set the ownership of the rule to the domain. For the domain option it is allowed to use a wildcard value.
P/PX
Option defining the entry pattern (option P is used to denote a simple entry, option PX is a regular expression). Required.
SC
Setting the rule tag (Injection, XSS, UWA, Scanner, Evasion, Other or another) and the numerical value indicator (from 1 to 12). Required.

Requests that fall within the scope of the rules with an indicator value of 12 are blocked without being sent to other analysis subsystems. More information is available in the corresponding section.

Z

Zone application of the rule. To apply a signature to all zones, use the empty "Z:" parameter. Required.

To apply a signature to multiple zones, use the delimiter "Z:URL|BODY".

To force the request to the Nemesida AI MLA module use MLA value in rules application zone ("Z:MLA") while digital significance indicator (SC) must be less than 12.

To disable sending a request to the Nemesida AI MLA module, use the NoMLA value in the rule application area ("Z: NoMLA").

To disable IP address blocking due to the occurrence of a signature that has score = 12, use the value NoBan in the rule area ("Z:NoBan"). Entering a signature with such a zone does not reduce the nwaf_limit counter, meaning it does not block the IP address.

To clarify the zone, you can use additional options. For "Z:ARGS|$URL:/templates" the rule will work only in the ARGS zone with /templates parameter in URL.

For signatures in zone area, you can use the NoAPI value to disable sending a message to the Nemesida WAF API that falls under the action of the created signature. For all triggers of this signature, the message will not be sent to the Nemesida WAF API, but it will be blocked and written to the log file. This value is only used for signatures that have score = 12. For signatures with a different score, this value is ignored.

RL ID:50001 "P:select from" "SC:SQL:12" "Z:ARGS|URL|BODY|NoAPI";

For zone area you can use the LM value, which activates the LM mode for a request that falls under the action of the created signature. Each request that falls under the signature will not be blocked, but a corresponding message will be sent to the Nemesida WAF API about it, which is also recorded in the log file. The LM value is only used for signatures that have score = 12. For signatures with a different score, this value is ignored.

RL ID:50002 "P:select from" "SC:SQL:12" "Z:ARGS|URL|BODY|LM";

To reduce the number of false positives during the creation signature rules it is neсessary to specificate them maximally.

Creating a signature exclusion rule

In case the inquiry falls under action of a signature, in addition to sending the incident to the Nemesida WAF API, the following line will be displayed in the error log of the Nginx software:

Nemesida WAF: the request ххх contains a rule id 1 in zone HEADERS, ...

or, if the request contains a signature with a maximum allowable digital indicator of significance (score = 12):

Nemesida WAF: the request ххх blocked by rule id 1 in zone HEADERS, ...

where:

  • 1 – attack signature ID;
  • HEADERS – signature entry zone.

To display absolutely all occurrences of signatures in the request (if there are occurrences), including those occurrences that did not lead to the subsequent blocking of the request, activate the nwaf_log_mr_all; parameter, in the nwaf.conf file Nemesida WAF.

The information about current signature list is on the page rlinfo.nemesida-security.com.

Examples of creating attack signature exception rules

WL ID:1 "Z:"; – using these parameters, the entry of the rule with the identifier 1 will be excluded from all zones for all virtual hosts.

WL ID:1 "Z:ARGS|HEADERS"; – using these parameters the entry of the rule with the identifier 1 will be excluded from the ARGS and HEADERS zones for all virtual hosts.

WL ID:1 "Z:ARGS|Cookie"; – using these parameters the entry of the rule with the identifier 1 will be excluded from the ARGS and Cookie for all virtual hosts.

WL ID:1 domain=.example.com "Z:URL"; – using these parameters the entry of the rule with the identifier 1 will be excluded from the URL zone for the virtual host example.com and its subdomains.

WL ID:1 domain=example.com "Z:URL|$URL:/index/index.php"; – using these parameters the entry of the rule with the identifier 1 will be excluded from the URL zone for the virtual host example.com for URI http://example.com/index/index.php.

WL ID:* domain=example.com "Z:ARGS"; – with these parameters, the ARGS zone of all requests to the virtual host example.com will be excluded from the signature analysis processing.

WL ID:* domain=example.com "Z:$URL:/test"; – using these parameters the entry of the rule all requests to example.com/test will be excluded from the signature analysis processing.

For parameter ID:* it is possible to use any of zones as a refinement, but as a condition of using the rule – only parameter $URL_X.

For security reasons, when creating exclusion rules, you need to specify them as much as possible.

Switching the blocking rules to LM mode

Requests that fall under the LM mode are recorded in the database, but are not blocked. When the rule is triggered, a message will be sent to the Nginx error log and to the Nemesida WAF API about switching the corresponding blocking rule to LM mode, but the request will not be blocked. The rules for switching to the LM mode must be defined by the LM parameter, have the ID of the corresponding blocking rule and take the following form:

Examples of creating rules for switching to LM mode
LM ID:1 "Z:URL"; – with these parameters, requests with occurrence the signature with the ID 1 in the URL zone will be switched to the LMmode;

LM ID:50001 "Z:URL"; – with these parameters, requests with the occurrence of a user-defined signature with the identifier 50001 in the URL zone will be switched to the LMmode;

LM ID:50002 domain=example.com "Z:URL"; – with these parameters, requests with the occurrence of a user-defined signature with the identifier 50002 in the URL zone will be switched to the LM mode for the domain example.com;

LM ID:50003 domain=*.example.com "Z:ARGS"; – with these parameters, requests with the occurrence of a user-defined signature with the identifier 50003 in the ARGS zone will be switched to the LM mode for the domain example.com and his subdomains.

Request processing statuses (BT)

Each request processed by the Nemesida WAF module is assigned a blocking identifier (BT – blocking type).
The identifier can be assigned regardless of whether the request was blocked.

The pivot table shows the types of identifiers and their corresponding request processing status.

0 The request was not blocked.
1 The request is blocked by signature method and the request did not contain any signature with the score = 12.
2 The request is blocked by signature method and the request contained a signature which has the score = 12.
3 The request is blocked by Nemesida AI MLA.
4 The request is blocked by ClamAV module.
5 The request is blocked because of the internal error.
6 The request is blocked because of the oversubscription of blocked requests from one IP address.
7 The request is detected like the brute force attempt and blocked by Nemesida AI MLC.
8 The request is blocked by Nemesida AI MLC (additional traffic analysis of all nonblocked requests by Nemesida WAF module) and is managed by ai_extra parameter in mlc.conf file.
9 The request is detected as a flood attempt and blocked by the Nemesida AI MLC.
10 The request is detected like the DDoS attacks and blocked by Nemesida AI MLC.

For BT 1, 2, 3, 4, 8 events, the nwaf_limit parameter in the nwaf.conf file allows you to adjust the allowed number of blocked requests. If the allowed number of requests specified by the rate option is exceeded, the IP address will be blocked for the time (in seconds) specified in block_time.

For BT 7, 9, 10 events the blocking of the IP address of the request source will occur after the first blocked request, regardless of the value of the rate parameter (the allowed number of blocked requests) in nwaf_limit.

Unlocking an IP address using captcha
If suspicious activity is detected, Nemesida WAF allows you to block the IP address of the source of requests for a certain time (the duration of the block is regulated by the nwaf_limit option in the nwaf.conf configuration file). Using the captcha functionality, the user can unlock their own IP address if it has been identified as a source of brute force attacks (BT 7), floods (BT 9) or denial of service attacks (BT 10).

For correct work of the IP address unlocking mechanism, you need to activate the nwaf_captcha_unban parameter by adding it in the virtual host file to the server section for a specific URL address. Example of activating the nwaf_captcha_unban parameter

server {
    listen  80;
    server_name example.com;

    location /captcha_ip_unban_path {
             ...
             nwaf_captcha_unban on;
             ...
    }
    ...
}

Also, you need to setup a web application with a captcha. Detailed instructions for integrating Nemesida WAF with reCAPTCHA functionality can be found at link.

After activating the nwaf_captcha_unban on; parameter and restarting Nginx, the IP address unblocking functionality becomes available for the target URL address with the nwaf_captcha_unban option set. For example: example.com/captcha_ip_unban_path.

If the answer is correct, the web application with captcha sends a request delete_banned_ip to the Nemesida WAF module to unlock the IP address. The user will then be redirected to the original page.

For security reasons, it is recommended to allow requests to the server URL with the Nemesida WAF dynamic module installed and the nwaf_captcha_unban IP address unlocking parameter activated only from the IP address of the server with captcha.

Syncing the list of blocked IP addresses

Not used in Nemesida WAF Free.

The Nemesida WAF functionality allows you to automatically synchronize blocked IP addresses between servers. The IP address blocking time is determined by the nwaf_limit option in the nwaf.conf configuration file. To activate synchronization of blocked IP addresses on all servers that use this functionality, you must:

  • set the same values for the nwaf_limit and nwaf_sync_ban_ip_key parameters in the nwaf.conf file;
  • define the target URL for syncing using the nwaf_sync_ban_ip_host parameter in the nwaf.conf file.
  • define a path with the nwaf_ban_ip_sync on; parameter set in the virtual host file in the location section to get a list of blocked addresses, for example:
    location /ban_ip_sync_path
    {
        ...
        nwaf_ban_ip_sync on;
        ...
    }
    

For security reasons, it is recommended to restrict access to the Blocked IP Address Synchronization Management interface to the list of allowed IP addresses to preventmanagement of Nemesida WAF work processes to unauthorized users.

Getting file with GeoIP2 base
The GeoIP2 base file provides IP-based geographic location information (identifying the country by the attacker’s IP address) and is used for the extended blocking mechanism. To obtain it, you must perform the following steps:

  • go to the address and select GeoLite2 Free Geolocation Data in the Products section;
  • register an account by filling in all required fields and providing a valid email address;
  • confirm the registration by clicking on the link from the email;
  • go to the Account Summary section in your personal account and follow the link Download Database;
  • on the page that opens, find GeoLite2 Country and download the archive by clicking on the link Download GZIP;
  • unpack the resulting archive to get the mmdb file with the base.

Error messages sources
During the Nemesida WAF operation the information about errors can be contained in:

  • system logs;
  • run-time journal of the Nginx software;
  • run-time journal of the RabbitMQ software;
  • run-time journal of the Nemesida WAF modules /var/log/nwaf/.
Technical support

Technical support of Nemesida WAF Free is on the forum only.

In case of unforeseen errors in the operation of the Nemesida WAF software, contact technical support by email (Mon-Fri from 10:00 to 19:00 time, GMT+3) or leave a message on the forum.