A guide for installing, initial configuration and operating the Nemesida AI MLC machine learning module, designed for building behavioral models, detecting brute-force attacks, flood and DDoS attacks, as well as generating query schemes based on the OpenAPI specification.

Installation

Not used in Nemesida WAF Community Edition.

Before installing Nemesida WAF components, add repository information to the system:

DebianUbuntuRHELDocker
# apt update
# apt install apt-transport-https gnupg2 curl
Debian 11
# echo "deb https://nemesida-security.com/repo/nw/debian bullseye non-free" > /etc/apt/sources.list.d/NemesidaWAF.list
# curl -s https://nemesida-security.com/repo/nw/gpg.key | gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/trusted.gpg --import
# chmod 644 /etc/apt/trusted.gpg.d/trusted.gpg
# apt update && apt upgrade
Debian 12
# echo "deb https://nemesida-security.com/repo/nw/debian bookworm nwaf" > /etc/apt/sources.list.d/NemesidaWAF.list
# curl -s https://nemesida-security.com/repo/nw/gpg.key | gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/trusted.gpg --import
# chmod 644 /etc/apt/trusted.gpg.d/trusted.gpg
# apt update && apt upgrade
# apt update
# apt install apt-transport-https gnupg2 curl
Ubuntu 20.04
# echo "deb [arch=amd64] https://nemesida-security.com/repo/nw/ubuntu focal non-free" > /etc/apt/sources.list.d/NemesidaWAF.list
# wget -O- https://nemesida-security.com/repo/nw/gpg.key | apt-key add -
# apt update && apt upgrade
Ubuntu 22.04
# echo "deb [arch=amd64] https://nemesida-security.com/repo/nw/ubuntu jammy non-free" > /etc/apt/sources.list.d/NemesidaWAF.list
# curl -s https://nemesida-security.com/repo/nw/gpg.key | gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/trusted.gpg --import
# chmod 644 /etc/apt/trusted.gpg.d/trusted.gpg 
# apt update && apt upgrade
Configure the SELinux policy or deactivate it with the command:

# setenforce 0

then bring the file /etc/selinux/config to the form:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

Add the Nemesida WAF repository by bringing the file /etc/yum.repos.d/NemesidaWAF.repo to the form:

[NemesidaWAF]
name=Nemesida WAF Packages for RHEL
baseurl=https://nemesida-security.com/repo/nw/rhel/$releasever/$basearch/
gpgkey=https://nemesida-security.com/repo/nw/gpg.key
enabled=1
gpgcheck=1

Install packages:

# dnf update
Information about using Nemesida AI MLC in a Docker container is available in corresponding section.

The Nemesida AI module consists of Nemesida AI MLA modules (is included in the installation package of the Nemesida WAF module) and Nemesida AI MLC, whose interaction is possible in normal mode (modules operate on the same server) and mode “dot-multipoint” (the Nemesida AI MLC module operates on a dedicated server).

Python pip packages
For machine learning modules to work correctly, it is necessary to use unified versions of Python3 pip packages on servers with Nemesida AI MLA and Nemesida AI MLC installed.

During analyse requests against their schema matching in OpenAPI format, all integer data formats are treated as type integer.

Installation

DebianUbuntuRHEL
# apt update
# apt install python3 python3-venv python3-pip python3-dev python3-setuptools libc6-dev rabbitmq-server gcc memcached
# apt install nwaf-mlc

During the installation of the module, the following PIP packages are additionally installed:
wheel cython pandas simple-crypt pika logutils scikit-learn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize netaddr pymemcache genson pyarrow

# apt update
# apt install python3 python3-venv python3-pip python3-dev python3-setuptools libc6-dev rabbitmq-server gcc memcached
# apt install nwaf-mlc

During the installation of the module, the following PIP packages are additionally installed:
wheel cython pandas simple-crypt pika logutils scikit-learn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize netaddr pymemcache genson pyarrow

RHEL 8 and derivatives
1. Add the RabbitMQ repository according to the manual.

2. Install the packages:

# dnf install epel-release
# dnf config-manager --set-enabled powertools
# dnf install rabbitmq-server

3. Check the correctness of the service:

# systemctl enable rabbitmq-server
# service rabbitmq-server restart
# service rabbitmq-server status

4. Install Nemesida AI MLC:

# dnf update
# dnf install python39 python39-devel python39-setuptools python39-pip gcc memcached
# dnf install nwaf-mlc
RHEL 9 and derivatives

1. Install the packages:

# dnf install epel-release
# dnf config-manager --set-enabled crb
# dnf install centos-release-rabbitmq-38
# dnf install rabbitmq-server

2. Check the correctness of the service:

# systemctl enable rabbitmq-server
# service rabbitmq-server restart
# service rabbitmq-server status

3. Install Nemesida AI MLC:

# dnf update
# dnf install python3 python3-devel python3-setuptools python3-pip gcc memcached
# dnf install nwaf-mlc

During the installation of the module, the following PIP packages are additionally installed:
wheel cython pandas simple-crypt pika logutils scikit-learn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize netaddr pymemcache genson pyarrow

Configuring

Pre-setup

For the correct operation of the Nemesida AI MLC module, a connection to the RabbitMQ installed on the filtering node is required. To do this, follow these steps on the server with the filtering node:

1. Allow access to RabbitMQ via the Nginx web server. To do this, add the appropriate entries to the /etc/nginx/nginx.conf file:

stream {
    server {
        listen 5673;
        proxy_pass 127.0.0.1:5672;
        allow x.x.x.x;
        deny all;
    }
}

where x.x.x.x — the IP address of the server where Nemesida AI MLC is installed.

2. Restart Nginx:

# nginx -t && service nginx reload

3. Allow access from the server on which the Nemesida AI MLC module is installed to the server with a filtering node on port 5673 (TCP).

Initial setup

After installing the module, it is necessary to make the initial configuration. The configuration file /opt/mlc/mlc.conf contains a list of available parameters of the Nemesida AI MLC module, which cannot be configured using Nemesida WAF Cabinet or API.

mlc.conf parameters
Default parameter
Description of the parameter

[main]
The section responsible for the general settings of the Nemesida AI MLC module.
nwaf_license_key
Installing the Nemesida WAF license key when working on a dedicated server.

Usage example:

nwaf_license_key = 1234567890
sys_proxy
Configuring the proxy server address for accessing external resources (checking the license key, downloading the list of virtual hosts, etc.).

Example:

sys_proxy=http://proxy.example.com:3128

It is allowed to use authentication parameters when using a proxy server.

Example:

sys_proxy=http://<user>:<password>@proxy.example.com:3128
api_proxy
Configuring the proxy server address to access the Nemesida WAF API.

Example:

api_proxy=http://proxy.example.com:3128

It is allowed to use authentication parameters when using a proxy server.

Example:

api_proxy=http://<user>:<password>@proxy.example.com:3128
api_uri
The Nemesida WAF API address for sending information about the training status of models and information about detected anomalies. If the parameter value is empty, no information will be sent.
debug
Debug mode.
[run]
The section responsible for connection parameters with the RabbitMQ service.
rmq_host
Connection parameters with the RabbitMQ service on filtering node for collecting data necessary for building a behavioral model, as well as messages about blocking requests for transmitting them to the Nemesida WAF API.

It is allowed to use multiple values separated by a space.

Example:

rmq_host = guest:guest@192.168.0.1:5673 guest:guest@192.168.0.2:5673

It is allowed to use a secure connection:

rmq_host = ssl://guest:guest@example.com:5673

To use an arbitrary port, it must be specified, otherwise the standard port 5672 will be used.

Before using a secure connection, it must be configured on each filtering node.

rmq_host_local
Connection parameters with the RabbitMQ service for local queue placement.

Example:

rmq_host_local = guest:guest@127.0.0.1

If the parameter is omitted, the following values will be used: guest:guest@127.0.0.1.


[training]
Learning process management section.
dataset_limit
Sets the maximum number of unique queries included in the training sample.

If the Nemesida WAF API are not configured yet, then the parameters api_uri and api_proxy can be specified later.

After making changes, restart the server or restart the service and check its operation:

# systemctl restart mlc_main rabbitmq-server memcached
# systemctl status mlc_main rabbitmq-server memcached

After the initial configuration of the module, you must check for errors in the component operation event logs /var/log/nwaf/mlc.log.

Managing settings using WebApp and API

To manage the settings of the Nemesida AI MLC settings, use Nemesida WAF Cabinet or API.

Additional information

For the component to work correctly, the following files and directories are located on the server:

  • /opt/mlc/mlc.conf – configuration file that allows you to perform the initial configuration of the component;
  • /opt/mlc/ml/ – the directory that contains:
    • all the behavioral models created by Nemesida AI MLC (files .ml);
    • training samples for the time of creating a behavioral model (files .db);
    • backups of the training sample used for retraining the behavioral model (backup directory);
    • requests (file mt.json), exported via the Nemesida WAF Cabinet;
    • files used by Nemesida AI MLC for automatic generation of specifications in the OpenAPI format (files openapi_*.json).

Operating in multipoint mode
The Nemesida AI MLC module supports a multipoint operation mode scheme when a single server with the Nemesida AI MLC module installed interacts with a variety of filtering nodes, including a variety of Nemesida WAF virtual clusters. To activate the operation in this mode, follow the settings below.

Filtering node

1. Allow access to RabbitMQ via the Nginx web server. To do this, add the appropriate entries to the /etc/nginx/nginx.conf file:

stream {
    server {
        listen 5673;
        proxy_pass 127.0.0.1:5672;
        allow x.x.x.x;
        deny all;
    }
}

where x.x.x.x — the IP address of the server where Nemesida AI MLC is installed.

2. Restart Nginx:

# nginx -t && service nginx reload

3. Allow access from the server on which the Nemesida AI MLC module is installed to the server with a filtering node on port 5673 (TCP).

Nemesida AI MLC

1. Move the /opt/mlc/mlc.conf file to the /opt/mlc/conf directory, rename it (for example, server-1.conf) and update the settings, including the connection settings to the RabbitMQ server located on the filtering node:

...
rmq_host = guest:guest@x.x.x.x:5673
...

or (if more than 1 filtering nodes is used in the virtual cluster Nemesida WAF):

...
rmq_host = guest:guest@x.x.x.x:5673 guest:guest@y.y.y.y:5673
...

where x.x.x.x/y.y.y.y are the IP addresses of the filtering nodes.

2. Restart the service:

# systemctl restart mlc_main
# systemctl status mlc_main

After configuration, you must check the error information contained in the component’s event logs for each server, for example: /var/log/nwaf/server-1/mlc.log.