Nemesida AI MLC | Nemesida WAF

A guide for installing, initial configuration and operating the Nemesida AI MLC machine learning module, designed for building behavioral models, detecting brute-force attacks, flood and DDoS attacks, as well as generating query schemes based on the OpenAPI specification.

Content

Installation
Configuring
Additional modes of operation of the Nemesida AI MLC module

Installation

Not used in Nemesida WAF Community Edition.

Before installing Nemesida WAF components, add repository information to the system:

DebianUbuntuCentOSDockerVirtual Appliance

# apt install apt-transport-https gnupg2 curl

Debian 9

# echo "deb https://nemesida-security.com/repo/nw/debian stretch non-free" > /etc/apt/sources.list.d/NemesidaWAF.list

Debian 10

# echo "deb https://nemesida-security.com/repo/nw/debian buster non-free" > /etc/apt/sources.list.d/NemesidaWAF.list

Debian 11

# echo "deb https://nemesida-security.com/repo/nw/debian bullseye non-free" > /etc/apt/sources.list.d/NemesidaWAF.list
# curl -s https://nemesida-security.com/repo/nw/gpg.key | gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/trusted.gpg --import
# chmod 644 /etc/apt/trusted.gpg.d/trusted.gpg
# apt update && apt upgrade

# wget -O- https://nemesida-security.com/repo/nw/gpg.key | apt-key add -
# apt update && apt upgrade

# apt install apt-transport-https gnupg2 curl

Ubuntu 20.04

# echo "deb [arch=amd64] https://nemesida-security.com/repo/nw/ubuntu focal non-free" > /etc/apt/sources.list.d/NemesidaWAF.list
# wget -O- https://nemesida-security.com/repo/nw/gpg.key | apt-key add -
# apt update && apt upgrade

Ubuntu 22.04

# echo "deb [arch=amd64] https://nemesida-security.com/repo/nw/ubuntu jammy non-free" > /etc/apt/sources.list.d/NemesidaWAF.list
# curl -s https://nemesida-security.com/repo/nw/gpg.key | gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/trusted.gpg --import
# chmod 644 /etc/apt/trusted.gpg.d/trusted.gpg 
# apt update && apt upgrade

CentOS 7

# rpm -Uvh https://nemesida-security.com/repo/nw/centos/nwaf-release-centos-7-1-6.noarch.rpm
# yum update
# yum install epel-release

CentOS 8 Stream

# rpm -Uvh https://nemesida-security.com/repo/nw/centos/nwaf-release-centos-8-1-6.noarch.rpm
# dnf update
# dnf install epel-release

CentOS 9 Stream

# rpm -Uvh https://nemesida-security.com/repo/nw/centos/nwaf-release-centos-9-1-6.noarch.rpm
# dnf update
# dnf install epel-release

Information about using Nemesida AI MLC in a Docker container is available in corresponding section.

Information about using Nemesida WAF in the form of Virtual Appliance (virtual disk for KVM/VMware/VirtualBox) and Yandex VM is available in corresponding section.

The Nemesida AI module consists of Nemesida AI MLA modules (is included in the installation package of the Nemesida WAF module) and Nemesida AI MLC, whose interaction is possible in normal mode (modules operate on the same server) and mode “dot-multipoint” (the Nemesida AI MLC module operates on a dedicated server).

Python pip packages
For machine learning modules to work correctly, it is necessary to use unified versions of Python3 pip packages on servers with Nemesida AI MLA and Nemesida AI MLC installed.

During analyse requests against their schema matching in OpenAPI format, all integer data formats are treated as type integer.

Installation

DebianUbuntuCentOS

Debian 10

# apt install python3 python3-venv python3-pip python3-dev python3-setuptools libc6-dev rabbitmq-server gcc memcached

Debian 11

# apt install python3 python3-venv python3-pip python3-dev python3-setuptools libc6-dev rabbitmq-server gcc memcached

Install Nemesida AI MLC:

# apt install nwaf-mlc

During the installation of the module, the following PIP packages are additionally installed:
wheel cython pandas simple-crypt pika logutils scikit-learn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize python-geoip-python3 python-geoip-geolite2 netaddr pymemcache genson

Ubuntu 20.04

# apt install python3 python3-venv python3-pip python3-dev python3-setuptools libc6-dev rabbitmq-server gcc memcached

Ubuntu 22.04

# apt install python3 python3-venv python3-pip python3-dev python3-setuptools libc6-dev rabbitmq-server gcc memcached

Install Nemesida AI MLC:

# apt install nwaf-mlc

CentOS 7

# yum update
# yum install centos-release-scl
# yum install rh-python38 rh-python38-python-pip rh-python38-python-devel rh-python38-python-setuptools gcc rabbitmq-server memcached
# yum install nwaf-mlc

CentOS 8 Stream

Add the RabbitMQ repository by bringing the file /etc/yum.repos.d/RabbitMQ.repo to the form:

[rabbitmq_erlang]
name = rabbitmq_erlang
baseurl = https://packagecloud.io/rabbitmq/erlang/el/8/$basearch
repo_gpgcheck = 0
gpgcheck = 0
enabled = 1

[rabbitmq_server]
name = rabbitmq_server
baseurl = https://packagecloud.io/rabbitmq/rabbitmq-server/el/8/$basearch
repo_gpgcheck = 0
gpgcheck = 0
enabled = 1

Install the package:

# dnf update
# dnf install rabbitmq-server

Check the correctness of the service:

# systemctl enable rabbitmq-server
# service rabbitmq-server restart
# service rabbitmq-server status

Install Nemesida AI MLC:

# dnf install gcc python39 python39-devel python39-setuptools python39-pip memcached
# dnf install nwaf-mlc

CentOS 9 Stream

Install the packages:

# dnf install dnf-utils
# dnf install centos-release-rabbitmq-38
# dnf install rabbitmq-server

Check the correctness of the service:

# systemctl enable rabbitmq-server
# service rabbitmq-server restart
# service rabbitmq-server status

Install Nemesida AI MLC:

# dnf install gcc python3 python3-devel python3-setuptools python3-pip memcached
# dnf install nwaf-mlc

Initial setup

After installing the module, it is necessary to make the initial configuration. The configuration file /opt/mlc/mc.conf contains a list of available parameters of the Nemesida AI MLC module, which cannot be configured using Nemesida WAF Cabinet or API.

mlc.conf parameters

Default parameter

Description of the parameter

[main]

The section responsible for the general settings of the Nemesida AI MLC module.

nwaf_license_key

Installing the Nemesida WAF license key when working on a dedicated server.

Usage example:

nwaf_license_key = 1234567890

sys_proxy

Configuring the proxy server address for accessing external resources (checking the license key, downloading the list of virtual hosts, etc.).

Example:

sys_proxy=http://proxy.example.com:3128

It is allowed to use authentication parameters when using a proxy server.

Example:

sys_proxy=http://<user>:<password>@proxy.example.com:3128

api_proxy

Configuring the proxy server address to access the Nemesida WAF API.

Example:

api_proxy=http://proxy.example.com:3128

It is allowed to use authentication parameters when using a proxy server.

Example:

api_proxy=http://<user>:<password>@proxy.example.com:3128

api_uri

The Nemesida WAF API address for sending information about the training status of models and information about detected anomalies. If the parameter value is empty, no information will be sent.

debug

Debug mode.

[run]

The section responsible for connection parameters with the RabbitMQ service.

rmq_host

Connection parameters with the RabbitMQ service.

It is allowed to use multiple values separated by a space.

Example:

rmq_host = guest:guest@192.168.0.1 guest:guest@192.168.0.2

It is allowed to use a secure connection:

rmq_host = ssl://guest:guest@example.com:5673

To use an arbitrary port, it must be specified, otherwise the standard port 5672 will be used.

Before using a secure connection, it must be configured on each server with the Nemesida WAF dynamic module installed.

rmq_host_local

Connection parameters with the RabbitMQ service for local queue placement.

Example:

rmq_host_local = guest:guest@127.0.0.1

If the parameter is omitted, the following values will be used: guest:guest@127.0.0.1.

[training]

Learning process management section.

dataset_limit

Sets the maximum number of unique queries included in the training sample.

If the Nemesida WAF API are not configured yet, then the parameters api_uri and api_proxy can be specified later.

After making changes, restart the server or restart the service and check its operation:

# systemctl restart mlc_main rabbitmq-server memcached
# systemctl status mlc_main rabbitmq-server memcached

After the initial configuration of the module, you must check for errors in the component event logs /var/log/nwaf/mlc.log.

Additional information

For the component to work correctly, the following files and directories are located on the server:

/opt/mlc/mlc.conf - configuration file that allows you to perform the initial configuration of the component;
/opt/mlc/ml/ - the directory that contains:
- all the behavioral models created by Nemesida AI MLC (files .ml);
- training samples for the time of creating a behavioral model (files .db);
- backups of the training sample used for retraining the behavioral model (backup directory);
- requests (file mt.json), exported via the Nemesida WAF Cabinet;
- files used by Nemesida AI MLC for automatic generation of specifications in the OpenAPI format (files openapi_*.json).

Managing settings using WebApp and API

To manage the settings of the Nemesida WAF dynamic module settings, use Nemesida WAF Cabinet or API.

Additional modes of operation of the Nemesida AI MLC module

Working in Multipoint Mode

To build behavioral models, the Nemesida AI MLC module requires a significant amount of free RAM. When using more than one server with the Nemesida WAF module, you can save hardware resources by using the point-to-multipoint operation scheme (one server with the Nemesida AI MLC module installed interacts with many servers with Nemesida WAF modules installed).

On a server with the Nemesida WAF module installed

Create a user of the RabbitMQ service:

# rabbitmqctl add_user USER PASSWORD
# rabbitmqctl set_permissions -p / USER ".*" ".*" ".*"

# rabbitmqadmin declare user name=USER password=PASSWORD tags=user
# rabbitmqadmin declare permission vhost=/ user=USER configure=".*" write=".*" read=".*"

where USER and PASSWORD are the username and password for connecting the Nemesida AI MLC module.

Make changes to the configuration file /etc/rabbitmq/rabbitmq-env.conf:

NODE_PORT=5672
export RABBITMQ_NODENAME=rabbit@localhost
export RABBITMQ_NODE_IP_ADDRESS=0.0.0.0
export ERL_EPMD_ADDRESS=127.0.0.1

Allow access from the server on which the Nemesida AI MLC module is installed to the RabbitMQ port (by default 5672 TCP). and complete the RabbitMQ setup:

# service rabbitmq-server restart

On a server with the Nemesida AI MLC module installed

Create additional configuration files in the /opt/mlc/conf/ directory by copying the /opt/mlc/mlc.conf file.

For example:

/opt/mlc/conf/server_1.conf
/opt/mlc/conf/server_2.conf
...

Make changes to the new configuration files to work with the remote RabbitMQ server. After making the changes, restart the service:

# systemctl restart mlc_main rabbitmq-server memcached
# systemctl status mlc_main rabbitmq-server memcached

In additional configuration files nwaf_license_key is a required parameter. The license key used in the Nemesida AI MLC settings and the remote Nemesida WAFs must have the same WAF ID. When using additional configuration files, it is recommended to delete the /opt/mlc/mlc.conf file.

Using remote RabbitMQ services, the Nemesida AI MLC module will collect queries and then train models in the same way as in normal operation.

After configuration, you must check the error information contained in the event logs for each server /var/log/nwaf/server_*/mlc.log.