A guide for installing, initial configuration and operating the Nemesida AI MLC machine learning module, designed for building behavioral models, detecting brute-force attacks, flood and DDoS attacks and parasitic bots activity (for example, countering web scraping), as well as generating request scheme based on the OpenAPI specification.
Not used in Nemesida WAF Community Edition.
The Nemesida AI module consists of Nemesida AI MLA modules (is included in the installation package of the Nemesida WAF module) and Nemesida AI MLC, whose interaction is possible in normal mode (modules operate on the same server) and multipoint mode (the Nemesida AI MLC module operates on a dedicated server).
When certain types of events are detected, Nemesida AI MLC transmits the IP address and type of the detected event (for example, DDoS/Brute-force, etc.) to the service API (nemesida-security.com) to improve threat analysis.
Python pip packages
For machine learning modules to work correctly, it is necessary to use unified versions of Python3 pip packages on servers with Nemesida AI MLA and Nemesida AI MLC installed.
Setup component
1. Download script;
2. Run the installation script with the command:
# /bin/bash ./5-mlc-deploy.sh 'nwaf_lic_key=%Licence key%' 'api_url=%Nemesida WAF API URL%' 'rmq_endpoints=%RabbitMQ endoints info%' 'sys_proxy=%Proxy server address%' 'api_proxy=%Proxy server address%'
where:
nwaf_lic_key– a license key;api_url– the address of the server from which the Nemesida WAF API component will be accessed after its configuration (e.g.http://api.example.com:8080/nw-api/);rmq_endpoints– details of connecting to the RabbitMQ queue on filtering node (e.g.guest:guest@127.0.0.1);sys_proxy– (Optional) IP address of the proxy server for component access to external resources (e.g.http://proxy.example.com:3128);api_proxy– (Optional) IP address of the proxy server for interacting with the Nemesida WAF API (e.g.http://proxy.example.com:3128).
# apt update && apt upgrade # apt install apt-transport-https gnupg2 curl
Add the Nemesida WAF repository:
Install the packages:
# apt install python3 python3-venv python3-pip python3-dev python3-setuptools libc6-dev rabbitmq-server gcc memcached # apt install nwaf-mlc
During the installation of the module, the following pip packages are additionally installed:
wheel cython pandas simple-crypt pika logutils scikit-learn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize netaddr pymemcache genson pyarrow
# apt update && apt upgrade # apt install apt-transport-https gnupg2 curl
Add the Nemesida WAF repository and install packages:
Install the packages:
# apt install nwaf-mlc
During the installation of the module, the following pip packages are additionally installed:
wheel cython pandas simple-crypt pika logutils scikit-learn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize netaddr pymemcache genson pyarrow
# setenforce 0
then bring the file /etc/selinux/config to the form:
# This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of three two values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted
Add the Nemesida WAF repository by bringing the file /etc/yum.repos.d/NemesidaWAF.repo to the form:
[NemesidaWAF] name=Nemesida WAF Packages for RHEL baseurl=https://nemesida-security.com/repo/nw/rhel/$releasever/$basearch/ gpgkey=https://nemesida-security.com/repo/nw/gpg.key enabled=1 gpgcheck=1
Install the packages:
Install the Nemesida AI MLC:
# dnf install nwaf-mlc
During the installation of the module, the following pip packages are additionally installed:
wheel cython pandas simple-crypt pika logutils scikit-learn requests sqlalchemy fuzzywuzzy levmatch psutil config python-Levenshtein unidecode fsspec func_timeout url-normalize netaddr pymemcache genson pyarrow
For security reasons, it is recommended to prohibit external incoming connections to network services.
Pre-setup
For the correct operation of the Nemesida AI MLC module, a connection to the RabbitMQ installed on the filtering node is required. To do this, follow these steps on the server with the filtering node:
1. Allow access to RabbitMQ via the Nginx web server. To do this, add the appropriate entries to the /etc/nginx/nginx.conf file:
stream {
server {
listen 5673;
proxy_pass 127.0.0.1:5672;
allow x.x.x.x;
deny all;
}
}
where x.x.x.x — the IP address of the server where Nemesida AI MLC is installed.
2. Restart Nginx:
# nginx -t && service nginx reload
3. Allow access from the server on which the Nemesida AI MLC module is installed to the server with a filtering node on port 5673 (TCP).
Initial setup
After installing the module, it is necessary to make the initial configuration. The configuration file /opt/mlc/mlc.conf contains a list of available parameters of the Nemesida AI MLC module, which cannot be configured using Nemesida WAF Cabinet or API.
When using the automatic installation script, file editing is not required.
After making changes, restart the server or restart the service and check its operation:
# systemctl restart mlc_main rabbitmq-server memcached # systemctl status mlc_main rabbitmq-server memcached
After the initial configuration of the module, you must check for errors in the component operation event logs:
/var/log/nwaf/mlc.log;/var/log/rabbitmq/rabbit@%hostname%.log;/var/log/rabbitmq/rabbitmq-server.error.log.
Managing settings using WebApp and API
To manage the settings of the Nemesida AI MLC settings, use Nemesida WAF Cabinet or API.
Additional information
For the component to work correctly, the following files and directories are located on the server:
/opt/mlc/mlc.conf– configuration file that allows you to perform the initial configuration of the component;/opt/mlc/ml/– the directory that contains:- all the behavioral models created by Nemesida AI MLC (files
.ml); - training samples for the time of creating a behavioral model (files
.db); - backups of the training sample used for retraining the behavioral model (
backupdirectory); - requests (file
mt.json), exported via the Nemesida WAF Cabinet; - files used by Nemesida AI MLC for automatic generation of specifications in the OpenAPI format (files
openapi_*.json).
- all the behavioral models created by Nemesida AI MLC (files
Filtering node
1. Allow access to RabbitMQ via the Nginx web server. To do this, add the appropriate entries to the /etc/nginx/nginx.conf file:
stream {
server {
listen 5673;
proxy_pass 127.0.0.1:5672;
allow x.x.x.x;
deny all;
}
}
where x.x.x.x — the IP address of the server where Nemesida AI MLC is installed.
2. Restart Nginx:
# nginx -t && service nginx reload
3. Allow access from the server on which the Nemesida AI MLC module is installed to the server with a filtering node on port 5673 (TCP).
Nemesida AI MLC
1. Move the /opt/mlc/mlc.conf file to the /opt/mlc/conf directory, rename it (for example, server-1.conf) and update the settings, including the connection settings to the RabbitMQ server located on the filtering node:
... rmq_host = guest:guest@x.x.x.x:5673 ...
or (if more than 1 filtering nodes is used in the virtual cluster Nemesida WAF):
... rmq_host = guest:guest@x.x.x.x:5673 guest:guest@y.y.y.y:5673 ...
where x.x.x.x/y.y.y.y are the IP addresses of the filtering nodes.
2. Restart the service:
# systemctl restart mlc_main # systemctl status mlc_main
After configuration, you must check the error information contained in the component’s event logs for each server, for example: /var/log/nwaf/server-1/mlc.log.