A guide for installing, configuring and operating the Nemesida AI MLC machine learning module, designed for building behavioral models, as well as detecting brute force, flood and DDoS attacks.
Not used in Nemesida WAF Free.
Before installing Nemesida WAF components, add repository information to the system:
# apt install apt-transport-https gnupg2
# wget -O- https://nemesida-security.com/repo/nw/gpg.key | apt-key add - # apt update && apt upgrade
# apt install apt-transport-https gnupg2
The Nemesida AI module consists of Nemesida AI MLA modules (is included in the installation package of the Nemesida WAF module) and Nemesida AI MLC, whose interaction is possible in normal mode (modules operate on the same server) and mode “dot-multipoint” (the Nemesida AI MLC module operates on a dedicated server).
Python pip packages
For machine learning modules to work correctly, it is necessary to use unified versions of Python3 pip packages on servers with Nemesida AI MLA and Nemesida AI MLC installed.
Installation
Install Nemesida AI MLC:
# apt install nwaf-mlc
Install Nemesida AI MLC:
# apt install nwaf-mlc
Initial setup
After installing the module, it is necessary to make the initial configuration by specifying the following parameters:
After making changes, restart the server or restart the service and check its operation:
# service mlc_main restart # service mlc_main status
The cloud settings management functionality is enabled by default, but it can be disabled by sending a request to email.
In order to avoid errors when configuring the module, we recommend using a cloud WebApp.
Manage settings using a cloud WebApp and cloud API
Information about configuring Nemesida AI MLC using cloud WebApp and cloud API available in the relevant sections.
Manage settings using configuration files
Nemesida AI MLC
To configure the module, make the necessary changes to the configuration file /opt/mlc/mlc.conf. The /opt/mlc/mlc-example.conf file contains a complete list of available parameters.
After making changes, restart the server or restart the service and check its operation:
# service mlc_main restart # service mlc_main status
Working in Multipoint Mode
To build behavioral models, the Nemesida AI MLC module requires a significant amount of free RAM. When using more than one server with the Nemesida WAF module, you can save hardware resources by using the point-to-multipoint operation scheme (one server with the Nemesida AI MLC module installed interacts with many servers with Nemesida WAF modules installed).
On a server with the Nemesida WAF module installed
– Create a user of the RabbitMQ service:
# rabbitmqctl add_user USER PASSWORD # rabbitmqctl set_permissions -p / USER ".*" ".*" ".*"
where USER and PASSWORD are the username and password for connecting the Nemesida AI MLC module.
– Make changes to the configuration file /etc/rabbitmq/rabbitmq-env.conf:
NODE_PORT=5672 export RABBITMQ_NODENAME=rabbit@localhost export RABBITMQ_NODE_IP_ADDRESS=0.0.0.0 export ERL_EPMD_ADDRESS=127.0.0.1
– Allow access from the server on which the Nemesida AI MLC module is installed to the RabbitMQ port (by default 5672 TCP).
– Complete the RabbitMQ setup:
# service rabbitmq-server restart
On a server with the Nemesida AI MLC module installed
Create additional configuration files in the /opt/mlc/conf/ directory by copying the /opt/mlc/mlc.conf file. Make changes to the new configuration files to work with the remote RabbitMQ server. After making the changes, restart the service:
# service mlc_main restart # service mlc_main status
In additional configuration files
nwaf_license_keyis a required parameter. The license key used in the Nemesida AI MLC settings and the remote Nemesida WAFs must have the sameWAF ID. When using additional configuration files, it is recommended to delete the/opt/mlc/mlc.conffile.
Using remote RabbitMQ services, the Nemesida AI MLC module will collect queries and then train models in the same way as in normal operation.
Working with the Nemesida AI MLS cloud server
The Nemesida AI cloud server is designed to generate behavioral models based on a copy of traffic coming from remote servers. The cloud server is used in cases when the Nemesida WAF software user does not have enough RAM for the Nemesida AI MLC module to work. To use the capabilities of the Nemesida AI cloud server, contact the service technical support.
Accuracy of behavioral models
During the training period, in order to build better models, it is not recommended to scan the web application for vulnerabilities, as well as send other illegitimate requests. Immediately after the first training, it is recommended to retrain the models. False alarms are controlled using the module Nemesida WAF Signtest.
Storage of behavioral models
Behavioral models created by the Nemesida AI MLC module are transmitted to the remote Nemesida AI MLS server and automatically distributed to all running instances of Nemesida AI MLA and Nemesida AI MLC in accordance with the WAF ID.
Retraining of Nemesida AI models
To improve the accuracy of detecting attacks, it is recommended to retrain models once a week. To do this, you can use the cloud WebApp, cloud API or add to virtual value host symbol ^ when using configuration files.
Example:
vhosts_list = example.com^
After making the changes, restart the service:
# service mlc_main restart
After retraining the models, it is recommended to delete the exported BT 12 requests (the requests are contained on the False Positive page available on the web interface of the module Nemesida WAF Signtest). When training models for a virtual host, BT 12 requests will be included in the training sample and will not be required further.
To retrain behavioral models when using cloud services, you need to use them to set a list of virtual hosts.
Increasing the learning time of Nemesida AI behavioral models
The correct construction of models requires about 400,000-800,000 unique requests. By default, the training period is 4 days. The training period of the model can be changed directly by specifying the list of domain names in the vhosts_list parameter in mlc.conf: x:example.com , where x is the training period in days. For example, 5:example.com – the training of the model will last 5 days.
After making the changes, restart the service:
# service mlc_main restart
Additional training of models using a backup copy of the training sample
The correct construction of models requires about 400,000-800,000 unique requests. If the number of requests was insufficient during the training, then you can restart it and use the requests from the previous sample. To do this, follow these steps:
1. Stop the Nemesida AI MLC service:
# service mlc_main stop
2. Move the file /opt/mlc/ml/backup/[vhost].d_[timestamp], where [timestamp] is the date of creation of a backup copy of the training sample created by Nemesida AI MLC before starting the model construction, in /opt/mlc/ml/[vhost].d. For example, for the model example.com :
# mv /opt/mlc/ml/backup/example.com.d_1613587613 /opt/mlc/ml/example.com.d
3. Start the training. To do this, add the symbol ^ to the value of the virtual host.
Example:
vhosts_list = example.com^
Launch the Nemesida AI MLC service:
# service mlc_main start
After the end of the training period (the period can be changed), a behavioral model will be created based on queries from the general sample.
Removing Nemesida AI models
In case of incorrect training of behavioral models or significant changes in the web application that lead to a lot of false positives, it is recommended to delete the models. To do this, send a request using the settings management functionality using cloud API or use cloud WebApp manage settings.
Example of deleting a model using the cloud API:
# curl 'https://nemesida-security.com/nw/ml/mgmt/del_models_uri' --data 'key=1234567890&vhost=example.com'
where:
key– license key or its SHA256 hash;vhost– the name of the virtual host for which you want to delete the model.
Training sample for building behavioral models
Queries defined as BT 1, BT 2, BT 3 and BT 4 are not added to the training sample, even if they fall under the LM mode.
When the ai_extra mode is enabled, queries defined by the Nemesida AI MLC module as illegitimate are not added to the training sample.