During Nginx operation, errors may occur, leading to an emergency interruption of the execution of requests, as a result of which Nemesida WAF cannot process the request correctly. To track and correct errors in the code, the process memory image recording analysis (core dump) is used.

Setup

To activate the tracking functionality and record information about core dump, you need to perform the following actions:

1. Create a directory to store records about core dump:

# mkdir /var/log/nginx/core_dumps
# chown root:root /var/log/nginx/core_dumps
# chmod 1777 /var/log/nginx/core_dumps

2. Remove the limit on the maximum file size with the process memory image:

# ulimit -c unlimited

If the command ends with the message "Cannot modify limit: operation not allowed", run the command:

# sh -c "ulimit -c unlimited && exec su $LOGNAME"

3. Activate the recording of messages in the system:

# echo "/var/log/nginx/core.%e.%p" | tee /proc/sys/kernel/core_pattern
# sysctl -w fs.suid_dumpable=2
# sysctl -p

4. Activate message recording in Nginx:

4.1 At the beginning of the Nginx configuration file /etc/nginx/nginx.conf add:

worker_rlimit_core      2G;
working_directory       /var/log/nginx/core_dumps/;

4.2 Save the changes and restart Nginx:

# systemctl restart nginx
Additional features
Activate writing to the file of requests that are in the memory of the workflow at the time of the error related to core dump by adding the parameter nwaf_coredump_request_path to the configuration file of the dynamic module Nemesida WAF /etc/nginx/nwaf/conf/global/nwaf.conf:

nwaf_coredump_request_path /var/log/nginx/coredump_requests;

When using the parameter, each Nginx workflow will be allocated 25 MB of memory for intermediate storage of requests. After filling 75% of the allocated amount of memory for each request is allocated by 64 KB, truncating the request body.

Activation of registration of records about core dump and requests stored in the memory of the workflow significantly affect the performance of Nginx and generate large files, so it is recommended to disable the functions after receiving the necessary information.

Analysis of the received data

To view the collected records about core dump, you need to use the debugger GDB by running the command gdb path/to/nginx -c path/to/core_dumps:

Output example
# gdb /usr/sbin/nginx -c /var/log/nginx/core_dumps/core.XX.YY
...
[New LWP XXXX1]
[New LWP XXXX2]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `nginx: worker process                   '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00005600c259077a in ngx_radix32tree_find ()
[Current thread is 1 (Thread 0x7f6ca08e9740 (LWP XXXXX))]
(gdb)

Execute the command bt (backtrace – output of the function call stack until the error occurs):

Output example
(gdb) bt
#0  0x00005600c259077a in ngx_radix32tree_find ()
#1  0x00005600c25ffd84 in ?? ()
#2  0x00005600c25ce948 in ngx_http_get_indexed_variable ()
#3  0x00005600c25cf7e6 in ngx_http_script_copy_var_len_code ()
#4  0x00005600c25cfa9a in ngx_http_complex_value ()
#5  0x00005600c2603428 in ?? ()
#6  0x00005600c25ce948 in ngx_http_get_indexed_variable ()
#7  0x00005600c25cea47 in ngx_http_get_flushed_variable ()
#8  0x00005600c25d1783 in ngx_http_script_var_code ()
#9  0x00005600c2604956 in ?? ()
#10 0x00005600c25be8ec in ngx_http_core_rewrite_phase ()
#11 0x00005600c25ba0be in ngx_http_core_run_phases ()
#12 0x00005600c25ba163 in ngx_http_handler ()
#13 0x00005600c25c4d08 in ngx_http_process_request ()
#14 0x00005600c25c5284 in ?? ()
#15 0x00005600c25c55c4 in ?? ()
#16 0x00005600c25c576f in ?? ()
#17 0x00005600c25ac12d in ?? ()
#18 0x00005600c25a2996 in ngx_process_events_and_timers ()
#19 0x00005600c25aa5d9 in ?? ()
#20 0x00005600c25a8cd2 in ngx_spawn_process ()
#21 0x00005600c25a9874 in ?? ()
#22 0x00005600c25ab100 in ngx_master_process_cycle ()
#23 0x00005600c2583ae2 in main ()
(gdb)