Multi-Tenant RIN Installation Guide
Transcript of Multi-Tenant RIN Installation Guide
SNYPR 6.3.1
Multi-TenantRIN Installation Guide
Date Published: 3/22/2021
Securonix Proprietary Statement
This material constitutes proprietary and trade secret information of Securonix, and shall not be disclosed to any
third party, nor used by the recipient except under the terms and conditions prescribed by Securonix.
The trademarks, service marks, and logos of Securonix and others used herein are the property of Securonix or their
respective owners.
Securonix Copyright Statement
This material is also protected by Federal Copyright Law and is not to be copied or reproduced in any form, using any
medium, without the prior written authorization of Securonix.
However, Securonix allows the printing of the Adobe Acrobat PDF files for the purposes of client training and
reference.
Information in this document is subject to change without notice. The software described in this document is
furnished under a license agreement or nondisclosure agreement. The software may be used or copied only in
accordance with the terms of those agreements. Nothing herein should be construed as constituting an additional
warranty. Securonix shall not be liable for technical or editorial errors or omissions contained herein. No part of this
publication may be reproduced, stored in a retrieval system, or transmitted in any form or any means electronic or
mechanical, including photocopying and recording for any purpose other than the purchaser's internal use without
the written permission of Securonix.
Copyright © 2021 Securonix. All rights reserved.
Contact Information
Securonix
5080 Spectrum Drive, Suite 950W
Addison, TX 75001
(855) 732-6649
SNYPR Remote Ingestion Guide 2
Table of ContentsIntroduction 4
Server Recommendation 4Prerequisites 5
Remote Ingester Node Installation 8
Remote Management for RIN 19
Communication Flow 19Manage RIN in SNYPR 20Configure Syslog Filters() and Source() in Activity Import 23
Network Performance Tuning 27
Reference Server Configuration 27RIN Tuning Process 28Best Practices for Network Tuning 37Troubleshooting Common Errors 39
Uninstall the RIN 40
Troubleshoot the RIN 45
Appendix A: Configure Proxy Setup 59
Sample HTTP and HTTPS Settings 59
SNYPR Remote Ingestion Guide 3
Introduction
IntroductionThe Remote Ingestion Node (RIN) is a lightweight Java program that is used to forward logs, in real-time, from a remote server to the SNYPR ingestion nodes (also referred to as Kafka Brokers). Forwarding logs in real-time provides the ability to ingest and analyze events as soon as they are generated with a minimum delay.
The RIN offers you the following advantages:
l Forwards logs from various data centers and locations
l Compresses the data to reduce network bandwidth utilization
l Encrypts data to secure the communications
l Maintains a local cache and retransmits data in case of communications failure
Server RecommendationThe following table describes the RIN sizing recommendations for a small, medium and large configuration:
Recommendation Minimal (<1000 EPS)Small (1000 EPS)
Medium (5000 EPS)
Large (10000 EPS)
CPU 2 4 6 8
Memory (GB) 4 8 16 64
Storage (/Securonix) 100 GB 100 GB 1 TB 2 TB
Note: You can add additional storage if the data retention on the RIN server is
longer than 4 days. Additional CPUs may be required if large number of data
sources are scheduled.
SNYPR Remote Ingestion Guide 4
Introduction
Note: For a system to support large number of TCP connections, irrespective of the
EPS, 10 GB NIC is required.
Prerequisites You have to ensure that the following prerequisites are available before you install the RIN:
l Server Requirements: The RIN servers can be physical servers or virtual machines.
l Firewall Configuration: The firewall ports must be open for the RIN server to com-municate with SNYPR.
l General Requirement: The network validation utility must be installed and SELinux must be permissive.
l Root Permission: The RIN installation requires a non-root user account with sudo permission to install the Linux services.
The following table describes the prerequisites:
Requirement
TypeRequirement Description
Server Operating System CentOS 7 or Red Hat 7.x
Server Data Retention on the RIN
4 days
Firewall Ports SNYPR ConsoleOutbound port 80, 443, 8080, and 8443 must be open.
Firewall Ports KAFKA Brokers Outbound port 9092 or 9093 must be open.
SNYPR Remote Ingestion Guide 5
Introduction
Requirement
TypeRequirement Description
Firewall Ports RIN Syslog sources
Inbound port 514 must be open.
Note: Use TCP for Syslog sources to
improve the reliability of data
transfer.
General Network Validation Utility
tcptraceroute must be installed.
General SELinux Module
The SELinux module must be in the permissive
mode.
You can run getenforce as the root user to
check the status of SELinux.
If the status is not permissive, you can run
setenforce 0 as the root user to change it
to permissive.
SNYPR Remote Ingestion Guide 6
Introduction
Requirement
TypeRequirement Description
General Root Permission
You must have root permission to assign sudo
permission to a non-root user account.
To add a non-root user, such as securonix, with
sudo permission, enter the following command
in the Terminal application:
useradd securonix
passwd securonix
sudo usermod -aG wheel securonix
SNYPR Remote Ingestion Guide 7
Remote Ingester Node Installation
Remote Ingester Node InstallationThis section describes how you can install and configure the RIN for data collection. The RIN can be installed using the command line or Graphical User Interface (GUI).
Step 1: Pre Installation Setup
This section describes the following steps and checks that you must perform before installing RIN:
l Create a Securonix directory.
l Download the RIN Installer package.
l Run the prerequisite validation script to check if all the prerequisites are available.
Note: The IP and Hostname cannot be changed after the RIN installation.
Complete the following pre-installation tasks to prepare for the installation:
1. Access the Terminal application from your Linux server.
2. Create a directory with name as Securonix under the / (root) directory using the
following command:
sudo mkdir /Securonix
This step is required as syslog is deployed in the Securonix directory under the / root directory.
SNYPR Remote Ingestion Guide 8
Remote Ingester Node Installation
Note: Ensure that the Securonix directory is created by the user who has sudo
permission and is going to install RIN. For more information, see Root
Permission.
3. Perform the following steps if you are not logged in using the securonix account:
a. Assign ownership to the securonix account.
sudo chown -R securonix:securonix /Securonix
b. Log in as the securonix user.
su - securonix
c. Enter the password for the securonix account.
When you successfully login as the securonix user, securonix is displayed in the command prompt as highlighted below.
4. Run the following command to change the directory to /Securonix, if needed:
SNYPR Remote Ingestion Guide 9
Remote Ingester Node Installation
cd /Securonix
5. Download the RIN package and copy it in the /Securonix directory created in Step2.
For example: If you can are copying the RIN package from a system with Mac OS, then you will use SCP command.
scp SNYPR-RIN-{tenantname}.tar username@{IPaddress}:/username
If you are copying the RIN package from a system with Windows OS, then you have to use file transfer tools such as WINSCP.
The RIN package is in the tar format.
6. Untar (extract) the RIN package in the /Securonix directory using the following command:
tar xvf SNYPR-RIN-{tenantname}.tar
Note: The files are extracted in the RIN folder available at the following
location: /Securonix/SNYPR-RIN-{tenantname}/{tenantname}/. You have to
navigate this folder to run the validation script.
7. Run the following command to view the files available in the /Securonix directory:
ls /Securonix
SNYPR Remote Ingestion Guide 10
Remote Ingester Node Installation
The SNYPR-RIN-{tenantname} folder has the extracted RIN installation files.
8. Run the following command to change the directory to SNYPR-RIN-{tenantname}:
cd SNYPR-RIN-{tenantname}
The directory has changed to SNYPR-RIN-{tenantname} in the following screen:
9. Run the following command to view the files available in the /Securonix/SNYPR-RIN-{tenantname} folder:
ls
The SNYPR-RIN-{tenantname} folder is available:
SNYPR Remote Ingestion Guide 11
Remote Ingester Node Installation
10. Run the following command to change the directory to {tenantname}/RIN:
cd {tenantname}/RIN
11. Run the following command to view the files available in the RIN folder:
ls
The following files and folders are available inside the RIN folder:
12. Run the validation scrips to check if you have all the prerequisites available for RIN installation, using the following command:
SNYPR Remote Ingestion Guide 12
Remote Ingester Node Installation
sh validation.sh pre-check
Note: Ensure that you are running the command for RIN directory.
If prerequisites are not installed or available, the system generates an error. Refer to RIN Pre Installation Issues for information on error codes.
Step 2: Prep Installation Setup
This section describes the following steps and checks that you must perform before installing RIN:
l Validate the connectivity with SNYPR.
l Specify the RIN installation location.
Complete the following prep-installation tasks to prepare for the installation:
1. Ensure that you are in the following folder path in Terminal: /Securonix/SNYPR-RIN-{tenantname}/{tenantname}/RIN.
Note: You can run the pwd command to validate your current folder location.
2. Run the following command in Terminal:
sh validation.sh prepare-to-install
SNYPR Remote Ingestion Guide 13
Remote Ingester Node Installation
The command prompt displaysChoose Install Folde.
3. Type /Securonix and press Enter. The command prompt displays the confirmation message.
4. Type Y. The screen displays the Enter SUDO Password section.
5. Enter the non-root password of the securonix account and press Enter.
The system validates the connection details. If the connection is unsuccessful, the system generates an error. Refer to RIN Prep Installation Issues for information on error codes.
Step 3: RIN Installation
1. Run the following command to change the directory to /RIN, if needed:
cd /Securonix/SNYPR-RIN-{tenantName}/{tenantname}/RIN
2. Run the following command to launch the installer:
./<filename>.bin
The ingester installation file has .bin as suffix.
Note: You must install RIN as a non-root user with the sudo permission.
3. Press Enter. The installation starts.
SNYPR Remote Ingestion Guide 14
Remote Ingester Node Installation
Note: The SNYPR Console must be running and accessible on the network from
the server where the RIN is installed.
The command prompt displays the "The Installation of Ingester is complete" message once the installation is complete.
If the steps fail, refer to the Troubleshooting section for information on error codes.
Note: You can refer to the Readme files located at <INGESTER Install Loca-
tion>/README file for information on post installation steps. You can also
refer to installer logs from <INGESTER Install Location>/SNYPR_install-
ation/Logs/.
Step 4: Post Installation
Check the network connectivity by running the following command from Terminal:
1. Run the following command:
source /home/securonix/.bash_profile
2. Run the following command for post validation:
sh validation.sh post-check
SNYPR Remote Ingestion Guide 15
Remote Ingester Node Installation
3. Start the gateway with the following command:
sudo systemctl start scnx-gateway
4. Check the gateway status with the following command to confirm that it has started:
sudo systemctl status scnx-gateway
5. Start the Remote Ingester as a securonix user with the following command:
sudo systemctl start scnx-ingester
6. Check the Remote Ingester status with the following command to confirm that it
has started:
sudo systemctl status scnx-ingester
If there is an error or you want to check the Remote Ingester logs, use this
command:
tail -1234f /<your_installation_path>/logs/Ingester.log
7. Start the Syslog server as securonix user with the following command:
sudo systemctl start scnx-syslog-ng
8. Check the status of the Syslog server with the following command:
SNYPR Remote Ingestion Guide 16
Remote Ingester Node Installation
sudo systemctl status scnx-syslog-ng
You can check the logs for the Syslog server by using the following command:
journalctl -f -u scnx-syslog-ng -n 1234
Note: Ensure the directory is set to /Securonix before running the command.
If connection is unsuccessful, the system generates an error. Refer to the Troubleshooting section for information on error codes.
Note: If there is a proxy server configured between RIN and SNYPR application,
then you have to perform some additional settings. For information on proxy server
setup, refer to the Appendix A: Configure Proxy Setup section.
Step 5: Verify the RIN Connectivity to Console
When the RIN starts, it validates the token with the application. If the connection is successful, the Token Validated message is displayed. If the connection fails, the
Remote Ingester shuts down.
You can verify the RIN connectivity from the Menu > Administrator > Settings > Manage Ingesters.
SNYPR Remote Ingestion Guide 17
Remote Ingester Node Installation
SNYPR Remote Ingestion Guide 18
Remote Management for RIN
Remote Management for RINSNYPR provides remote management for the RIN with the new Gateway.
SNYPR Gateway
SNYPR Gateway is a remote access solution that enables you to control your application servers from anywhere in the world. It provides the ability to take selective actions on the edge-node applications used by SNYPR, directly from the console. This allows unified management and running of all SNYPR applications and services, as well as simplify maintenance of these services to enhance end-user experience.
SNYPR Gateway for the RIN provides the ability to start, stop, and restart the RIN from any location directly from SNYPR. This also allows you to access and download RIN logs so you can quickly collect logs for troubleshooting.
This section covers the following topics:
l Communication Flow
l Manage RIN in SNYPR
l Configure Syslog Filters() and Source() in Activity Import
Communication FlowThe diagram below shows how the SNYPR UI uses Gateway to communicate with the RIN:
SNYPR Remote Ingestion Guide 19
Remote Management for RIN
Manage RIN in SNYPRIn SNYPR, you can manage the status of one or multiple RINs using the Manage Ingesters screen. This screen shows individual ingester details, ingester status, and available actions per ingester.
To access this screen, navigate to Menu > Administration > Settings.
SNYPR Remote Ingestion Guide 20
Remote Management for RIN
Once ingester is registered in the SNYPRUI, it displays under the Manage Ingesters screen. Each ingester has a color code that reflects the status of the ingester. The color codes are:
l Green: The application is successfully connected to the ingester and is running.
l Yellow: The ingester status is refreshing.
l Red: The application failed to connect to the Ingester and has stopped.
These colors also display in the Ingesters header, which shows a quick view of the total, running, and stopped ingesters on the screen.
Understanding the Action IconsThe action icons on the Manage Ingesters screen are used to perform specific activities. The table below lists the action icons along with a brief description of its purpose:
Icon Name Description
Stop Stops the individual ingester.
Start Starts the individual ingester.
Restart Restarts the ingester.
DownloadDownloads the logs for individual ingester.
SNYPR Remote Ingestion Guide 21
Remote Management for RIN
Icon Name Description
Additional options
l Create new syslog source ()
l View syslog configuration
Refresh allRefreshes content for all the ingesters on the screen.
Creating a New Syslog Source()You can add or edit a new syslog source() for every ingester. To add a new syslog source for a specific ingester, click the Additional Options icon, then select Create/Edit Source.
A list of existing sources will display for the ingester. From here, you can:
l Add, edit, or delete a syslog-ng source () block
l Configure multiple sources ()
Click + Create new towards the bottom of the screen to add a syslog-ng source block. An Add new source section will display:
SNYPR Remote Ingestion Guide 22
Remote Management for RIN
On the Add new source screen you will specify the details for the new source. The Source name is used as an identifier in the source statement of the syslog-ng configuration file to receive log messages. The Source expression is used to build the
source statement.
Once you have completed the fields in this section, click Create.
Configure Syslog Filters() and Source() in Activity ImportYou can add / edit syslog filters per ingester for each datasource directly from the Activity Import screen. You also have the ability to select multiple syslog sources () for each datasource from the same screen.
To access this screen, navigate to Menu > Add Data > Activity. Click + > Add Data for Existing Device Type. For more information on how to add a datasource, see the Activity Data section in the Integration Guide.
SNYPR Remote Ingestion Guide 23
Remote Management for RIN
To configure a syslog filter, click + in the filters section. An Add New filter pop-up will display where you will specify the details for your filter. Once you have completed all the information in the pop-up, click Add.
The action icons on the Activity Import screen are used to perform multiple activities. The table below lists the action icons along with a brief description of its purpose:
Icon Name Description
AddAdds a new syslog filter per datasource.
Add Ingester Adds an ingester.
CollapseCollapses all configurations.
SNYPR Remote Ingestion Guide 24
Remote Management for RIN
Icon Name Description
Settings
There are three
options when you
click this icon,
including:
l Validate ingester: This option allows you to validate configurations for one or multiple ingesters.
l View logs: This option lets you view logs for the
ingester.
l Remove: This option lets you remove an ingester.
The changes you make are written to a specific, separate section in the syslogng.conf file that have been reserved for sources() and syslog filters() configured from the UI. Do not make changes to this section of the syslogng.conf file from the back-end
Any change made from the UI will override the changes made from the back-end.
SNYPR Remote Ingestion Guide 25
Remote Management for RIN
Ingesting Data from Multiple RINs You can ingest data coming from multiple RINs into the same datasource. For example, you can have Windows data from multiple data-centers and geographical locations, and ingest the data as part of the same datasource, so you can efficiently search and analyze related events.
To ingest data from multiple ingesters as part of the same datasource, the log format must be the same. If you have two different log formats for Windows, you must create two different datasources.
You also have the ability to add as many ingesters as needed. Once configured, you can validate the configurations for one or multiple ingesters.
The diagram below provides an overview of how multiple RINs (RIN1, RIN2, RIN3) are processed to the same datasource (RG1 and RG2):
Data for a single datasource must be published to the same Kafka RAW Topic from all the RINs as seen in the image above.
SNYPR Remote Ingestion Guide 26
Network Performance Tuning
Network Performance Tuning This topic explains how to tune your network for improved performance. You can perform network performance tuning at the time of RIN installation.
The configuration has been tested to support TCP connections for 3K - 5K hosts providing a continuous stream of data. The NIC on the server is 10GB to support the increased loads.
When data is forwarded from a SIEM to RIN, the number of TCP connections established are minimum (less than 50). In this scenario, high number of connections are not a bottleneck and aggressive tuning is not suggested. For high Events Per Second (EPS) environment, dedicated resources are used, for example CPU, RAM, and networking resources when referencing to the Virtual Machine environment.
Reference Server Configuration 1. Run the following command in Terminal to view you server configuration:
lscpu
2. Compare your server details with the following reference server configuration details:
Server Component / Setting Configuration Value
Architecture x86_64
CPU op-mode(s) 32-bit, 64-bit
Byte Order Little Endian
CPU(s) 8
On-line CPU(s) list 0-7
SNYPR Remote Ingestion Guide 27
Network Performance Tuning
Server Component / Setting Configuration Value
Thread(s) per core 1
Core(s) per socket 2
Socket(s) 4
NUMA node(s) 1
Vendor ID GenuineIntel
CPU family 6
Model 58
Model nameIntel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz
Stepping 0
CPU MHz 2700.000
BogoMIPS 5400.00
Hypervisor vendor VMware
Virtualization type full
L1d cache 32K
L1i cache 32K
L2 cache 256K
L3 cache 30720K
NUMA node0 CPU(s) 0-7
RIN Tuning Process The RIN tuning process consists of the following steps:
Server Preparation / OS Tuning
The server preparation is recommended to support the high EPS tuning. The steps
SNYPR Remote Ingestion Guide 28
Network Performance Tuning
performed are to tune the OS to support high number of client connections and high EPS.
It is recommended to:
l Add network monitoring tools to gather statistics and debug when errors are present.
l Install netstat, rpm -ivh net-tools-1.60-114.el6.x86_64.rpm.
For more rpm packages, refer to the following:
https://rpmfind.net/linux/rpm2html/search.php?query=%2Fbin%2Fnetstat
l Install ethtool, rpm -ivh ethtool-3.5-6.el6.x86_64.rpm.
For more rpm packages, refer to the following:
http://fr2.rpmfind.net/linux/rpm2html/search.php?query=ethtool
l Use dedicated resources to ensure optimal performance for the collectors.
l Set the latency sensitivity as High for the following scenarios:
l Unfiltered EPS> 10K
l Inbound TCP connections > 10
l Complex Filters
Note: You can set the latency setting from: Edit Settings > VM Options > Latency
Sensitivity.
Server Network Parameters Tuning
1. Edit sysctl.conf file using the following command:
SNYPR Remote Ingestion Guide 29
Network Performance Tuning
vi /etc/sysctl.conf
2. Add the following parameters to sysctl.conf:
SNYPR Remote Ingestion Guide 30
Network Performance Tuning
SNYPR Remote Ingestion Guide 31
Network Performance Tuning
3. Reload the sysctl configurations using the following command:
sysctl -p
4. Increase Transmit Queue Length for 10G NICs.
/sbin/ifconfig <interface where you will receive data>
txqueuelen 10000
5. Set the txqueuelen permanently:
SNYPR Remote Ingestion Guide 32
Network Performance Tuning
vi /etc/rc.local
Syslog-NG Tuning
l High EPS environment tuning: For Improving performance with lots of connections, use the following settings:
l max_connections = active_connections
l log_iw_size = number of active_connections * EPS
l log_fetch_limit = 10000
l flush_lines = 10000
l log_fifo_size = log_iw_size * 10
SNYPR Remote Ingestion Guide 33
Network Performance Tuning
SNYPR Remote Ingestion Guide 34
Network Performance Tuning
SNYPR Remote Ingestion Guide 35
Network Performance Tuning
l LoC EPS environment tuning: For improving performance with a few connections but high amount of traffic, use the following settings:
l log_iw_size = number of active connections * 100,000 or number of active con-nections * EPS whichever is greater
l log_fetch_limit = number of active connections * 100,000 or number of active connections * EPS whichever is greater
l log_fifo_size = log_fifo_size = log_iw_size * 10
l flush_lines = 10,000 or greater
SNYPR Remote Ingestion Guide 36
Network Performance Tuning
Best Practices for Network TuningData Collection
The fastest way the syslog-ng application can receive log messages from the network is using plain TCP transport with the network source driver. By default, syslog-ng runs
SNYPR Remote Ingestion Guide 37
Network Performance Tuning
in multi-threaded mode to scale to multiple CPUs or cores for the increased performance.
A TCP-based network source scales based on the number of active connections. This means that if there are 10 incoming TCP connections all coming to the same network source, then that source can use 10 threads, one thread for each connection.
Higher stats_level decreases the performance. For example, stats_level(2) means -10% in performance.
Data Processing and Filtering
Message processors, such as filters, rewrite rules, and parsers, are executed by the reader thread in a sequential manner. Simple filtering (for example, filtering on facility or tag) has no impact on performance at all. However , regular expressions, even simple ones, significantly decrease the message-processing rate by about 40-45%.
It is advised to use the simplest filters when filtering incoming messages. If a message can be filtered with several types of filters, check the measured data. A message when filtered with a regexp , the performance of syslog-ng can drop down to 55-60% of the original performance level. If the tag or facility filters are used, there is no decrease in performance.
When using multiple filters one after the other, or connecting filters with the logical AND/OR operators, the order of filters has a significant impact on performance. You can prioritize filters that are the most likely to match the incoming log messages to the top of the configuration file.
Data Connections
If there are several thousand active connections simultaneously, it is advised to place relay syslog-ng-s on another computer before the syslog-ng server. The volume of
SNYPR Remote Ingestion Guide 38
Network Performance Tuning
incoming messages is usually not significant but switching between active connections is time-consuming. You can use relays, since they are collecting the logs, to resolve this issue. The syslog-ng solution can easily handle lots of log messages sent from a few connections.
Note: For a system to support large number of TCP connections, irrespective of the
EPS, 10 GB NIC is required. The NIC bonding can be carried out if VM cannot
provide dedicated 10G NIC.
Troubleshooting Common ErrorsThis section explains the common errors during the tuning process:
Error Description & Resolution
Rx DropsSignifies that there is a network issue, for example: faulty
network, faulty cable, or bad interface.
Interface not sending ACK
Implies that there is a contention on the NIC and the NIC is unable to handle the load.
Files not getting created
Signifies that either there is a configuration error in syslog-ng or the environment's file handler limit has been met for the user who is creating the files.
SNYPR Remote Ingestion Guide 39
Uninstall the RIN
Uninstall the RINUse the RIN_uninstall.sh script to uninstall RIN.
1. Access the Terminal application from your Linux server.
2. Add the following command to change the directory to /Securonix:
cd /Securonix
3. Add the following command to access the folder where RIN_uninstall.sh is located:
cd Uninstall
4. Add the following command to run RIN_uninstall.sh:
./RIN_uninstall.sh
SNYPR Remote Ingestion Guide 40
Uninstall the RIN
5. Type yes for Are you sure you want to process and uninstall Remote Ingester?
and press Enter. The screen displays the Enter SUDO Password section.
6. Enter the password of the securonix account and press Enter. The uninstall process
starts.
SNYPR Remote Ingestion Guide 41
Uninstall the RIN
7. Add the following command to access bash_profile:
vi ~/.bash_profile
8. Locate the INGESTER_HOME entry.
SNYPR Remote Ingestion Guide 42
Uninstall the RIN
9. Press i on your keyboard to display the Terminal application in the Insert mode.
The Terminal application displays "Insert" when the mode is enabled.
SNYPR Remote Ingestion Guide 43
Uninstall the RIN
10. Delete export INGESTER_HOME=/Securonix/Ingester.
11. Press Esc to exit from the Insert mode.
12. Type :wq! to save the changes. The RIN is uninstalled.
SNYPR Remote Ingestion Guide 44
Troubleshoot the RIN
Troubleshoot the RINThis section highlights some common troubleshooting issues that may appear with the RIN on the SNYPR Console.
RIN Pre Installation Issues
Error CodeValidation
TypeTroubleshooting
RIN-PRE-001
Operating System version
Signifies that the operating system is not correct.
RIN Installer works only with CentOS 7 and Red
Hat 7.x .
RIN-PRE-002Check if running as root user
Signifies that the user is running the pre-
installation steps as a root user.
You have to switch to non-root user account using
the following command:
su <your_user_account_name>
For example :- su securonix
SNYPR Remote Ingestion Guide 45
Troubleshoot the RIN
Error CodeValidation
TypeTroubleshooting
RIN-PRE-003Local Firewall configuration
Signifies that ports are not open in firewall.
Check the firewall configuration to ensure that
ports are open.
If you want to turn off the firewall, then use the
following commands:
1. systemctl stop firewalld
2. systemctl disable firewalld
See Firewall Ports for more information.
SNYPR Remote Ingestion Guide 46
Troubleshoot the RIN
Error CodeValidation
TypeTroubleshooting
RIN-PRE-004selinux configuration
Signifies that the SELinux module is not in the
permissive mode.
l Option 1: You can run getenforce as the
root user to check the status of SELinux. If the
status is not permissive, you can run
setenforce 0 as a root user to change it to
permissive.
You must reboot the server to save the
changes by running sudo reboot.
l Option 2: As a root user execute the following
commands:
1. vi /etc/selinux/config
2. SELINUX=permissive
RIN-PRE-005rsyslog disable
As the root user execute the following commands:
1. systemctl stop rsyslog
2. systemctl disable rsysylog
SNYPR Remote Ingestion Guide 47
Troubleshoot the RIN
Error CodeValidation
TypeTroubleshooting
RIN-PRE-006
Securonix Directory
Signifies the user does not have permission or
ownership to access the /Securonix directory.
As the root user execute
1. mkdir -p /Securonix
2. chown INSTALLATION_USER:INSTALLATION_USER /Securonix
chmod 775 /Securonix
RIN-PRE-007
disk space
Signifies insufficient disk space. Ensure that you
have provided at least 10GB disk space to
/Securonix.
See Server Recommendation for more
information.
RIN-PRE-008
Source bash profile with user installing the service
Execute bash_profile and validate Ingester_Home
by using the following commands:
source ~/.bash_profile
echo $INGESTER_HOME
When you run the echo command, the command
prompt displays the installation path of the
Ingester.
SNYPR Remote Ingestion Guide 48
Troubleshoot the RIN
RIN Prep Installation Issues
Error CodeValidation
TypeTroubleshooting
RIN-PREP-001 Get Install Directory
Signifies either the installation directory is not
created or it does not have the correct ownership.
Ensure that you have specified the correct
directory name that you have created in the Step2,
/Securonix.
RIN-PREP-002Check Sudo Access
Signifies that the sudo password is incorrect.
Enter the sudo password, the non-root password of
the securonix account.
SNYPR Remote Ingestion Guide 49
Troubleshoot the RIN
Error CodeValidation
TypeTroubleshooting
RIN-PREP-003Connection Error
Signifies that there is a connection error between
RIN and SNYPR.
1. Open installer.properties using the following command:
vi installer.properties
2. Press i on your keyboard to display the Terminal application in the Insert mode. The Terminal application displays "Insert" when the mode is enabled.
3. Enter or verify the values for the settings in installer.properties.
4. Press Esc to exit from the Insert mode.
5. Type :wq! to save the changes.
6. Run the following command in Terminal:
sh validation.sh prepare-to-
instal
SNYPR Remote Ingestion Guide 50
Troubleshoot the RIN
RIN Post Installation Issues
Error CodeValidation
TypeTroubleshooting
RIN-POST-001
INGESTER_HOME set properly Ingester is Running
As the non-root user (securonix user) execute
the following commands:
l source ~/.bash_profile
l Start - sudo systemctl start scnx-ingester
l Status - sudo systemctl status scnx-ingester
l Stop - sudo systemctl stop scnx-ingester
l Restart - sudo systemctl restart scnx-
ingester
RIN-POST-002 Syslog Running
As the non-root user (securonix user) execute
the following commands:
l Start - sudo systemctl start scnx-
syslog-ng
l Status - sudo systemctl status scnx-syslog-ng
l Stop - sudo systemctl stop scnx-syslog-ng
l Restart - sudo systemctl restart scnx-syslog-ng
SNYPR Remote Ingestion Guide 51
Troubleshoot the RIN
Error CodeValidation
TypeTroubleshooting
RIN-POST-003
Gateway running
As the non-root user (securonix user) execute
the following commands:
l Start - sudo systemctl start scnx-gateway
l Status - sudo systemctl status scnx-gateway
l Stop - sudo systemctl stop scnx-gateway
l Restart - sudo systemctl restart scnx-gateway
SNYPR Remote Ingestion Guide 52
Troubleshoot the RIN
Error CodeValidation
TypeTroubleshooting
RIN-POST-004
Fetch Kafka Broker Details
Option1 : Perform the following steps:
1. Make sure Snypr application is running.
2. If firewall is running, ensure that the
firewall has port 443 open or https service
enabled.
firewall-cmd --list-all
If port is not open, then run the following
command:
firewall-cmd --permanent --add-
port=443/tcp
Option2: Perform the following steps if you
want to turn off the firewall:
l systemctl stop firewalld
l systemctl disable firewalld
RIN-POST-005 SNYPR Console Access
Signifies the SNYPR application is not running.
Ensure that the SNYPR application is running.
SNYPR Remote Ingestion Guide 53
Troubleshoot the RIN
Error CodeValidation
TypeTroubleshooting
RIN-POST-006
Check Kafka Broker Network Access
Option1 : Perform the following steps:
1. Make sure Snypr application is running.
2. If firewall is running, ensure that the
firewall has port 9093 open or https
service enabled.
firewall-cmd --list-all
If port is not open, then run the following
command:
firewall-cmd --permanent --add-
port=9093/tcp
Option2: Perform the following steps if you
want to turn off the firewall:
l systemctl stop firewalld
l systemctl disable firewalld
SNYPR Remote Ingestion Guide 54
Troubleshoot the RIN
Error CodeValidation
TypeTroubleshooting
NAAuthentication Checks
See Appendix A for the instructions to create
the ingestercloud.properties file.
l Token validation fails
l URL or token is not provided in the ingestercloud.properties file
NAKafka Publishing fails with SSL error
l If the Kafka Brokers are protected with SSL
and they are using self signed certificates,
ensure the following:
a. The truststore and SSL config file, sslconfig.properties, located in the INGESTER_HOME/conf folder must be configured to point to the
truststore.jks
b. The public keys of the Kakfa brokers or the signing certificate must be imported to the truststore.jks. See Appendix A for instructions.
l If the Kafka Brokers are configured with
mutual SSL authentication, a client
certificate must be imported into the
keystore for the Ingester. The SSL config
file, sslconfig.properties, located in the
INGESTER_HOME/conf folder, must be
configured to point to the ingester-
client.jks. See Appendix A for instructions.
SNYPR Remote Ingestion Guide 55
Troubleshoot the RIN
Error CodeValidation
TypeTroubleshooting
NAGateway is Down
You can check if the Gateway is running or
down from the SNYPR application. If the
Gateway is down, it is displayed in red color.
You can perform the following steps to
troubleshoot:
1. Check the gateway log by running the
following command from Terminal:
tail -1234f /<your_
installation_
path>/Gateway/logs/Gateway.lo
g
2. Review the logs to understand why the gateway is down.
3. Open the SSLConfig.properties file and verify if the Ingester path is correct. If it is incorrect, update the path.
4. Validate Ingester_Home by using the
following commands:
echo $INGESTER_HOME
When you run the echo command, the
command prompt displays the installation path
of the Ingester.
SNYPR Remote Ingestion Guide 56
Troubleshoot the RIN
RIN and Syslog Servers Issues
The RIN installer automatically starts the RIN and Syslog server once the installation is complete. This section explains how to start and stop services if there are any issues.
Follow these steps to start the RIN and the Syslog Service:
Note: When you are manually restarting the RIN, you must first restart the
Gateway using the following command: sudo systemctl start scnx-
gateway .
1. Start RIN as a securonix user with the following command:
sudo systemctl start scnx-ingester
2. Check the RIN status to confirm that it has started with the following command:
systemctl status scnx-ingester
If there is an error or you want to check RIN logs, use this command:
tail -1234f /$INGESTER_HOME/logs/Ingester.log
3. Start the Syslog server as a securonix user with the following command:
SNYPR Remote Ingestion Guide 57
Troubleshoot the RIN
lsudo systemctl start scnx-syslog-ng
To stop or check the status of the Syslog server, use the following command:
sudo systemctl stop scnx-syslog-ng
systemctl status scnx-syslog-ng
To check the logs for the Syslog server, use the following command:
journalctl -f -u scnx-syslog-ng -n 1234
RIN Log File Issues
To troubleshoot or examine the RIN log file, use this command:
tail -1234f <INGESTER_HOME>/logs/Ingester.log
Generally, the default log level is set to debug in the RIN log file. If you would like to define a custom log level, change the log4j2.xml log level to trace. The file is available
at INGESTER_HOME/conf/log4j2.xml.
SNYPR Remote Ingestion Guide 58
Appendix A: Configure Proxy Setup
Appendix A: Configure Proxy Setup If the RIN is behind a web proxy server, you have to configure the HTTP or HTTPs
parameters for the following RIN files:
l /etc/systemd/system/scnx-ingester.service
l /etc/systemd/system/scnx-gateway.service
l /Securonix/Ingester/bin/runingester.sh
l /Securonix/Gateway/rungateway.sh
Add or update the above files with the following information:
Parameters Type Description
HTTP
Add or update the following settings:
l Dhttp.proxyHost=<hostName>
l Dhttp.proxyPort=<portNumber>
HTTPS
Add or update the following settings:
l Dhttps.proxyHost=<hostName>
l Dhttps.proxyPort=<portNumber>
Sample HTTP and HTTPS SettingsBelow are the HTTP and HTTPS configurations:
Files HTTP Configuration Sample
Ingester Service file
ExecStart=/bin/sh -c '${JAVA_HOME} -
Dhttp.proxyHost=<hostName> -Dhttp.proxyPort=<portNumber>
-cp ${INGESTER_HOME}/lib/ingester-6.2.jar:${INGESTER_
HOME}/lib/* com.securonix.ingester.Main -mode:cloud'
SNYPR Remote Ingestion Guide 59
Appendix A: Configure Proxy Setup
Files HTTP Configuration Sample
Gateway Service file
ExecStart=/bin/sh -c '${JAVA_HOME} -
Dhttp.proxyHost=<hostName> -Dhttp.proxyPort=<portNumber>
-XX:+UseG1GC -XX:+UseStringDeduplication -cp ${GATEWAY_
HOME}/snypr-gateway-1.0.jar:${GATEWAY_HOME}/lib/*
com.securonix.snypr.gateway.SnyprGateway ${GATEWAY_
HOME}/conf'
runingester.sh file
/Securonix/Ingester/Java/jre/bin/java -
Dhttp.proxyHost=<hostName> -Dhttp.proxyPort=<portNumber>
-cp /Securonix/Ingester/lib/ingester-
6.2.jar:/Securonix/Ingester/lib/* com.securonix.ingester.Main -
mode:cloud
rungateway.sh file
/Securonix/Ingester/Java/jre/bin/java -Dhttp.proxyHost=<hostName> -Dhttp.proxyPort=<portNumber> -XX:+UseG1GC -XX:+UseStringDeduplication -cp snypr-gateway-1.0.jar:lib/* com.securonix.snypr.gateway.SnyprGateway conf
Files HTTPS Configuration Sample
Ingester Service file
ExecStart=/bin/sh -c '${JAVA_HOME} -
Dhttps.proxyHost=<hostName> -
Dhttps.proxyPort=<portNumber> -cp ${INGESTER_
HOME}/lib/ingester-6.2.jar:${INGESTER_HOME}/lib/*
com.securonix.ingester.Main -mode:cloud'
Gateway Service file
ExecStart=/bin/sh -c '${JAVA_HOME} -
Dhttps.proxyHost=<hostName> -
Dhttps.proxyPort=<portNumber> -XX:+UseG1GC -
XX:+UseStringDeduplication -cp ${GATEWAY_HOME}/snypr-
gateway-1.0.jar:${GATEWAY_HOME}/lib/*
com.securonix.snypr.gateway.SnyprGateway ${GATEWAY_
HOME}/conf'
SNYPR Remote Ingestion Guide 60
Appendix A: Configure Proxy Setup
Files HTTPS Configuration Sample
runingester.sh file
/Securonix/Ingester/Java/jre/bin/java -
Dhttps.proxyHost=<hostName> -
Dhttps.proxyPort=<portNumber> -cp
/Securonix/Ingester/lib/ingester-6.2.jar:/Securonix/Ingester/lib/*
com.securonix.ingester.Main -mode:cloud
rungateway.sh file
/Securonix/Ingester/Java/jre/bin/java -Dhttps.proxyHost=<hostName> -Dhttps.proxyPort=<portNumber> -XX:+UseG1GC -XX:+UseStringDeduplication -cp snypr-gateway-1.0.jar:lib/* com.securonix.snypr.gateway.SnyprGateway conf
SNYPR Remote Ingestion Guide 61