Log Review and SIEM Tools Analysis

Log Review and SIEM Tools

Reviewing logs is crucial for keeping your systems secure. Instead of just looking at logs after something bad happens, you should regularly review them to spot potential threats early on.

Different systems create logs in different ways, so you need a system that can collect and analyze logs from all your devices. This helps you identify patterns and spot unusual activity that could signal a problem. By proactively analyzing logs, you can prevent security breaches and respond quickly to any issues that arise.

Security incidents aren’t always obvious. Sometimes, combining seemingly harmless events can reveal a bigger problem.

Example: Imagine Ben, a sales rep traveling to an another country, logs into the company network from one city. Shortly after, the system records her entering the office in another city. Individually, these events might seem normal. However, together they suggest a potential security issue, like someone might be impersonating Ben.

SIEM (Security Information and Event Management) systems help analyze these events in real-time. They collect logs from different sources, look for patterns, and alert you to potential threats.

To use SIEM effectively:

• Focus on the right logs: Collect important information without overwhelming the system.

• Clearly define what you’re looking for: Determine what constitutes a threat for your company.

• Have a plan for responding to alerts: Know what to do when the system flags a potential issue.

• Regularly review logs: Look for threats that might have slipped past the initial alerts.

SIEM helps you proactively identify and respond to security threats, making your organization more secure.

Splunk

Spliuk is a powerful tool used to collect and analyze large amounts of data from your computer systems and applications. It can gather information from various sources like operating systems and software, and store it for later analysis.

Splunk uses a special language (SPL) to search this data, allowing you to identify patterns and trends. The results can be displayed in easy-to-understand dashboards and reports, or even be used to trigger alerts when something suspicious is found.

Splunk can be installed on your own computers (local enterprise software) or accessed through the cloud. There are different versions available depending on your needs, including a free option for smaller networks and a security-focused edition with pre-built features for identifying and responding to security threats.

ELK Stack (Elastic Stack)

The ELK Stack, now known as the Elastic Stack with the addition of Beats, is a collection of open-source tools that provide Security Information and Event Management (SIEM) functionality. Here’s a breakdown of its components:

• Elasticsearch: This is the engine behind the ELK Stack. It’s a powerful search and analytics tool that allows you to query and analyze the vast amount of data collected by the system.

• Logstash: This tool acts as the central nervous system of the ELK Stack. It collects logs from various sources, parses them into a common format, and then ships them to Elasticsearch for storage and analysis.

• Kibana: This is the visualization tool of the ELK Stack. It allows you to create dashboards and reports to easily understand the data collected by Logstash and stored in Elasticsearch. You can use Kibana to identify trends, patterns, and potential security threats.

• Beats: These are lightweight agents that can be deployed on various devices (endpoints) to collect logs and send them to Logstash for further processing.

The ELK Stack is a versatile platform that can be implemented locally on your own servers or as a cloud-based service.

ArcSight

ArcSight, now owned by HP through Micro Focus, is a commercial SIEM solution that provides log management, analytics, and security intelligence. It goes beyond basic SIEM functionalities by offering features like:

• Compliance reporting: ArcSight helps you generate reports to demonstrate compliance with various regulations such as HIPAA, SOX, and PCI DSS, which is crucial for many organizations.

• Cybersecurity threat response: ArcSight provides tools and insights to help you identify, investigate, and respond to cybersecurity threats effectively.

QRadar

QRadar is IBM’s SIEM platform that offers log management, analytics, and compliance reporting functionalities. Similar to ArcSight, it provides a comprehensive solution for organizations looking to strengthen their security posture.

Alien Vault and OSSIM (Open-Source Security Information Management)

Alien Vault’s OSSIM (Open-Source Security Information Management) is a free and open-source SIEM product. Alien Vault also offers commercial versions of OSSIM with additional features and support. Here’s what makes OSSIM stand out:

• Open-source integration: OSSIM can integrate with other popular open-source security tools like Snort (intrusion detection system) and OpenVAS (vulnerability scanner), allowing you to build a comprehensive security ecosystem.

• Web-based management: OSSIM provides a user-friendly web interface for managing the entire security environment, simplifying administration tasks.

Graylog

Graylog is another open-source SIEM solution with a focus on enterprise needs. It also offers a commercial version that caters to compliance requirements and supports IT operations and DevOps teams.

Security Data Collection and Use Cases

Security intelligence loses value quickly, making real-time or near-real-time analysis crucial. Timely insights can minimize or even prevent damage from attacks. 1 However, gathering and analyzing security intelligence is a complex process. It involves identifying relevant data, collecting it from various sources, transforming it into a usable format, aggregating and correlating different data points, analyzing the data to identify security patterns, and finally, determining appropriate responses to these identified threats. 2 This process often involves many tedious and time-consuming tasks.

SIEMs automate much of the security intelligence cycle, particularly data collection and processing, providing faster insights than manual methods. They can even automate some analysis, production, and dissemination tasks.

Effective SIEM implementation starts with careful planning. Collecting all logs is counterproductive. Focus on security-relevant events identified through risk management analysis. Excessive data overwhelms the SIEM, creates unnecessary network traffic, and increases analysis workload.

Early SIEMs were complex and often generated more noise than value. False positives and negatives are common. To mitigate these, develop “use cases” – specific conditions to be reported. For example, suspicious log-ons from privileged accounts to high-value assets. Use cases define data sources, correlation queries, and response actions.

Key considerations for use cases include:

• When: Event start and end times.

• Who: Involved users or entities.

• What: Specific details of the event.

• Where: Host, file system, network port, etc.

By carefully defining use cases, you can optimize your SIEM for maximum effectiveness and minimize the burden of managing security alerts.

Making Sense of Security Data: Security Data Normalization

Security data comes from various sources like network devices, servers, and user machines. This raw data often needs processing before it becomes useful for identifying security threats. Imagine searching for a specific book in a library. If all the books were piled haphazardly, finding the right one would be very difficult. Similarly, security data analysis requires organization for efficient threat detection.

This organization process is called security data normalization. It involves reformatting and restructuring security data to make it easier to analyze. Normalization ensures consistency across data formats, allowing for efficient pattern recognition and anomaly detection. This can be done manually by security analysts or through automation using SIEM (Security Information and Event Management) systems.

SIEMs collect data from various sources using different methods:

1. Agent-based collection: A lightweight agent software is installed on each device. This agent monitors the device for security events, filters and aggregates the relevant data, and then sends it to the central SIEM server for analysis and storage. Examples include Elastic Stack Beats agents for application logs and OSSEC for intrusion detection data.

2. Listener/collector method: Instead of agents, devices can be configured to send data directly to the SIEM server using protocols like syslog or SNMP (Simple Network Management Protocol). The SIEM server then parses and normalizes this data from various sources.

3. Sensor-based collection: SIEMs can also collect network traffic data using sensors deployed at strategic points on the network. These sensors capture and forward network traffic information to the central SIEM management system for analysis.

By collecting and normalizing data from various sources, SIEMs provide a holistic view of your security posture, helping you identify and respond to potential threats more effectively.

Parsing and Normalization: Making Security Data Speak the Same Language

Security data analysis is crucial for identifying and responding to potential threats. However, this analysis becomes a challenge when data comes from various sources, each with its own format. Imagine trying to understand a conversation where everyone speaks a different language. Security data normalization solves this problem by standardizing data formats, allowing SIEM (Security Information and Event Management) systems to effectively analyze information from diverse sources.

The Data Format Challenge

Security data comes in a multitude of formats, including:

• Proprietary binary formats: Specific to a particular software or device.

• Delimited formats: Separated by tabs (TSV) or commas (CSV).

• Database log storage systems: Structured data stored in databases.

• syslog: A common format for network device logs.

• SNMP (Simple Network Management Protocol): Used for network device configuration and monitoring.

• XML (eXtensible Markup Language) or JSON (JavaScript Object Notation): More structured formats gaining popularity.

These formats can vary in readability. Some are human-readable plain text, while others require specialized tools or parsers. Even seemingly simple differences like line endings (Linux vs Windows) or character encoding (ASCII, ANSI, Unicode) can create parsing challenges.

SIEM Connectors and Parsing: Unlocking the Code

SIEM solutions overcome these hurdles using connectors or plug-ins. These act as translators, interpreting (parsing) logs from different systems and accounting for vendor-specific variations. Parsing typically involves regular expressions, which are powerful tools for pattern matching in text. SIEMs use these regular expressions tailored to each log format to identify specific data points (attributes and content) within the logs. These data points are then mapped to standard fields within the SIEM’s reporting and analysis tools.

Date and Time Synchronization: Putting Events in Order

Another challenge is timestamp consistency. Devices might have incorrect internal clocks, use different time zones, or record timestamps in non-standard formats. This makes it difficult to correlate events and reconstruct the sequence of activities. To ensure accurate timeline analysis, it’s crucial to synchronize all logging sources to a common time reference using Network Time Protocol (NTP). SIEMs also need to handle variations in time zones and daylight saving time consistently. If automatic correction isn’t possible, enforcing a standard timestamp format (like RFC 3339) across all logging sources can help mitigate this issue.

Security and Resource Considerations for SIEMs

Security is paramount when dealing with logs. SIEMs need secure channels to prevent attackers from tampering with data in transit. Additionally, the SIEM data store itself must adhere to the CIA triad:

• Confidentiality: Only authorized users should be able to access log data.

• Integrity: Data must be protected from unauthorized modification.

• Availability: Security measures shouldn’t hinder authorized access to log data for analysis.

Finally, logging can be resource-intensive. Large organizations can generate massive amounts of data (gigabytes or even terabytes per hour). This requires significant storage capacity (disk space) and processing power (CPU and memory) to handle data analysis. SIEM deployments need to be adequately resourced to ensure efficient log collection, storage, and analysis.

By addressing these data format challenges, time synchronization considerations, and security and resource requirements, SIEMs can effectively transform raw security data into actionable intelligence, empowering organizations to proactively identify and respond to security threats.

Event Logs: Unveiling the System Story

Event logs are the silent chroniclers of a system’s activity, recording every interaction between users, software, and the operating system itself. These logs serve as a vital source of security information, providing insights into potential threats and system health.

Understanding Log Formats and Content

The structure and content of event logs vary depending on the operating system. Common platforms like Windows, macOS, and Linux each have their own logging mechanisms. Often, the type of information captured within these logs can be customized to meet specific needs.

Events are categorized based on their nature and the area of the operating system they impact. Here’s a breakdown of the five main categories of Windows event logs:

• Application: Tracks activity from applications and services, including failures like a service that can’t start.

• Security: Records security-related events, such as failed login attempts or denied file access.

• System: Captures events generated by the operating system and its core services, like monitoring storage health.

• Setup: Logs events related to the Windows installation process.

• Forwarded Events: Contains events sent from other computers to the local machine.

Within some of these categories, events are further classified by severity:

• Information: Records successful events.

• Warning: Flags potential issues that might become problems in the future.

• Error: Indicates significant problems that can impact system functionality.

• Audit Success/Failure: Unique to the Security log, these events track whether a user or service complied with system access policies.

Centralized Logging with syslog

For non-Windows systems, syslog reigns supreme. Designed for a client-server model, syslog facilitates the centralized collection of events from diverse sources. It also uses an open format for event messages, making it a widely adopted standard for logging in distributed systems. Devices like Cisco routers, switches, servers, and workstations can all generate syslog messages that are then collected in a central database for analysis. syslog leverages TCP/IP protocol and typically operates on UDP port 514.

By understanding event log formats, content, and management tools like syslog, security professionals gain valuable insights into system behavior. These insights are crucial for identifying security threats, troubleshooting system issues, and maintaining overall system health.