A Comprehensive Approach to Intrusion Detection Alert Correlation

• Background Knowledge and Insights
• Goals and Contributions
• Data
• Feature
• Architecture
  • Components
• Alert Process
  • Meta-alerts
  • Example Attack Scenario
  • Alert Normalization
  • Alert Preprocessing
  • Alert Fusion
  • Alert Verification
  • Attack Thread Reconstruction
  • Attack Session Reconstruction
  • Attack Focus Recognition
  • Multistep Correlation
  • Impact Analysis
  • Alert Prioritization
• References

Background Knowledge and Insights

• How to cover multiple aspects "as a whole"
  • Different attack manifestations
    • Network packets
    • OS calls
    • Audit records
    • Application logs
  • Different types of intrusion detection
    • Host vs network
    • IT environment (e.g., Windows vs Linux)
    • Levels of abstraction (e.g., Kernel level vs application level)

Goals and Contributions

• Aggregate outputs of multiple IDSs
• Filter out irrelevant alerts
• Provide succinct view of security-related activity on the network

Data

• Two existing data sets
  • DARPA
    • MIT Lincoln Laboratory 1999
    • MIT Lincoln Laboratory 2000
  • Defcon 9
  • Shortcoming
    • Created to evaluate IDS, not include sensor alerts
    • Not real environment, not real time
    • Lack of network health monitoring information
      • Difficult to determine the actual impact of the attacks
    • Not include anomalous traffic, non-anomalous behavior
• Three additional data sets
  • Two honeypot systems
    • RedHat 7.2 Linux
    • Microsoft Windows 2000 Server
  • Cyber Treasure Hunt competition
  • Rome Air Force Research Laboratory's networks
    • No successful attacks
• Manually perform the correctness of the correction process

Feature

Architecture

• Optional
• Parallel
• Feedback as input

Components

• Normalization
  • Translate alerts to a common format
  • Alerts from different sensors be encoded in different formats
• Preprocessing
  • Augment normalized alerts by assigning meaningful values to all alert attributes
    • Start time
    • End time
    • Source
    • Target
• Fusion
  • Combine alerts representing the same attack by different IDSs
• Verification
  • Determine the success of the single attack corresponding to the alert
  • Decrease the influence of the failed attacks
• Thread reconstruction
  • Combine series of alerts due to attacks by a single attacker against a single target
• Session reconstruction
  • Associate network-based alerts and host-based alerts
• Focus recognition
  • Identify hosts that are source or target of many attacks
    • DoS
    • port scanning
• Multistep correlation
  • Identify common attack patterns
    • Sequence of individual attacks at different points of network
      • Island hopping attack
• Impact analysis
  • Determine the attack impact for the specific target network
• Prioritization
  • Assign priorities to alerts

Alert Process

Meta-alerts

• Definition
  • Higher-level alerts made via merging
  • Attribute values derived from those of original alerts
• Example
  • a "portscan" alert composed of a series of alerts referring to individual network probe packets
  • Target attribute, all hosts that were port-scanned
• Representation
  • A tree with IDS alerts at the leaves
  • Merging done in a BFS fashion

Example Attack Scenario

• Victim network
  • Vulnerable Apache Web service on a Linux host (IP: 10.0.0.1)
  • Host-based IDS (H)
  • Application-based IDS (A): monitors Apache Web logs for malicious activity
  • Two different network-based IDSs (N1 and N2)
• Attack process
  • Attacker (IP: 31.3.3.7) first portscans host
    • Discovers vulnerable Apache server (Alerts 2, 3)
  • During scan a worm (IP: 80.0.0.1) attempts Microsoft IIS exploit and fails (Alert 1)
  • After scan, attacker exploits Apache buffer overflow (Alerts 4, 5)
    • Gets interactive shell as Apache user
  • Using a local exploit against linuxconf, attacker becomes root (Alerts 6, 7)
• Desired output of correlation
  • Single meta-alert for a multi-step attack against victim host
  • Step 1: Initial scanning (Alerts 2, 3)
  • Step 2: Remote attack against web server (Alerts 4, 5)
  • Step 3: Privilege escalation (Alerts 6, 7)
• Alert 1 should be discarded as irrelevant

Alert Normalization

• Goal: Unify alert formats form different types of sensors
  • Not need to normalize for only one type of sensor
• Intrusion Detection Message Exchange Format (IDMEF)
  • Proposed by the Internet Engineering Task Force
• Implemented using wrapper modules for different IDSs

Alert Preprocessing

• Some necessary fields are omited
  • Start/End-time
  • Attck Source/Target
  • Additional Information
• Goal: Supply missing alert attributes as accurately as possible
  • Use several heuristics

Alert Fusion

• Goal: Combine alerts representing independent detection of a same attack by different IDSs
  • Not combine alerts produced by the same sensor
• Fusion: Temporal difference between alerts and information they contain
  • Keep sliding time window of alerts
    • Low window size causes related alerts to escape fusion
  • Alerts within the time window stored in a time-ordered queue
  • Upon new alert, compared to alerts in queue
  • Match if all overlapping attributes are equal and new alert is produced by a different sensor
  • Upon a match, alerts are merged; resulting meta-alert replaces the matched alert in the queue
    • New timestamp: earlier start/end-time

Alert Verification

• Three types Alert
  • True positive
  • Irrelevant positive
    • Failed attack
  • False positive
• Goal: Extending intrusion detection signatures with an expected "outcome" of the attack
  • Real-time verification
  • Visible and verifiable traces left by attack
  • Example: temporary file, outgoing connection
• Passive techniques
  • Depend on the priori information
    • Whether the attack target
      • Exist
      • Running
      • Reachable
      • Reassemble the packets as expected by the intruder
  • Advantage
    • Not interfere with the normal operation
  • Disadvantage
    • Knowledge base/priori information undated not in time
    • Limitation of the type of information
• Active techniques
  • Looking for the evidence of the success of an attack
    • Established network connection
    • Unknown ports opened (backdoor)
  • Vulnerability scanners
    • Advantage
      • Information is current
    • Disadvantage
      • Visible on the network
      • Consume network resources
      • Cause crash
      • Raise false alarm
  • Remote login
    • Advantage
      • Gather high-quality data
    • Disadvantage
      • Configure the target machine

Attack Thread Reconstruction

• Goal: Combines a series of alerts due to attacks by one attacker against a single target
• Merging alerts with equivalent source and target attributes in temporal proximity
  • Set time windows: 120s
• Reduce a lot of alerts caused by brute-force attack

Attack Session Reconstruction

• Goal: Link network-based alerts to related host-based alerts
• Rough spatial and temporal correspondence between the alerts
  • The host-based attack occurs a short time after the network-based attack
    • Simple but imprecise
  • Priori information
    • Prepare for or to be a precondition for another attack
    • Manually encoded in a knowledge base
  • Sliding time window with extended timeout model
    • New related alert arrives, extend timeout

Attack Focus Recognition

• Goal: identify hosts that are either the source or the target of a substantial number of attacks
  • DDoS: many2one
  • Port scan: one2many

Multistep Correlation

• Goal: identify high-level attack patterns that are composed of several individual attacks
• Example
  • Scans a victim host
  • Breaks into a user account on that host
  • Escalates privileges to the root

Impact Analysis

• Goal: determine the impact of an attack on the proper operation of the protected network
• Heartbeat monitor

Alert Prioritization

• Classify alerts
• Discard irrelevant or less importance alerts

References

• A Comprehensive Approach to Intrusion Detection Alert Correlation, Valeur et al, 2004
• Lecture 20