Syslog RFC Standards: Parsing RFC 3164 and RFC 5424 for SOC Pipelines

Syslog is still the most common transport carrying firewall, router, proxy, and Unix host telemetry into a Security Operations Center — and it is also the format most likely to break a parser, because two incompatible RFCs share the same UDP port and the same leading <PRI> marker. Getting syslog wrong does not produce a clean error; it produces silently misclassified events — a dropped firewall deny parsed as informational noise, a hostname shifted one field to the left, a timestamp defaulted to ingest time so correlation windows slip. This page sits inside the broader SOC Log Architecture & Taxonomy pipeline as the parsing contract for the syslog source class: it builds a deterministic parser that distinguishes RFC 3164 from RFC 5424, decodes the priority byte safely, extracts structured data into real key-value pairs, and emits normalized events that downstream validation and correlation can trust.

Problem Framing

A single SOC syslog receiver on UDP/514 routinely sees three dialects interleaved on the same socket. A Cisco ASA emits BSD-style RFC 3164: <166>Mar 15 10:22:01 fw01 %ASA-6-302013: Built outbound TCP connection. A modern rsyslog relay emits RFC 5424 with structured data: <134>1 2024-03-15T10:22:01.123Z fw01 firewall 8842 ID47 [meta@32473 action="drop"] Connection blocked. A misconfigured appliance emits a truncated line with no hostname at all. A parser that assumes one format treats the other two as garbage — and “garbage” usually means the line is shoved into a message blob with severity defaulted to a constant, so an emergency (severity 0) and a debug (severity 7) become indistinguishable to the correlation engine.

The cost is concrete. Suppose the ASA above is reporting %ASA-1-... alerts (severity 1, alert) during an active intrusion while also emitting thousands of severity-6 connection-build messages per second. If the PRI byte is not decoded into facility and severity, every line carries the same weight; the high-severity alerts drown in informational volume, rate limiters shed them indiscriminately, and the alert correlation and rule engines downstream never see a clean severity signal to key on. Deterministic RFC parsing is what restores that signal: each line is unambiguously framed, its priority is split into the facility that drives routing and the severity that drives triage, and its timestamp is normalized to UTC before anything downstream tries to join it against other sources.

Prerequisites & Environment

The core parser targets Python 3.11+ and uses only the standard library — re, json, logging, and datetime — so the hot path carries no third-party dependency. Two optional libraries appear at the integration edges:

pip install "pydantic>=2.7"          # strict ECS validation downstream of the parser
pip install "python-dateutil>=2.9"   # robust RFC 3164 timestamp/timezone reconstruction

Operational assumptions:

• Syslog arrives framed as one event per line for UDP, or octet-counted per RFC 5425/6587 for TLS streams; this page parses the line payload after framing.
• All compiled regex patterns are module-level constants so the pattern is compiled once, not per event.
• Wall-clock normalization targets UTC unambiguously. RFC 3164 carries no year and no timezone, so those are reconstructed from receive context, never guessed silently.
• The parser is pure — it takes a str and returns a dict, performing no I/O — which keeps it trivially unit-testable and safe to run inside an asyncio worker pool.

Architecture Overview

The parser is a single stage between the syslog receiver and the normalization gate. Each line is classified RFC 5424 → RFC 3164 → unparseable, in that order, because the RFC 5424 pattern is strict enough that a false match against a 3164 line is effectively impossible (it requires the 1 version token immediately after <PRI>). A matched line is decoded, its PRI split, its timestamp coerced to UTC, and its structured data expanded; an unmatched line is routed to a dead-letter path with a typed error code rather than force-fit into a partial record.

Four invariants keep the parser safe, and every Troubleshooting entry below is a violation of one: (1) format detection is ordered and strict, so a 3164 line never half-matches the 5424 pattern; (2) the PRI integer is range-checked to [0, 191] before bit math, so a malformed marker cannot produce a negative facility; (3) every timestamp resolves to a timezone-aware UTC datetime, so no naive datetime ever reaches correlation; and (4) an unparseable line is dead-lettered with its original bytes intact, so a parse failure is distinguishable from a dropped event.

Step-by-Step Implementation

1. Compile strict RFC patterns and a disposition model

Define both RFC patterns as module-level compiled constants and an explicit error enum so every parse outcome is auditable. The RFC 5424 pattern anchors on the <PRI>VERSION pair; the RFC 3164 pattern anchors on the BSD Mmm dd hh:mm:ss timestamp shape.

#!/usr/bin/env python3
"""Deterministic syslog parser for SOC ingestion (RFC 3164 + RFC 5424)."""
from __future__ import annotations

import re
import json
import logging
from enum import Enum
from datetime import datetime, timezone
from typing import Optional, Any

logger = logging.getLogger("syslog_parser")


class SyslogError(str, Enum):
    UNRECOGNIZED = "ERR_SYSLOG_001"
    BAD_PRI = "ERR_SYSLOG_002"
    BAD_TIMESTAMP = "ERR_SYSLOG_003"
    BAD_STRUCTURED_DATA = "ERR_SYSLOG_004"


RFC5424_PATTERN = re.compile(
    r"^<(?P<pri>\d{1,3})>"
    r"(?P<version>\d{1,2})\s+"
    r"(?P<timestamp>\S+)\s+"
    r"(?P<hostname>\S+)\s+"
    r"(?P<appname>\S+)\s+"
    r"(?P<procid>\S+)\s+"
    r"(?P<msgid>\S+)\s+"
    r"(?P<sd>-|(?:\[[^\]]*\])+)"
    r"(?:\s+(?P<msg>.*))?$",
    re.DOTALL,
)

RFC3164_PATTERN = re.compile(
    r"^<(?P<pri>\d{1,3})>"
    r"(?P<timestamp>[A-Z][a-z]{2}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2})\s+"
    r"(?P<hostname>\S+)\s+"
    r"(?P<appname>[^\s:\[]+)(?:\[(?P<procid>\d+)\])?:\s*"
    r"(?P<msg>.*)$",
    re.DOTALL,
)

2. Decode the PRI byte into facility and severity

The <PRI> value encodes both fields as PRI = facility * 8 + severity. Decode it with bit operations after a strict range check — never with eval or string slicing — so a hostile marker cannot influence control flow.

def decode_pri(pri_str: str) -> dict[str, int]:
    """Split a syslog PRI into facility and severity, range-checked."""
    pri_int = int(pri_str)  # ValueError caught by the caller
    if not 0 <= pri_int <= 191:
        raise ValueError(f"PRI {pri_int} outside valid range [0, 191]")
    return {
        "pri": pri_int,
        "facility": pri_int >> 3,   # high bits
        "severity": pri_int & 0b111,  # low 3 bits
    }

Facility >> 3 yields 0–23 (0 = kernel, 4 = auth, 16–23 = local0–local7); severity & 7 yields 0 (emergency) through 7 (debug). The fixed mapping of those codes — and the routing decisions they drive — is covered in depth in best practices for syslog priority levels.

3. Expand RFC 5424 structured data into real key-value pairs

The original parser kept the structured-data block as an opaque string; for correlation it must become typed pairs. Each SD element is [SD-ID PARAM="value" ...]. Parse the ID and its parameters, unescaping the RFC 5424 escape sequences (\", \], \\).

_SD_ELEMENT = re.compile(r"\[(?P<id>[^\s\]]+)(?P<params>(?:\s+[^\s=]+=\"(?:\\.|[^\"\\])*\")*)\]")
_SD_PARAM = re.compile(r"(?P<key>[^\s=]+)=\"(?P<val>(?:\\.|[^\"\\])*)\"")


def parse_structured_data(sd: str) -> dict[str, dict[str, str]]:
    """Expand an RFC 5424 SD block into {sd_id: {param: value}}."""
    if sd == "-" or not sd:
        return {}
    out: dict[str, dict[str, str]] = {}
    for element in _SD_ELEMENT.finditer(sd):
        params = {
            m["key"]: m["val"].replace('\\"', '"').replace("\\]", "]").replace("\\\\", "\\")
            for m in _SD_PARAM.finditer(element["params"])
        }
        out[element["id"]] = params
    if not out:
        raise ValueError("structured data present but no SD elements parsed")
    return out

4. Normalize timestamps to timezone-aware UTC

RFC 5424 timestamps are ISO 8601 and may carry an offset; RFC 3164 timestamps carry neither year nor zone, so reconstruct them from the receive time. Always return a timezone-aware datetime — a naive one silently breaks every cross-source time window.

def normalize_timestamp(ts: str, received_at: datetime) -> datetime:
    """Coerce a syslog timestamp to a timezone-aware UTC datetime."""
    if "T" in ts or "-" in ts[:10]:  # RFC 5424 ISO 8601
        dt = datetime.fromisoformat(ts.replace("Z", "+00:00"))
        if dt.tzinfo is None:
            dt = dt.replace(tzinfo=timezone.utc)
        return dt.astimezone(timezone.utc)
    # RFC 3164: "Mmm dd hh:mm:ss" — reconstruct year + assume UTC
    dt = datetime.strptime(ts.strip(), "%b %d %H:%M:%S").replace(
        year=received_at.year, tzinfo=timezone.utc
    )
    # Handle a December log received in January (year rollover)
    if dt - received_at > _ONE_DAY:
        dt = dt.replace(year=received_at.year - 1)
    return dt


_ONE_DAY = __import__("datetime").timedelta(days=1)

5. Assemble the normalized event and dead-letter failures

Tie the steps together. A matched line yields an ECS-shaped dict; any failure returns a structured dead-letter record carrying the original payload and a typed error code rather than a half-populated event.

def parse_syslog(raw_line: str, received_at: Optional[datetime] = None) -> dict[str, Any]:
    """Parse one syslog line into a normalized ECS-shaped event or a DLQ record."""
    received_at = received_at or datetime.now(timezone.utc)
    line = raw_line.rstrip("\n")
    rfc = "5424"
    match = RFC5424_PATTERN.match(line)
    if not match:
        match, rfc = RFC3164_PATTERN.match(line), "3164"
    if not match:
        return _dead_letter(line, SyslogError.UNRECOGNIZED, "no RFC pattern matched")

    g = match.groupdict()
    try:
        pri = decode_pri(g["pri"])
    except ValueError as exc:
        return _dead_letter(line, SyslogError.BAD_PRI, str(exc))
    try:
        ts = normalize_timestamp(g["timestamp"], received_at)
    except ValueError as exc:
        return _dead_letter(line, SyslogError.BAD_TIMESTAMP, str(exc))
    try:
        sd = parse_structured_data(g.get("sd") or "") if rfc == "5424" else {}
    except ValueError as exc:
        return _dead_letter(line, SyslogError.BAD_STRUCTURED_DATA, str(exc))

    return {
        "@timestamp": ts.isoformat(),
        "event": {"kind": "event", "severity": pri["severity"], "original": line},
        "log": {"syslog": {
            "facility": {"code": pri["facility"]},
            "severity": {"code": pri["severity"]},
            "priority": pri["pri"],
            "version": int(g["version"]) if rfc == "5424" else 0,
            "structured_data": sd,
        }},
        "host": {"hostname": g.get("hostname") or "unknown"},
        "process": {"name": g.get("appname") or "unknown",
                    "pid": int(g["procid"]) if (g.get("procid") or "").isdigit() else None},
        "message": (g.get("msg") or "").strip(),
        "observer": {"vendor": "syslog", "rfc": rfc},
    }


def _dead_letter(line: str, code: SyslogError, detail: str) -> dict[str, Any]:
    logger.warning("syslog parse failed", extra={"error_code": code.value, "detail": detail})
    return {"_dlq": True, "error_code": code.value, "detail": detail, "raw": line}


if __name__ == "__main__":
    samples = [
        '<134>1 2024-03-15T10:22:01.123Z fw01 firewall 8842 ID47 '
        '[meta@32473 action="drop" src="192.168.1.50"] Connection blocked',
        "<14>Mar 15 10:22:01 web01 sshd[1234]: Accepted publickey for admin from 10.0.0.5",
        "INVALID SYSLOG LINE",
    ]
    for s in samples:
        print(json.dumps(parse_syslog(s), default=str))

Schema & Validation Integration

The parser emits fields named for Elastic Common Schema — @timestamp, event.severity, host.hostname, process.name, log.syslog.* — so a syslog event and a JSON cloud event line up on the same field names downstream. That alignment is the whole point: it is what lets JSON event normalization and syslog parsing converge on one event model that correlation rules can query without per-source special cases. The dict this stage returns is intentionally not yet trusted — it flows into the strict schema validation pipelines, where a Pydantic ECS model enforces types (severity is an int 0–7, @timestamp is timezone-aware, process.pid is int | None) before the event reaches the correlation tier.

Structured-data parameters are the high-value payload of RFC 5424 and deserve explicit mapping: a [meta@32473 action="drop" src="192.168.1.50"] block should be lifted into event.action and source.ip so it can be cross-referenced during enrichment. Those extracted indicators are exactly what threat intel feed mapping consumes when it scores a parsed source.ip against STIX/TAXII feeds. Where flat or batch-exported syslog arrives as delimited text rather than line-framed events, the CSV ingestion patterns guidance handles the column-to-field coercion before the same ECS normalization applies.

Error Handling & DLQ Routing

Every line that does not parse cleanly produces a labeled, replayable dead-letter record rather than a silent drop or a partial event — because in security telemetry a quietly malformed line can be an evasion attempt (deliberately broken framing to slip past a parser under T1562.006 Indicator Blocking). Codes follow the established ERR_CATEGORY_NNN convention:

Lines that parse structurally but carry an unexpected SD-ID or an out-of-range field are early signals of vendor schema drift — route those into the error categorization framework so the drift is classified and surfaced rather than absorbed into the message blob. The non-negotiable rule is that a parse failure never defaults severity or timestamp to a placeholder that the correlation engine would treat as real signal; an unparseable line is held in the DLQ with its original bytes so a corrected pattern can replay it without data loss.

Performance Tuning

Three factors govern syslog parser throughput, and each has a measured, not guessed, target:

• Pattern ordering and compilation. Patterns are compiled once at module load; matching RFC 5424 first costs one extra regex attempt on every 3164 line, so on a fleet that is overwhelmingly BSD-syslog (legacy network gear), reverse the order and test 3164 first. Profile with the real source mix — the cheaper-first ordering can shift parse cost by 20–30% on a 100k events/second receiver.
• Avoid catastrophic backtracking. The SD pattern uses [^\]]* and the message uses .* with re.DOTALL anchored at end-of-line, so neither nests a quantifier inside a quantifier. A naive (\[.*?\])* against an unterminated bracket is the classic ReDoS that lets one malformed line stall a worker — keep the character-class negation that makes each step linear.
• Worker model. The parser is pure CPU with no I/O, so an asyncio single thread saturates one core; partition the receive socket across a process pool (one parser per core) when parsing — not framing or validation — becomes the bottleneck. A single typed parser comfortably clears well over 50,000 lines/second/core on commodity hardware.

Memory stays flat because the parser allocates only the result dict per line and holds no per-source state. The dominant cost downstream is the structured-data dict for chatty RFC 5424 sources; if SD parameters are mostly unused, gate parse_structured_data behind a per-source flag.

Verification & Observability

Confirm correctness along three axes — coverage, fidelity, and disposition:

• Coverage. Replay a captured PCAP or a labeled corpus of both RFC dialects and assert every line either yields an event or a DLQ record — never an exception that escapes the worker:

def test_no_uncaught_failures():
    corpus = [
        '<134>1 2024-03-15T10:22:01Z fw01 fw 1 ID47 [m@1 a="drop"] blocked',
        "<14>Mar 15 10:22:01 web01 sshd[1234]: Accepted publickey",
        "<999>broken", "garbage", "",
    ]
    for line in corpus:
        out = parse_syslog(line)
        assert "_dlq" in out or out["log"]["syslog"]["priority"] is not None
    # An otherwise-valid line with an out-of-range PRI is dead-lettered, not crashed
    bad_pri = "<999>1 2024-03-15T10:22:01Z fw01 fw 1 ID47 - msg"
    assert parse_syslog(bad_pri)["error_code"] == "ERR_SYSLOG_002"

• Fidelity. Spot-check that facility/severity round-trip: decode_pri("134") must yield facility=16 (local0), severity=6 (info), and re-encoding 16*8+6 must return 134. Assert timestamps are timezone-aware (dt.tzinfo is not None) on every parsed event.
• Disposition. Export counters for parsed_total, dlq_total partitioned by error_code, and rfc_version (3164 vs 5424). A sudden rise in ERR_SYSLOG_001 means a new source dialect or a framing change; a rise in ERR_SYSLOG_003 means a source clock or locale regression. These same parsed events feed identity and network joins in cross-source event linking, so a drop in parse fidelity shows up there as missing correlation matches.

Troubleshooting

• RFC 5424 lines are being dead-lettered as ERR_SYSLOG_001. The relay is emitting the BOM or a multi-line MSG that breaks line framing before the parser sees it. Confirm framing is octet-counted (RFC 6587) for TLS streams rather than newline-delimited.
• Hostnames and app names are shifted one field. A 3164 source omitted the hostname (non-compliant but common on appliances), so the parser read the app name into hostname. Add a source-specific pre-check or a relaxed 3164 variant for that vendor rather than loosening the global pattern.
• Every event shows the same year and the timestamps drift in January. The RFC 3164 year reconstruction is firing without the rollover guard, dating December logs to the new year. Verify the dt - received_at > _ONE_DAY correction is present.
• Severity is always 0 or always 7 for one source. The device is hard-coding its PRI (a known quirk of some IoT and appliance firmwares), so decode_pri is faithfully reporting a constant. This is a source configuration problem, not a parser bug — flag it and weight that source’s severity from message content instead.
• A single malformed line stalls a parser worker. A regex is backtracking on an unterminated bracket or quote. Confirm the SD and message patterns use bounded character classes ([^\]], [^\"\\]) and never a .* nested inside another quantifier.

FAQ

Related

• SOC Log Architecture & Taxonomy — parent architecture
• Best practices for syslog priority levels
• JSON Event Normalization
• CSV Ingestion Patterns
• Threat Intel Feed Mapping