Mukesh Asthana
Automation

Automating production log tracing and reporting with Python

By Mukesh Asthana, Database Architect · · 7 min read

In most support teams, someone starts the day by opening log files, searching for errors, copying lines into a spreadsheet and e-mailing a summary. It's slow, it depends on who's on shift, and it misses patterns that only show up across days. On one engagement, replacing that routine with Python scripts reduced the manual effort by about 80%. Here's the approach.

Stream, don't load

Production logs can run to gigabytes. Reading them line by line with a generator keeps memory flat, and Python opens gzip-rotated files just as easily.

import gzip, re
from pathlib import Path

LINE = re.compile(
    r"^(?P<ts>\d{4}-\d{2}-\d{2}[ T]\d{2}:\d{2}:\d{2})\S*\s+"
    r"(?P<level>ERROR|WARN|FATAL)\s+(?P<msg>.*)$"
)

def read_lines(path: Path):
    opener = gzip.open if path.suffix == ".gz" else open
    with opener(path, "rt", errors="replace") as fh:
        for line in fh:
            m = LINE.match(line)
            if m:
                yield m.groupdict()

Group errors by signature

Ten thousand lines of "ORA-00060: deadlock detected" with different session IDs are one problem, not ten thousand. Normalising the variable parts of a message produces a signature that can be counted.

NOISE = [
    (re.compile(r"\b\d{4,}\b"), "<n>"),                 # ids, pids, row counts
    (re.compile(r"[0-9a-f]{8}-[0-9a-f-]{27}"), "<uuid>"),
    (re.compile(r"'[^']*'"), "'<v>'"),                  # quoted values
]

def signature(msg: str) -> str:
    for rx, repl in NOISE:
        msg = rx.sub(repl, msg)
    return msg[:160]

With signatures in place, a collections.Counter gives you the top problems of the day, and comparing today's counter with yesterday's shows what's new or growing. "New since yesterday" is the line support leads actually read.

Correlate across systems

An error in the application log is far more useful next to the matching database or middleware entry. If the systems share a correlation or transaction ID, index one log by that ID and join the other to it. If they don't, a time window of a few seconds around each error is a workable fallback, and a good argument for adding correlation IDs in the next release.

Publish a report people will open

  • A short summary first: total errors, new signatures, the top five by count, and any FATAL lines.
  • A CSV attachment with every signature, count, first-seen and last-seen time, for anyone who wants to filter.
  • A link to raw lines for each top signature, so engineers can jump straight to context.

Schedule it with cron (or the job scheduler you already run) shortly before the support stand-up, and send it to the team channel or mailbox.

# crontab: weekdays at 07:30
30 7 * * 1-5 /opt/logtrace/venv/bin/python /opt/logtrace/run.py --since 24h --mail support-team

Guardrails

  • Mask personal data (phone numbers, account numbers, e-mail addresses) before anything leaves the server.
  • Run read-only with a service account that can read logs and nothing else.
  • Fail loudly: if the script finds no log files, that's an alert, not an empty report.
  • Version the patterns in Git so the team can add new signatures without editing code.

Automation doesn't remove the engineer from the loop. It removes the searching, so the engineer starts the day at the diagnosis instead of the grep.

The same pipeline extends naturally: push counts into a database table and chart them in Power BI, or hand the day's top signatures to an n8n workflow that raises tickets automatically when a threshold is crossed.

More articles