qarbaal

Extract, normalize, validate, and export indicators of compromise from text and files.

Documentation: https://ahur4.github.io/Qarbaal/

qarbaal reads reports, logs, and message files on your machine and returns classified indicators. It refangs defanged text, records where each value came from, and writes CSV, JSON, STIX 2.1, MISP, OpenIOC, and network rules. Results are pydantic models, and indicator types are IOCType members. File readers beyond the standard library are optional extras. It does not call an enrichment API.

Installation

Python 3.10 or newer.

pip install qarbaal

The base install depends on pydantic. HTML, PDF, Word, YAML, and packet-capture readers stay optional.

Installing optional extras

Text, JSON, CSV, XML, EML, and MBOX are read with the standard library. The other readers are extras, so a text-only install does not pull them in.

One extra:

pip install "qarbaal[html]"

Several at once:

pip install "qarbaal[html,pdf,docx]"

Every optional extra:

pip install "qarbaal[all]"

Input	Extra
Text, JSON, CSV, XML, EML, MBOX	none
HTML	html
PDF	pdf
DOCX	docx
YAML	yaml
PCAP	pcap

A missing extra raises OptionalDependencyError and names the install command. Extraction results are pydantic models, and indicator.type is an IOCType member.

Quickstart

from qarbaal import IOCExtractor, IOCType, extract

report = """
From: support[at]corp-it-desk[.]com
Lure: hxxps://login[.]corp-it-desk[.]com/session
Beacon: 185.220.101.42
MD5: d41d8cd98f00b204e9800998ecf8427e
"""

result = extract(report)
for indicator in result:
    print(indicator.type, indicator.value, indicator.tags, indicator.line)

extractor = IOCExtractor(
    extract_types=[IOCType.IPV4, IOCType.DOMAIN, IOCType.URL, IOCType.EMAIL],
    normalize=True,
    validate=True,
    deduplicate=True,
)
same = extractor.extract(report, source="ticket-14")

extract() uses the default type set. File paths, mutex names, and User-Agent strings stay out of that set until you ask for them with extract_types or "all".

A synthetic incident is included:

from qarbaal import SAMPLE, extract

print(extract(SAMPLE).stats.counts)

What you get back

Each indicator keeps three strings:

Field	Meaning
value	First-seen sieve form. Hashes are lower case, CVE IDs are upper case.
raw	Exact slice of the source, which may still be defanged.
normalized	Canonical form used for the fingerprint.

It also carries:

• Provenance on every sighting: source label, line, character offset, and page number for PDF.
• A SHA-256 fingerprint of the type and the normalized value. raw, value, and normalized are all kept.
• Local enrichment only. Address flags, a best-effort domain split, and URL parts are computed on the machine. Nothing is sent to a network service.
• Deduplication and cross-source correlation. occurrences counts sightings. Extraction.shared() lists indicators that appear in more than one named source.
• Custom PatternSpec patterns, allow and deny globs, and custom validators.
• Streaming through extract_iter for large text, with offsets preserved across chunks.
• Noise tags, a structural confidence heuristic, and pivot links you open yourself.

Exports are CSV, JSON, JSONL, a defanged list, STIX 2.1, MISP, OpenIOC, and Suricata and Snort rules for addresses, domains, and URLs. Pass path and filename to write a file and receive an ExportFile.

Using qarbaal in Python

These calls use the public API. Pass IOCType members such as IOCType.IPV4 and IOCType.URL. extract returns a pydantic Extraction, and each indicator is a pydantic model. Nothing below contacts a network service.

Quick extraction

extract uses the default type set. Paths, mutex names, and User-Agent strings stay out until you ask for them.

from qarbaal import IOCType, extract

text = "Beacon 185.220.101.42 and hxxps://login[.]evil[.]example/a"
result = extract(text)

for indicator in result.indicators:
    print(indicator.type, indicator.value)
    print(indicator.raw, indicator.normalized)
    print(indicator.confidence, indicator.fingerprint)
    print(indicator.sources, indicator.occurrences, indicator.valid)
    print(indicator.validation_notes)
    for sighting in indicator.sightings:
        print(
            sighting.source,
            sighting.line,
            sighting.offset,
            sighting.end_offset,
            sighting.page,
        )
    print(indicator.enrichment)

ips = result.get(IOCType.IPV4)
print(result.values(IOCType.IPV4))
print(
    result.stats.counts,
    result.stats.total,
    result.stats.sightings,
    result.stats.input_characters,
)

result is an Extraction. result.indicators is the deduplicated tuple in export order. Iterating result walks that same tuple.

result.get(IOCType.IPV4) returns the indicators of one type. result.values(IOCType.IPV4) returns their sieve strings. indicator.type is an IOCType member. result.stats is an ExtractionStats: counts (unique indicators per type), total, sightings, and input_characters, plus truncated, match_limit_hit, and sources. Call model_dump() when you want a dictionary.

This is the beginning of the README. Read the rest at the original. Read the full README ↗