PHMSA Pipeline Incident Extractor
Paste a free-text pipeline incident narrative and get back a structured record: what failed, what was done, what resulted, why it happened and how severe it was. A retrieval layer answers questions across the whole incident corpus.
0.61Entity F1, 840 held-out spans
69%Relation precision, 62 of 90
0.78Severity macro-F1
0.75Cause macro-F1
01The problem
The problem
The US Pipeline and Hazardous Materials Safety Administration collects incident reports under 49 CFR Parts 191 and 195. The most useful detail, what failed, why and what happened next, sits in free-text narratives that nobody can query or aggregate.
02The approach
The approach
- Schema first: 12 entity types, 6 relation types, 4 severity levels and 8 cause categories, versioned before any labelling.
- Weak supervision from PHMSA’s own structured fields produced labels at scale, refined with stratified annotation in Label Studio across gas distribution, gas transmission and hazardous liquids.
- Baselines before transformers: rule-based, spaCy statistical and fine-tuned transformer NER compared per commodity.
- Three DistilBERT models handle entities, severity and cause, with relation extraction on top.
- An “Ask the Data” retrieval layer answers questions across the extracted corpus.
03Results
Results
| Task | Score | Evaluated on |
|---|---|---|
| Entity tagging | F1 0.61 | 840 held-out spans |
| Relations | 69% precision | 62 of 90 sampled relations |
| Severity | Macro-F1 0.78 | 1,921 held-out narratives |
| Cause | Macro-F1 0.75 | 1,921 held-out narratives |
Reading the numbers: narrative-level classification is strong. Span-level tagging is harder because incident text is inconsistent and some labels came from weak supervision. More annotated spans is the clearest next lever.
04Engineering
Engineering
With no local GPU, CPU work runs locally and fine-tuning runs on Google Colab. AWS is used only for S3 artifact storage, a deliberate choice to avoid a recurring hosting bill. Models live on the Hugging Face Hub and the app runs on Streamlit Community Cloud.