This area covers everything that goes beyond deterministic extraction and rule-based analysis:
NER during ingestion, optional heavy NLP libraries, ML Classification, and the supervised triage experiments.
Specifically in development are semantic similarity, named entity recognition,
and content sanitization to complement existing pattern-based extraction, clustering, and visualizations.
Additionally, in the works: ML models run on cases within each extraction feature to surface
similarities, differences, and distinguishing patterns across stat-style cohorts — for example FBI cases vs local
law enforcement investigations, stranger cases vs family or teacher cases, and possible offender patterns across ages.
The main workflow for triage is the Triage tab (/triage): rule-based
tiers and evaluation, Random Forest model over the live database, and paste-in
“live triage” that never writes to storage. This page explains scope, status, and how those pieces fit together.
Pattern extraction and explicit tags remain the source of truth for case schema and filtering. ML here either augments extraction (NER) or learns from the existing rule-based priority scores (supervised triage tiers) so we can compare model behavior to transparent rules—not replace them silently. Semantic search, richer clustering, and comparative group studies are being explored where data supports it.
A random forest (and optionally a simpler decision tree) is trained on the structured features the system already derives from each case. Training targets are priority tiers from an explicit, rule-based rubric—so model output stays anchored to policy investigators can review, not an opaque score. The Triage workspace presents evaluation metrics and accuracy, a clear view of where the model disagrees with the rubric, and case lists by tier against the current case database; the same model can be run across the full corpus on demand and narrowed with standard filters, always reflecting live data rather than a frozen export. Live triage lets analysts paste narrative text and see suggested tiers in the session only—nothing is stored. The model file can be retrained and checked against benchmarks as the dataset grows. Overall this is an interpretable assist with measurable performance alongside transparent rules: built for scale, accountability, and analyst support—not black-box automation.
requirements-ml.txt. Not required
for the triage bundle. Intended for future similarity search and embedding-backed clustering without
disrupting the deterministic baseline.
scikit-learn / joblib — Triage training, metrics, saved bundle (core requirements.txt)stanza — Primary NER path in ingestion when installed and models are presentsentence-transformers — Optional embeddings (requirements-ml.txt)spaCy / transformers — Optional NER or summarization paths where enabledML components are optional enhancements, not replacements. Pattern-based extraction remains primary; ML provides additional capabilities. System reverts gracefully if ML models are unavailable or undesired.
Ingestion flow: Case text → pattern extraction → NER merge → storage. Triage inference does not affect stored cases.
What runs where: Cluster summaries for the Clusters view are precomputed and cached in the database for large corpora. Triage model classifies cases on demand from running the saved model over the current database (no static snapshot).
Deployment: Default requirements.txt includes sklearn/joblib for triage.
Install requirements-ml.txt when you need embeddings, spaCy, or full transformers stacks locally.
/triage: rule tiers, train/test-style metrics via API, demo Random Forest model over live data, paste-in live triagepython3 scripts/train_triage_model.py --out models/triage_bundle.joblibMergeProcessing)