Verification and evidence

Verification ties an answer to recorded work; it does not make the model a trusted grader.

​

Definition

​

Architecture

1. Tables referenced by cited SQL exist.
2. Distinct inexpensive SQL queries rerun to the same rows.
3. Numbers in the answer appear in summaries, outputs, or result cells.
4. SUM, AVG, or TOTAL is not applied suspiciously to text.
5. Exact-case filters do not overlook known mixed-case labels.
6. The SQL aggregate matches aggregate wording in the question.
7. A meaningful column named in the question was not dropped from the SQL.
8. A likely shared-column join was not missed.
9. Date grouping did not produce a NULL bucket.
10. Values from a multi-aggregate row are labelled with the correct result column.

pub fn run(
    engine: &dyn AnalyticsSource,
    question: &str,
    answer: &str,
    evidence: &[EvidenceItem],
) -> Vec<VerificationCheck> {
    let mut checks = Vec::new();
    check_tables(engine, evidence, &mut checks);
    rerun_queries(engine, evidence, &mut checks);
    check_numbers(answer, evidence, &mut checks);
    check_text_agg(engine, evidence, &mut checks);
    check_case_filter(engine, evidence, &mut checks);
    check_aggregate_verb(question, evidence, &mut checks);
    check_dropped_column(engine, question, evidence, &mut checks);
    check_multi_table_join(engine, question, evidence, &mut checks);
    check_null_group_key(evidence, &mut checks);
    check_row_value_labels(answer, evidence, &mut checks);
    checks
}

​

Flow

1. The model returns an answer with no more tool calls.
2. The harness runs the deterministic verifier.
3. Queries that originally took more than 500 ms or returned truncated rows are not rerun; the first result is retained with a check explaining the skip.
4. If a rerun now fails or changes, the default-on corrective path may make one tool-free model call to reconcile the answer (FELLA_VERIFY_REASK=0 disables it).
5. If any deterministic warning remains, the default-on self-check may make one stricter, tool-free model call from existing results (FELLA_SELF_CHECK=0 disables it).
6. The second answer is compared numerically with the first. A disagreement becomes another visible check; it does not replace deterministic verification.

​

Behavior

​

Limits

• Numeric grounding uses tolerant number matching. It is useful but cannot prove a derived percentage or narrative interpretation is correct in every form.
• A successful rerun proves repeatability at that moment, not source completeness or truth.
• Heuristic SQL checks can produce warnings on valid queries and miss semantic mistakes.
• Python output can support number grounding, but the verifier does not independently rerun arbitrary Python programs.
• A model second opinion costs another provider request and remains model output, not deterministic validation.

​

Test the failure shape, not just the happy path

let evidence = vec![run_sql_ev(
    "SELECT strftime('%Y-%m', Date) AS month, SUM(amount) AS total FROM ledger GROUP BY strftime('%Y-%m', Date)",
    &["month", "total"],
    vec![vec![Json::Null, Json::from(15_797)]],
)];

let mut checks = Vec::new();
check_null_group_key(&evidence, &mut checks);

assert!(!checks[0].ok);
assert!(hard_fail(&checks).is_some());

​

Next steps