The analytics engine

The engine turns supported local files into inputs and results the harness can record as evidence.

​

Definition

​

Why it exists

• deterministic enough to rerun and inspect;
• independent of the selected model, provider, UI, and conversation state;
• testable with ordinary Rust inputs without starting a model or a Tauri window.

​

What makes up the engine

​

Inputs and backends

pub trait AnalyticsSource {
    fn catalog(&self) -> Catalog;
    fn run_sql(&self, sql: &str) -> EngineResult<QueryResult>;
}

pub trait DataEngine: Send {
    fn add_source(&mut self, name: &str, kind: SourceKind, path: &str)
        -> EngineResult<SourceLoad>;
    fn add_rows(
        &mut self,
        name: &str,
        columns: &[(String, ColType)],
        rows: &[Vec<Cell>],
    ) -> EngineResult<i64>;
    fn drop_source(&mut self, name: &str);
    fn describe(&self, name: &str) -> EngineResult<Vec<ColumnInfo>>;
    fn query(&self, sql: &str, max_rows: usize) -> EngineResult<QueryOutcome>;
    fn python_bridge(&self) -> PythonBridge;
}

​

Flow

1. The scanner walks to a default depth of eight, does not follow symlinks, skips hidden entries, and applies a root .fellaignore.
2. Recognized files become catalog entries; relevant files that cannot be loaded appear in skipped with a reason instead of failing silently.
3. Tabular sources receive sanitized, collision-safe view names and are loaded into the active backend. Root fella.md is user context, not a data source. The resulting catalog carries a deterministic revision marker for the loaded snapshot.
4. Built-in tools ask for schema statistics, sample rows, SQL, document matches, Python output, or chart data. A chart request runs its own read-only SQL query and derives the labels and values from that result.
5. The engine returns bounded structured results. The tools layer turns them into ToolOutput; the harness records ToolOutput as evidence.
6. Verification receives only the AnalyticsSource capability it needs, reruns eligible SQL after the answer, and emits one typed status alongside the individual checks.

​

Behavior

pub fn run_sql(&self, sql: &str) -> EngineResult<QueryResult> {
    data::ensure_read_only(sql)?;
    let data = self.data.lock().unwrap_or_else(|e| e.into_inner());
    let t = Instant::now();
    let out = data.query(sql, DEFAULT_ROW_CAP)?;
    Ok(QueryResult {
        columns: out.columns,
        rows: out.rows,
        row_count: out.row_count,
        ms: t.elapsed().as_millis() as u64,
        truncated: out.truncated,
    })
}

​

Ingestion and normalization

• Delimited files use a quote-aware delimiter sniff over comma, semicolon, tab, and pipe. The reader examines the first 15 rows for a header, skips report preambles, synthesizes colN names when no header is found, and deduplicates repeated names with numeric suffixes.
• A trailing Total, Subtotal, Sum, or Grand total row is treated as a report summary rather than data, so a later SUM does not double-count it.
• CSV rows are read flexibly. Decode failures and the ingest cap are reported as notes; only the first FELLA_INGEST_ROW_CAP rows are loaded.
• JSON accepts one object or an array of objects. NDJSON reads non-empty object records line by line. Columns are the first-seen union of object keys and absent values become NULL.
• Excel uses calamine, creates one table per usable sheet, skips empty/unreadable sheets, applies the same header/summary-row handling, and loads typed rows through DataEngine::add_rows.

​

SQL and schema

​

Documents

​

Python analysis

​

Charts

​

Provenance

​

Verification boundary

​

Limits

• SQLite is the default and shipped analytics backend. Parquet is unavailable in that build; DuckDB support is feature-gated, not an automatically selected runtime mode.
• Scans, ingestion, queries, document reads, and displayed rows are bounded. Large inputs may be truncated with a note.
• Type inference and normalization are best effort. Ingest notes and verifier warnings should be inspected for messy data.
• run_python runs in the embedded WASM guest. It has no filesystem, network, environment, clock, or subprocess capability; Wasmi and the host data engine apply resource and query limits.
• The app also uses a separate SQLite database for settings, cached source metadata, and conversation metadata. Provider credentials are stored in auth.json, not that database.

​

Execution limits

​

Next steps