The engine turns supported local files into inputs and results the harness can record as evidence.
Definition
The analytics engine is Rust code with no LLM calls or conversation loop. It scans the workspace, loads supported tables, reads documents, executes analytics operations, and exposes the narrow interfaces used by built-in tools and deterministic verification.Why it exists
The model is allowed to propose a query or an analysis, but it is not allowed to be the computation layer. The engine makes the result:- deterministic enough to rerun and inspect;
- independent of the selected model, provider, UI, and conversation state;
- testable with ordinary Rust inputs without starting a model or Electron window.
What makes up the engine
engine/analytics/ contains the compute-and-check modules. The catalog and ingest modules are the input boundary immediately around it:
Inputs and backends
The shipped desktop build uses bundled SQLite as its data engine. Compiling with the non-defaultduckdb Cargo feature swaps SQLite for DuckDB; it does not run both analytics backends together.
Documents are not embedded or vector-indexed.
grep_files searches their text and read_file reads cataloged document content, independent of model-provider embedding support.
The default Cargo features are pdf and xlsx. duckdb is deliberately opt-in; the release data path is SQLite.
The analytics module depends on a deliberately small application capability:
DataEngine:
EngineState implements AnalyticsSource; open_engine selects SQLite by default or DuckDB only when the Cargo feature is compiled. Neither trait contains model or UI behavior.
Flow
- The scanner walks to a default depth of eight, does not follow symlinks, skips hidden entries, and applies a root
.fellaignore. - Recognized files become catalog entries; relevant files that cannot be loaded appear in
skippedwith a reason instead of failing silently. - Tabular sources receive sanitized, collision-safe view names and are loaded into the active backend. Root
fella.mdis user context, not a data source. The resulting catalog carries a deterministic revision marker for the loaded snapshot. - Built-in tools ask for schema statistics, sample rows, SQL, document matches, Python output, or chart data. A chart request runs its own read-only SQL query and derives the labels and values from that result.
- The engine returns bounded structured results. The tools layer turns them into
ToolOutput; the harness recordsToolOutputas evidence. - Verification receives only the
AnalyticsSourcecapability it needs, reruns eligible SQL after the answer, and emits one typed status alongside the individual checks.
Behavior
EngineState::run_sql is the narrow execution path used by both /sql and the run_sql tool:
data.query runs, ensure_read_only strips SQL comments, permits one statement, checks an allowlist of read-oriented starters, rejects mutation/attachment/extension operations, and blocks file-reading SQL functions such as read_csv_auto and read_parquet. SQLite then opens a fresh read-only connection as a second enforcement layer. The lexical guard is a guard rail, not a complete security boundary.
SQLite materializes at most 1,000 rows by default and interrupts queries after 15 seconds (FELLA_QUERY_TIMEOUT_SECS can override it). DuckDB is an opt-in build feature intended for Parquet and larger-file workloads; its backend uses an in-memory connection, four threads, and a 2 GB memory setting rather than SQLite’s interrupt watchdog. Current release builds remain SQLite-based unless explicitly built otherwise.
SQL and Python are peer analysis routes, not a primary path plus a fallback. SQL is suited to relational access and shaping—filters, joins, grouping, and simple aggregates. Python is often clearer for statistical or multi-stage numerical work such as medians, quantiles, dispersion, correlation, and regression. Fella’s model selects the route that fits the question, and can combine SQL-selected inputs with Python computation. For workspace calculations, Python retrieves data through its bounded sql(query) bridge rather than embedding copied rows in generated code. The embedded RustPython/WASM guest includes Python built-ins and Fella’s analytics helpers, but not pandas, NumPy, SciPy, or a package installer. Helper contracts are explicit: pearsonr(x, y) returns a scalar coefficient (not a p-value pair), and linregress(x, y) returns (slope, intercept, r). A coefficient alone does not establish statistical significance. Python code, inputs, and output are available in Analysis Details.
Charts are structured data in the supported families: bar, line, area, stacked
area, pie/donut, scatter, histogram, box plot, heatmap, and forecast. Inputs are
bounded analytical results validated in Rust and rendered by the client; the
model does not emit executable markup. See the chart examples for
the display and data-shape details.
Ingestion and normalization
The SQLite reader is intentionally opinionated about messy personal exports:- Delimited files use a quote-aware delimiter sniff over comma, semicolon, tab, and pipe. The reader examines the first 15 rows for a header, skips report preambles, synthesizes
colNnames when no header is found, and deduplicates repeated names with numeric suffixes. - A trailing
Total,Subtotal,Sum, orGrand totalrow is treated as a report summary rather than data, so a laterSUMdoes not double-count it. - CSV rows are read flexibly. Decode failures and the ingest cap are reported as notes; only the first
FELLA_INGEST_ROW_CAProws are loaded. - JSON accepts one object or an array of objects. NDJSON reads non-empty object records line by line. Columns are the first-seen union of object keys and absent values become
NULL. - Excel uses
calamine, creates one table per usable sheet, skips empty/unreadable sheets, applies the same header/summary-row handling, and loads typed rows throughDataEngine::add_rows.
Int, Float, Bool, Text, or Date. Blank-ish numeric values such as N/A, -, and null become SQL NULL. Written numbers such as $1,200, 1,200, 12%, and (45) can be coerced to Float with an ingest note; mixed columns remain Text with guidance to use the read-only parse_num() SQL helper. Named dates such as Aug 1, 2026 are normalized to ISO text for strftime() and date(); ambiguous numeric dates are intentionally left alone. Text label columns with case collisions such as Rent and rent receive a case-folding note.
SQL and schema
DataEngine gives the rest of the app one backend-neutral interface for adding sources, adding rows, dropping sources, describing columns, running queries, and building the Python bridge. open_engine() selects SqliteEngine by default or DuckEngine when compiled with --features duckdb.
SQLite stores the current workspace in a scratch analysis.db, rebuilt when the engine starts. It creates real tables with bulk inserts and registers a deterministic parse_num() scalar function on read-only query connections. describe() reports column type, null fraction, distinct count, minimum, and maximum. query() returns column names, a materialized row sample, the total row count, and whether the result was truncated.
DuckDB uses an in-memory connection, four threads, and a 2 GB memory setting. Path-backed CSV, TSV, JSON, NDJSON, and Parquet sources become DuckDB reader views; Excel still enters through the shared add_rows path. Its describe() uses SUMMARIZE. This backend is a feature-gated power build, not a runtime fallback in the shipped SQLite build.
Documents
Documents stay outside the SQL tables.ingest/docs.rs reads text files line by line and extracts PDFs when the pdf feature is present. PDF text is cached by path and modification time; a malformed PDF becomes a tool error, and a PDF with almost no text is reported as a likely scan. grep_files applies a case-insensitive regex without an index, streaming text files and scanning cached PDF text. read_file resolves a catalogued name rather than accepting an arbitrary path and returns bounded extracted text.
Python analysis
PythonBridge connects generated Python to the active backend without passing a path into Python. The host opens SQLite or DuckDB itself, checks the query, and returns a bounded JSON result that the guest turns into a list of dictionaries. median(), stdev(), pearsonr(), and linregress() are injected helpers; no third-party package is present.
pyexec::run() loads the checked-in wasm32-unknown-unknown RustPython guest into a fresh Wasmi store. The guest has no filesystem, network, environment, clock, or subprocess import. Wasmi caps fuel, linear memory, value-stack height, source, output, SQL rows, and SQL response size; the host data engine also keeps its query watchdog.
Charts
make_chart accepts a read-only SQL query or a complete prior result. Grouped-series, histogram, and box-plot queries have a 10,000-source-row materialization cap; other chart queries keep the 1,000-row cap on ordinary model-facing SQL results. This lets long-form observations be reshaped without relaxing the normal SQL boundary. Wide results use one label/date column and numeric series; tidy results use x_field, group_field, and value_field, with each group becoming a series. Duplicate x/group pairs are rejected rather than silently aggregated, and absent pairs remain gaps unless the model explicitly selects a missing-value treatment. analytics::chart::from_table() then creates typed ChartData, with chart-family limits for category counts, time points, groups, finite values, and missingness. The backend emits no SVG or HTML; Chart.svelte renders the validated data, thins labels and point markers on dense line charts while retaining every point, and offers a collapsed exact-values table as an accessible alternative.
Provenance
An answer captures the workspace path and catalog revision present when its run started. SQL-backed evidence maps referenced views back to catalogued files or workbook sheets and carries any source-level ingest note. The revision is a freshness marker, not a cryptographic integrity proof; it changes when loaded source metadata, schema, ingest notes, skipped files, or root context changes.Verification boundary
verify.rs belongs in the analytics module because its checks are deterministic operations over the catalog, SQL, answers, contracts, and recorded evidence. It receives only AnalyticsSource, never the full EngineState, and never calls a model. The agent owns bounded, model-driven repair; each verifier finding specifies its effect and target so the harness can preserve unrelated work. The verification page documents that authority boundary and its known false-negative/false-positive limits.
JSON-render presentations
The answer presentation layer consumes accepted results from the analysis record. The model composes a JSON-render tree using a registered component catalog; chart and table components bind to saved result IDs and fields. This is separate from the make_chart analysis tool: the presentation layer decides how to arrange and explain available results, while the analytics engine remains responsible for producing them. The renderer cannot execute arbitrary code or query the workspace. Invalid presentation output does not remove the saved answer or evidence.Limits
- SQLite is the default and shipped analytics backend. Parquet is unavailable in that build; DuckDB support is feature-gated, not an automatically selected runtime mode.
- Scans, ingestion, queries, document reads, and displayed rows are bounded. Large inputs may be truncated with a note.
- Type inference and normalization are best effort. Ingest notes and verifier warnings should be inspected for messy data.
run_pythonruns in the embedded WASM guest. It has no filesystem, network, environment, clock, or subprocess capability; Wasmi and the host data engine apply resource and query limits.- The app also uses a separate SQLite database for settings, cached source metadata, and conversation metadata. Provider credentials are stored in
auth.json, not that database.
Execution limits
The history behind this module matters: commit
08d0b6e made the analytics boundary explicit, while PR #102 added structured chart data and commit 9072c8d moved chart rendering to the client so the backend emits data rather than SVG markup.
Next steps
See the tools
Review the interfaces exposed over engine operations.
See the checks
Learn which engine results are rerun and compared.