Fella FolderQA Benchmark
FQA-Bench is Fella’s benchmark for model-driven analytics over mounted folders. It evaluates the complete product loop—inspection, interpretation, analysis, clarification, verification, and communication—not just whether a model can produce a plausible number in one prompt. FQA-Bench v0.1 is the first unreleased target-state specification. Its tasks assume the full product roadmap is implemented. They can measure roadmap capabilities even while today’s app is still building them; a current-build run is a diagnostic, not proof that the benchmark is wrong or that the target capability is already shipped.What it measures
The benchmark follows an analyst’s work: inspect the available data, interpret and decompose the request, analyze and verify the result, then explain it or ask a focused clarification if an important detail is missing. Tasks are organized by capability. Topics, formats, and data conditions are cross-cutting tags.
Analysis-family tags separately cover workspace discovery, semantic interpretation, normalization, descriptive, relational, statistical, temporal/predictive, data-quality, visualization, evidence, clarification/continuity, and proportionality/boundary tasks. Topics, formats, data conditions, and interaction shape are cross-cutting labels. Forecasts and scenarios are valid analyses, not an automatic refusal category. A missing source should be explained specifically; a material ambiguity should be clarified and then resolved in the same scored episode.
One episode can test several capabilities, but gets one primary capability for diagnosis. FQA-Bench reports results by capability, analytical family, and topic with task counts. A single overall percentage can hide important weaknesses.
For existing
agent_eval --json results, a small report utility joins case IDs to the canonical tags and prints separate slices: