Choose a benchmark for the question being evaluated: operator quality, end-to-end execution, query generation, or reproducibility.
Workloads for evaluating semantic operators, document analytics, and query execution. Their schemas, labels, and operator coverage differ.
Evaluates semantic engines across text, image and audio scenarios using filters, joins, mappings, ranking and classification.
Results are a snapshot of evolving systems, models and configurations; they do not establish a permanent leaderboard.
Source and context ↗Organizes Select, Match, Impute, Cluster and Order operators into single-operator and multi-operator workloads over relational databases.
Operator-level findings depend on model, implementation and workload; the benchmark complements rather than replaces end-to-end evaluation.
Source and context ↗Evaluates unstructured-document analysis against manually annotated relational views. The arXiv v1 workload contains five datasets and 240 queries covering extraction, filtering, aggregation, joins, and combinations of operators.
Ground truth is defined over selected schema attributes, so coverage does not extend to arbitrary open-ended document questions. Dataset counts differ between the cited arXiv version and repository descriptions; specify the artifact version when reporting results.
Source and context ↗Extends TPC-DS-derived relations with return reports, issue bulletins, and release notes to study semantic-join scaling. The JEVDB paper evaluates five queries at XS, S, and M tiers.
Introduced and evaluated by the JEVDB authors. The full SF1 workload is larger than the measured tiers; its candidate count must not be presented as an executed comparison. Shelob is not an official TPC benchmark or an independent validation of JEVDB.
Source and context ↗Evaluate the translation of a request into SQL with AI functions or vector search. These measure a different stage from engine execution.
Tests whether models generate Snowflake SQL that composes relational operations with AI functions for classification, filtering, sentiment analysis, extraction, similarity, and aggregation.
Evaluates SQL generation, not engine execution performance. The paper reports 465 instances; the public task release contains 393. Gold SQL and database resources are not bundled, and official scoring requires additional access.
Source and context ↗Benchmarks generation of queries combining relational operations and vector search, spanning three database backends and four source collections. Examines result omissions when SQL filters interact with vector retrieval.
An adjacent query-generation benchmark. Vector similarity is a retrieval operation and does not establish the quality or efficiency of arbitrary model-backed filters, joins, or aggregations.
Source and context ↗Broader tasks that combine database access with external evidence and agent coordination.
Evaluates agents on 380 tasks requiring both database queries and web evidence. Tasks exercise database-to-search, search-to-database, and parallel reasoning, with an emphasis on preserving constraints across tools.
Measures agent coordination and final-answer correctness across tools. It does not isolate semantic operator execution or query optimizer performance, and web evidence can change over time.
Source and context ↗Studies of scoring and reproducibility. These inform experimental design; they are not interchangeable with benchmark workloads.
Studies how repeated model decisions propagate into changing query answers and proposes reporting practices for reproducible evaluation.
October 2026 preprint. This is an evaluation study rather than a new engine; findings concern the tested models and workloads.
Source and context ↗Studies failures of exact-result scoring for SQL with stochastic AI operators. Proposes evaluating relational logic and semantic operations separately, with experiments on BigQuery and ThalamusDB.
An evaluation framework rather than a general-purpose workload benchmark. Reported validation accuracy applies to the tested query sets and judging protocol, not all SQL translations or semantic predicates.
Source and context ↗Task-specific metrics, label provenance, held-out data, and confidence intervals where justified.
Model and token usage, pricing date, infrastructure, and whether setup or training costs are included.
End-to-end runtime, concurrency, caching, retries, rate limits, and failures.
Dataset and code versions, model identifiers, prompts, repeated runs, and answer-level variability.
A benchmark result applies to a particular workload, artifact version, model, and execution configuration. Dataset counts and public releases can differ from a paper's experimental workload. Record these distinctions before comparing results.
We have not rerun the systems under a common protocol. The catalog therefore describes benchmark scope and evidence rather than ranking reported speedups. See Same Query, Different Answer for repeated-run evaluation and The Stochastic Shift for scoring SQL with AI operators.
Propose a reproduction report ↗
sqpe