DuckDB's columnar tricks, tried on an eval log: query files in place, ASOF joins, sorted Parquet
DuckDB's team explains why storing data by column makes it fast. We tried the ideas on a small LLM eval log, with short queries you can run in a minute.
Topic · 20 stories
20 stories tagged evals on AI Tamer, newest first.
DuckDB's team explains why storing data by column makes it fast. We tried the ideas on a small LLM eval log, with short queries you can run in a minute.
Tan's open harness evals held Opus 5.5 and GPT-6.1 Sol fixed across seven setups. Pass rates could not be told apart; cost per solved task ran $0.17 to $0.44.
Stanford researchers retested 50 published LLM claims on 20 models: 44% expired, 34% weakened, median life 14.7 months. A project page, not a paper.
A preprint reports psychiatrists preferred the RL-trained agent in 76.7% of matched pairs and under-triage fell to 12.9%. No real patients were involved.
Security lab Irregular says Z.ai's GLM-5.3 refused none of its offensive tasks and finished 3 of 13 multi-stage attack scenarios in simulated networks.
Surge AI tested agents on 50 realistic tax assignments. The leader, Claude Sonnet 5.5, passed 10.8% at a 95% rubric bar. Missed deadlines were common.
Prompts in Markdown, tools in TypeScript, evals and replay built in, deploy to Cursor or self-host on Node. What the docs say, and the fine print.
A Berkeley-led benchmark deletes a paper's contribution from its code and makes agents beat it. Claude Opus 5.5 passes 58%; no agent passes 16 of 48 tasks.
Scope, an MIT-licensed research preview, gives Copilot and Claude Code one task, scores runs on a criteria graph with an LLM Judge, and keeps the evidence.
OpenAI's MultiLangBench has models build a SQLite-compatible engine in 20 languages, and a COBOL migration eval checks output byte for byte. Vendor-run.
A ByteDance Seed preprint says token offset can swing DeepSeek-V4 retrieval. The 40-point gap is the base checkpoint. Post-trained gaps are smaller.
Build a three-sample Inspect task with a solver chain and a built-in scorer, run it with inspect eval, and read the log to tell a wrong answer from a format mismatch.
A benchmark score depends on the prompt, the scoring method, the number of samples and the set of models compared. Here is what to check before you trust a row in a table.
NaiveAI’s Naive-N0.5-Flash (Sep 27, 2026): open-weight 309B MoE / 15.5B active, native 1M hybrid SWA–DSA, MIT on HF. API ($0.10/$0.40/$0.01) and NaiveRT remain future-tense—NaiveRT GitHub still 404 at the time of writing (promised by Oct 12). Benches/tok/s are NaiveAI harness claims only.
Holo4 (Sep 28, 2026): 27B dense (Qwen3.8) + 35B-A3B MoE (Qwen3.6) for GUI/code/MCP/API computer-use. License split: 27B weights CC BY-NC 4.0; 35B-A3B Apache-2.0. Both on H Models API + HF. OSWorld/cost figures are H Company harness claims only. Prefer BF16; context 256K (config max 262,144).
HF (2026-09-30) launched the Open TTS Leaderboard—objective WER/CER (Qwen3 ASR), RTFx, TTFA, and WavLM SIM for open-source and multilingual TTS/voice cloning. Eval infrastructure, not a model launch: ASR/WER is a proxy for intelligibility and does not measure listener preference or naturalness. Named ranks = blog snapshot; eval scripts still “soon.”
NVIDIA announced Kumo Tabular (HF blog 2026-09-29): open tabular foundation models for cls/reg via structured-data-models. Weights OpenMDW-1.1 on HF; library code Apache-2.0. No Inference Provider—local weights path. The TabArena and #1 claims are NVIDIA’s own.
Google DeepMind announced Gemini 4 Argon on 2026-09-30: phased to Fairwind cyber defenders first, 1M output tokens (not full context), intro $2/$10 then $4/$20 per 1M tokens on the blog—no public model ID or ai.google.dev pricing row yet.
SpaceXAI’s Grok 4.7 matches Grok 4.6 API token prices. Independent evals show Coding Agent Index gains with Grok Build at xhigh — and higher token use that can raise cost-per-task even when $/token is flat.
OpenAI cut GPT-6 Sol and Luna API prices roughly in half versus GPT-5.6 promo rates. Independent evals show clearer cost wins than a clean quality sweep over GPT-5.6 or Astra.