Notes from building.
Research, engineering, and the decisions around AI products. 48 articles on what makes an idea useful beyond the demo.
How to Actually Evaluate an LLM Application
Build evaluation around user tasks, separate development from holdout data, and make release decisions from evidence rather than a single score.
Read the essay3 min · AI / ML
A2A Interoperability Still Needs a Product Contract
Specify remote-agent capabilities, task lifecycle, permissions, and recovery before treating protocol compatibility as successful collaboration.
3 min · AI / ML
Agent Memory Needs an Expiry Date
Treat persistent memory as governed data: track its origin, scope, confidence, correction, and removal rather than retaining every conversation.
3 min · AI / ML
Agent Observability Without Logging Everything
Record the events needed to explain an outcome while controlling sensitive content, retention, and trace access.
4 min · AI / ML
Agent Reliability: One Success Is Not Enough
Separate best-of-many capability from repeated success, then test robustness, failure severity, and the cost of recovery.
3 min · Strategic foresight
AI Horizon Scanning Needs a Counterevidence Lane
Use AI to organize research signals while preserving independent sources, conflicting evidence, and the distinction between observation and scenario.
3 min · Product development
An AI Release Gate Should Be a Decision Record
Connect task evaluations, critical blockers, uncertainty, rollout conditions, and rollback ownership in one reviewable release decision.
3 min · Product development
Cost Per Useful Outcome Beats Cost Per Token
Account for unsuccessful attempts, review, tool usage, hosting, and completed-task quality when comparing AI workflows.
3 min · AI / ML
Document RAG Has to See the Page
Evaluate visual and text retrieval separately from answering, especially when tables, figures, and page layout carry the evidence.
3 min · AI / ML
Durable Agents Need More Than Checkpoints
Separate recoverable workflow state from external side effects, and design retries around idempotency and status reconciliation.
3 min · AI / ML
Your LLM Judge Needs an Evaluation Too
Test rubrics, position bias, source support, disagreement, and score uncertainty before using a model judge to approve releases.
3 min · AI / ML
Evidence Graphs Before GraphRAG
Use graphs when the question needs relationships or global synthesis, while preserving source passages and testing extraction errors.
3 min · Product development
Human–Agent Handoffs Need Evidence and Authority
Design human intervention around the decision, the evidence, and the consequences, then test appropriate reliance rather than perceived trust.
3 min · AI / ML
Hybrid Retrieval: Measure the Candidate Set First
Combine lexical and dense retrieval deliberately, then evaluate fusion, reranking, evidence coverage, and permission handling.
3 min · AI / ML
Long Context Is Not a Retrieval Strategy
Choose between direct context, retrieval, and compaction by testing evidence coverage and answer quality on the actual workload.
3 min · AI / ML
MCP Connects Tools. It Does Not Authorize Every Action
Understand MCP discovery and authorization scope, then enforce resource-specific permissions at the tool boundary.
4 min · AI / ML
Prompt Injection Is a Data-to-Action Problem
Treat retrieved content as evidence, constrain tool authority, and evaluate attacks against the actions an agent can actually perform.
3 min · AI / ML
Small Models Need a Fair Job Description
Evaluate specialized language models against a precise task, including escalation, maintenance, and deployment constraints.
3 min · AI / ML
Test-Time Compute Is a Budget, Not a Magic Switch
Compare extra reasoning, search, and verification by task difficulty, cost, latency, and independently checked outcomes.
4 min · AI / ML
When More Agents Make a System Worse
Use task structure, controlled comparisons, and handoff failures to decide whether a multi-agent architecture earns its complexity.
3 min · AI / ML
Zero-Shot Forecasting Is a Starting Point, Not a Waiver
Evaluate current time-series foundation models with honest information cutoffs, task baselines, interval checks, and exact deployment licences.
4 min · AI / ML
Before You Add an LLM, Build the Simplest Useful Baseline
Compare a proposed AI feature with search, rules, and a manual workflow before paying for complexity that users may not need.
4 min · AI / ML
Designing a Latency Budget for AI Features
Measure time to a useful result, trace the critical path, and reduce waiting without quietly removing the checks that make the answer dependable.
4 min · Product development
Build Versus Buy Is an Ownership Decision
Compare the full operating responsibility of a capability, including integration, support, change, and exit—not just development effort against a subscription.
4 min · AI / ML
RAG, Fine-Tuning, or Neither? Start with the Failure
Choose retrieval, prompt changes, fine-tuning, or a deterministic workflow by diagnosing the task failure—not by following an architecture trend.
4 min · AI / ML
Data Contracts: The Less Glamorous Side of Reliable ML
Define units, timestamps, ownership, and failure behavior at data boundaries so a valid-looking input cannot silently change the meaning of a prediction.
4 min · Product development
Product Discovery Before the Backlog Gets Expensive
Investigate the task, workaround, and consequence before turning a feature request into months of delivery work.
4 min · Product development
Product Experiments When You Don't Have Huge Traffic
Match the method to the uncertainty: use qualitative work for understanding, controlled experiments for causal questions, and honest limits when data is sparse.
4 min · AI / ML
Human Review Is a Workflow, Not a Checkbox
Make AI review actionable with evidence, realistic capacity, clear authority, and a measured path from correction to a completed task.
4 min · AI / ML
ML Monitoring After Launch: From Dashboards to Decisions
Separate service health, data quality, model outcomes, and user impact—and give each alert an owner and an action.
4 min · Product development
An MVP Should Complete One Workflow
Reduce the breadth of a first release without removing the beginning, ending, and recovery steps that make it useful.
4 min · Product development
Turning Product Feedback into Decisions, Not Just Tickets
Preserve the problem behind a request, compare evidence across users, and prioritize changes without treating request volume as a complete roadmap.
4 min · Product development
Product Metrics That Change a Decision
Choose a small set of outcome, diagnostic, and guardrail measures with explicit denominators and a clear reason to act.
4 min · AI / ML
Treat Prompt Changes Like Software Releases
Version prompts with their dependencies, evaluate changed behavior, and use controlled rollouts and rollback paths instead of editing production instructions in place.
4 min · Product development
From Prototype to Product: The Work Around the Demo
A useful prototype tests an assumption. A usable product also needs ownership, permissions, recovery, observability, and an honest delivery path.
4 min · AI / ML
When an AI Product Should Say “I Don't Know”
Design uncertainty as a useful next step: distinguish missing evidence, ambiguous requests, insufficient confidence, and unavailable capabilities.
4 min · AI / ML
From Black Box to Glass Box: Why Multi-Agent Systems Need to Show Their Work
Make AI systems inspectable with source evidence, action histories, and recovery controls—not a polished story about hidden reasoning.
4 min · AI / ML
What Air Cargo Forecasting Taught Me About Real-World ML
Lessons from the air-cargo forecasting problem: honest baselines, time-aware validation, decision-relevant errors, and an operational handoff.
4 min · Strategic foresight
Building a Trend Radar with LLMs (Without Hallucinating the Future)
A source-led design for trend discovery: collect evidence, test candidate signals, and use language models to explain rather than invent the story.
4 min · AI / ML
How to Actually Evaluate an LLM Application
Build evaluation around user tasks, separate development from holdout data, and make release decisions from evidence rather than a single score.
3 min · Strategic foresight
What a Trend Radar Actually Does in a Boardroom
Connect technology signals to decisions, accountable owners, and review dates instead of treating a radar chart as the strategy itself.
3 min · AI / ML
Why RAG Isn't a Silver Bullet
Retrieval helps an answer find evidence, but authority, permissions, coverage, and unsupported synthesis remain separate engineering problems.
4 min · Strategic foresight
Bibliometrics for Beginners: Reading Research Signals Carefully
Use publication and citation data to investigate research activity while accounting for coverage, age, field differences, and naming drift.
4 min · AI / ML
Notes from Building Multi-Agent Eval Harnesses
Evaluate agent workflows at the task, trace, and side-effect levels, with reproducible fixtures and explicit limits on retries and tool use.
3 min · Strategic foresight
Strategic Foresight vs Forecasting: They're Not the Same
Forecasting estimates defined outcomes; foresight explores plausible changes and strategic options. Useful decisions often need both.
4 min · AI / ML
The Quiet Cost of Average Accuracy
Use segment counts, error consequences, and calibrated uncertainty to understand what an attractive average can hide.
3 min · Strategic foresight
The Patent Signal: What Filings Quietly Reveal
Read patent activity as evidence of technical and organizational interest, with publication delays, family duplication, and commercialization limits in view.
4 min · AI / ML
Production Time Series in 2026: The Boring Tools That Win
A practical forecasting stack starts with baselines, honest backtests, input checks, and recovery—not a claim that one model family always wins.
4 min · Strategic foresight
Designing for the 10-Year Question
Use long-horizon questions to test assumptions, preserve options, and define signals that would change a decision—not to pretend the future is known.