Research · Featured projects · Libraries and tools · Interactive demos · Tech stack
AI engineer focused on LLM evaluation and agent tooling. I grade frontier-model output for AI labs' evaluation programs and build the tooling that makes those grades worth trusting.
Website: ianalloway.xyz · Writing: Alloway AI
- Contract @ AgentsOnly (since Oct 2026) — side-by-side evaluations of consumer app pages, scored against a written policy with structured rationales for QA review
- Contract @ Handshake AI (Jun–Oct 2026) — graded frontier-model output on software-engineering and agent tasks; side-by-side preference evals and reviews of agent-built apps
- Contract @ Mercor & Turing — building agent eval environments, golden answers, and grader rubrics (audit/accounting, spreadsheet automation)
- Founder, Alloway LLC — applied ML products, analytics tooling, and autonomous agent systems (including AI Advantage Sports)
- University of South Florida — M.S. Artificial Intelligence (expected Dec 2027) · B.S. Information Science, cum laude
My projects emphasize inspectable behavior, reproducible demos, and tests that check the decisions a system makes.
SpendAudit: can an LLM auditor match an executable payment policy?
Question: when an LLM reviews an agent's payments against a written policy, does its verdict match what the policy actually computes, and does an instruction planted in a payment memo change it?
Design: 160 synthetic payment histories, each labeled by SOLVENT's deterministic spend policy and paired with a copy whose memo argues for the wrong verdict. Three models were tested under a verdict-only protocol and under one that requires rule-by-rule evidence and arithmetic, and results are compared on the scenarios each model answered in every condition.
Findings: requiring evidence cut false approvals on clean traces from 34 to 6 and from 47 to 18 (exact McNemar p < 10⁻⁷) without a single false rejection, but injected memos still got violating payments approved for every model. The dataset, every model response, and a one-command recount of each table are public.
Working paper (PDF) · Benchmark and data · Reproduce the tables
SOLVENT — agent economics
Problem: an agent that can spend needs pricing, a margin gate, and an auditable ledger.
Design: a Python business loop with SQLite treasury, policy-screened spend, and optional Stripe test-mode payments and NVIDIA Nemotron inference.
What you can inspect: the offline demo runs earn → fulfil → spend → P&L with no API keys. Its dollar figures are simulated, not production revenue. Stripe test mode is the default earn rail. An optional Nano settle rail runs in sim or live verify-only mode: it checks public ledger blocks before fulfilling and never holds a hot wallet. Guardrails include per-job budget envelopes and a human-approval hook.
pip install solvent-agent && solvent · Try it in the browser · Run the demo · Dashboard preview · Security model
juryrig — audit the judge
Problem: LLM evaluation scores can hide position, verbosity, and prompt-injection bias.
Design: zero-dependency Python audits, judge panels, pairwise agreement, and calibration against human labels. Protocol-based adapters keep the judge provider replaceable.
What you can inspect: a network-free demo compares fair and deliberately rigged mock judges. Agreement matrices and the disagreement atlas ship in PyPI 0.3.1.
Demo source · Try it in the browser · Evaluation lab · PyPI · Tests
AI Advantage Sports — sports decision product
Problem: model output needs a usable workflow for odds, stake sizing, and portfolio risk.
Design: React / TypeScript UI with Kelly-based sizing, calibration views, and server-side Stripe entitlement handlers.
What you can inspect: a deployed product and open source implementation. The related training demo uses synthetic data; its metrics are not evidence of live betting returns.
Live product · Case study · Product screenshot
| Area | Projects |
|---|---|
| Sports libraries & experiments | nba-ratings / nba-edge — ratings and probability · kelly-js — odds and stake math · sports-betting-ml — synthetic training demo |
| Agent extensions | openclaw-skills — reusable skills · openclaw-theme-neon — theme with a preview |
| Web & experiments | ian-web-forge — portfolio · matrix-rain — React animation · evm-sentry — on-chain scanner |
| Upstream | LiteLLM #48895 (open) — reap the killed Prisma engine so it no longer lingers as a zombie process |
| Archive | oss-archive — completed project snapshots |
Writing: The browser is the real agent benchmark · Archived benchmark checklist.
| Project | What to try | Links |
|---|---|---|
| Ballpark | A multiplayer estimation game: closest guess wins. | Play · Source |
| Beat the Close | A shared market game with private scouting reads and closing-line value. | Play · Source |
| Neon theme | Switch palettes, tune CSS variables, and preview the rain effect. | Preview · Source |
| ASCII portfolio | An experimental monogram landing page built with Three.js and GSAP. | Source and local setup |
Evaluation: LLM-as-judge auditing · rubric & preference evals · agent evals · calibration
Languages: Python · TypeScript · SQL · JavaScript
AI / ML: OpenAI · NVIDIA Nemotron · LangChain · scikit-learn · XGBoost · Pandas
Product: FastAPI · Next.js / React · Stripe · PostgreSQL · Docker · GitHub Actions
GitHub activity
One more thing: the SOLVENT meme
other agents can spend money · SOLVENT gates spend on margin
