A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.
Advanced RAG pipelines for medical (HealthBench, MedCaseReasoning, MetaMedQA, PubMedQA) and financial (FinanceBench, Earnings Calls) QA. LangGraph orchestration + BAML structructed generation, Milvus Hybrid search (Dense + BM25 + RRF), three-layer Metadata Enrichment, Contextual AI instruction-following reranker, and DeepEval evaluation.
Single-file, zero-dependency local LLM benchmark suite + live dashboard. Deterministic scoring (Wilson CIs, paired McNemar A/B), per-question drill-down & live system metrics — for any OpenAI-compatible server (llama.cpp/vLLM/Ollama/LM Studio/ds4). Broad + medical benchmark panel.
This repository contains code and resources related to an in-depth analysis of OpenAI's HealthBench, a benchmark designed for evaluating Large Language Models in the healthcare sector.
Open-source medical LLM safety evaluation pipeline with reproducible benchmarks and high-risk clinical failure analysis.
Accuracy is not readiness: an open-source robustness stress-test of frontier LLMs (Opus 4.8, GPT-5.5, Grok 4.3, Gemini 3.5 Flash) on medical QA. Re-implements & extends Gu et al. 'The Illusion of Readiness in Health AI'.
[EE6406]ANALYTIC & ENSEMBLE LEARNING Project:End-to-end toolkit for medical SFT dataset preparation, Gemma 3 4B IT fine-tuning (LoRA/DoRA/QLoRA), vLLM inference, and HealthBench evaluation.
Auditing the clinical-evidence claims in OpenAI's HealthBench — its own gold answers & rubrics — for hallucinated, overgeneralized, overlooked & misweighted evidence. By NoBSmed.
Audit & Chinese-context reconstruction of OpenAI HealthBench: three linguistic tests, rubric augmentation (legal/social-psychological/capability), first-hand verifiable data. 临床医生兼小说作者对 HealthBench 的系统性审计与本土化评测标准重构。
Asha AI: medical answers with their sources for your product. OpenAI-compatible API, A2A v1.0, MCP server, CLI. Paired benchmarks: HealthBench, MedQA, psychosis-bench.
Asha, DNAi's medical AI agent: the same language model alone and inside Asha, same items, same grader. Per-item, re-gradable artifacts for HealthBench (5,000, Hard, Professional, Vishwanath 500), MedQA, psychosis-bench, enclomiphene-bench, a locked pediatric chart, plus model cards and A2A agent cards.
Safety before certainty: an open-source public-data stress-test showing a structured evidence-sufficiency prompt cuts unsafe overconfident clinical answers across four frontier LLMs (Opus 4.8, GPT-5.5, Gemini 3.5 Flash, Grok 4.3), with matched circularity controls and blinded clinician judge-validation.
Archived. Moved to github.com/EndlessRay/asha-benchmarks (healthbench-2026-06-26/).
To associate your repository with the healthbench topic, visit your repo's landing page and select "manage topics."