Vals AI

MedQA

Evaluating language model bias in medical questions.

Not in index
Healthcare
Source ↗

Models94

Top result96.5%o1

Top-3 spread0.1 pts

Updated16 Apr 2026

Result and cost

95 of 95 results

Results

95 results

#ModelResultCostTime

01o196.5% ±0.3$0.05311 s

02GPT-5.196.4% ±0.2$0.009812 s

02Gemini 3.1 Pro Preview96.4% ±0.2$0.01839 s

04GPT-596.3% ±0.2$0.02135 s

05GPT-5.496.1% ±0.2$0.04423 s

05o396.1% ±0.2$0.00548.5 s

05GPT-5 mini96.1% ±0.2$0.00316 s

08Gemini 3 Pro96.0% ±0.2$0.02922 s

08o4-mini96.0% ±0.2$0.00286.6 s

10Claude Opus 4.5 Thinking95.9% ±0.2$0.14832 s

11Gemini 3 Flash Preview95.8% ±0.2$0.01326 s

12Claude Opus 4.6 (Adaptive)95.4% ±0.2$0.02730 s

13Qwen3.5 Plus Thinking95.2% ±0.2$0.008352 s

14o3-mini94.8% ±0.4$0.00538.4 s

15Claude Sonnet 4.5 Thinking94.7% ±0.2$0.03136 s

16Grok 4.20 (Reasoning)94.5% ±0.2$0.009112 s

17Kimi K2.5 Thinking94.4% ±0.2$0.00742 s

18GLM-5 Thinking94.3% ±0.2$0.00881 s

19GPT-5.294.1% ±0.2$0.02219 s

20DeepSeek V3.2 (Thinking)93.9% ±0.2$0.0007754 s

21GLM-4.793.7% ±0.2$0.004966 s

22Claude Opus 4.1 Thinking93.6% ±0.2$0.08824 s

23GPT-5 nano93.3% ±0.2$0.001320 s

24Claude Opus 4.593.2% ±0.2$0.04310 s

25Gemini 2.5 Pro Exp 03.2593.1% ±0.2$0.002911 s

26o1-preview93.0% ±0.5$0.10916 s

27Claude Opus 492.9% ±0.2$0.03812 s

28Claude Sonnet 4 Thinking92.7% ±0.2$0.02527 s

29Kimi K2 Thinking92.6% ±0.2$0.00624 min

30Claude Opus 4.192.5% ±0.2$0.04214 s

30MiniMax M2.592.5% ±0.2$0.002934 s

30Grok 492.5% ±0.2$0.02664 s

33Grok 292.3% ±0.5$0.00334.1 s

34GLM-4.692.2% ±0.2$0.004329 s

35Grok 4.1 Fast (Reasoning)92.1% ±0.2$0.0005718 s

35Grok 4 Fast (Reasoning)92.1% ±0.2$0.000745.3 s

35Claude Sonnet 4.692.1% ±0.2$0.03535 s

38Gemini 2.5 Flash (Preview)91.4% ±0.3$0.00138.5 s

38GPT-OSS 120B91.4% ±0.2$0.0006112 s

40GPT-4.191.2% ±0.3$0.0033.1 s

40Gemini 2.5 Flash Preview Thinking91.2% ±0.3$0.001318 s

40MiniMax M2.191.2% ±0.3$0.002436 s

43Gemini 2.5 Flash Preview 04.17 Thinking91.0% ±0.3$0.00388.9 s

44DeepSeek-R190.8% ±0.5$0.005942 s

45Qwen3 235B-A22B90.6% ±0.3$0.001226 s

46Claude Sonnet 490.3% ±0.3$0.00718.7 s

47Claude 3.7 Sonnet Thinking90.2% ±0.5$0.01716 s

47o1-mini90.2% ±0.5$0.00435.9 s

49Grok 3 Mini Fast High Reasoning90.1% ±0.3$0.000777.1 s

50GLM-4.590.0% ±0.3$0.003460 s

51Magistral Medium89.5% ±0.3$0.006917 s

51DeepSeek V3.289.5% ±0.3$0.0002313 s

53Gemini 2.5 Flash Lite Preview Thinking88.9% ±0.3$0.0001715 s

54Grok 3 Mini Fast Low Reasoning88.7% ±0.3$0.000564.9 s

55Meta Llama 3.1 405B Instruct Turbo88.2% ±0.6$0.00268.7 s

55GPT-4o88.2% ±0.6$0.00283.4 s

57Qwen3 Maxqwen3-max-preview87.4% ±0.3$0.003220 s

57Qwen3 Maxqwen3-max87.4% ±0.3$0.00320.0 s

59Gemini 2.5 Flash Preview 04.1786.7% ±0.3$0.00112.3 s

60Meta Llama 3.1 70B Instruct Turbo84.8% ±0.6$0.000624.8 s

61GPT-4.1 mini84.6% ±0.3$0.000521.9 s

62Kimi K2 Instruct84.0% ±0.3$0.00111 s

63Grok 3 [Beta]83.8% ±0.3$0.00627.5 s

64Claude 3.5 Sonnet83.2% ±0.7$0.0065.9 s

65GPT-OSS 20B82.9% ±0.3$0.0003425 s

66Magistral Small82.4% ±0.3$0.00178.0 s

67Mistral Large 382.2% ±0.3$0.0007717 s

68DeepSeek V3 032482.0% ±0.4$0.0006113 s

68GPT-4 Turbo82.0% ±0.7$0.0128.4 s

70Gemini 2.0 Flash 00181.5% ±0.7$0.000111.9 s

71DeepSeek V380.9% ±0.4$0.000586.8 s

72Command A80.5% ±0.4$0.00352.9 s

73Gemini 2.5 Flash Lite Preview80.3% ±0.4$0.000285.1 s

74Claude Haiku 4.5 Thinking79.6% ±0.4$0.01424 s

75Mistral Medium78.2% ±0.4$0.000647.3 s

76Qwen2.5 72B Instruct Turbo77.4% ±0.7$0.000845.6 s

77Gemini 1.5 Pro 00276.5% ±0.8$0.0035.9 s

78Mistral Large 241176.2% ±0.8$0.00244.9 s

79Grok 4.1 Fast Non Reasoning76.0% ±0.4$0.000176.6 s

80Grok 4 Fast Non Reasoning75.4% ±0.4$0.000482.4 s

81GPT-4o mini72.4% ±0.8$0.000162.3 s

82Mistral Small 3.169.1% ±0.4$0.00013.6 s

83GPT-4.1 nano68.2% ±0.4$0.000151.4 s

84Jamba 1.5 Large68.1% ±0.8$0.00226.0 s

85Meta Llama 3.1 8B Instruct Turbo62.6% ±0.9$0.000122.4 s

86Mixtral 8x22B Instruct v0.162.1% ±0.9$0.000928.9 s

87GPT-3.5 Turbo58.5% ±0.9$0.000421.6 s

88Mistral Small 240257.0% ±0.9$0.000212.8 s

89Jamba 1.5 Mini55.2% ±0.9$0.000151.1 s

90Mixtral 8x7B V0.153.2% ±0.9$0.000453.5 s

91Jamba Mini 1.652.5% ±0.5$0.00022.2 s

92Llama 4 Scout 17B 16E Instruct50.9% ±0.5$0.000334.8 s

93Jamba Large 1.650.7% ±0.5$0.00317.3 s

94Llama4 Maverick Instruct Basic43.3% ±0.5$0.000697.8 s

95Command R Plus2.7% ±0.3$0.00356.1 s