Benchmark - All Results
BioCypher query generation
In this set of tasks, we test LLM abilities to generate queries for a BioCypher Knowledge Graph using BioChatter.
The schema_config.yaml of the BioCypher Knowledge Graph and a natural language query are passed to BioChatter.
Individual steps of the query generation process are tested separately, as well as the end-to-end performance of the process.
| Full model name | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|
| gpt-3.5-turbo-0125 | 8 | 8 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | 8 | 8 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | 9 | 9 | 1 | 5 |
| gpt-4o-2024-05-13 | 8 | 8 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | 8 | 9 | 0.888889 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | 8 | 9 | 0.888889 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | 8 | 9 | 0.888889 | 5 |
| gpt-4-0613 | 8 | 9 | 0.888889 | 5 |
| gpt-3.5-turbo-0613 | 8 | 9 | 0.888889 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | 7 | 8 | 0.875 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | 7 | 8 | 0.875 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | 7 | 8 | 0.875 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | 7 | 8 | 0.875 | 5 |
| gpt-4-0125-preview | 7 | 9 | 0.777778 | 5 |
| chatglm3:6:ggmlv3:q4_0 | 6 | 8 | 0.75 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | 5 | 9 | 0.555556 | 5 |
| code-llama-instruct:7:ggufv2:Q3_K_M | 4 | 8 | 0.5 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | 4 | 8 | 0.5 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | 3.8 | 8 | 0.475 | 5 |
| code-llama-instruct:13:ggufv2:Q3_K_M | 3.6 | 8 | 0.45 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | 4 | 9 | 0.444444 | 5 |
| llama-2-chat:7:ggufv2:Q8_0 | 4 | 9 | 0.444444 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | 4 | 9 | 0.444444 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | 4 | 9 | 0.444444 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | 4 | 9 | 0.444444 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | 4 | 9 | 0.444444 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | 3.8 | 9 | 0.422222 | 5 |
| llama-2-chat:7:ggufv2:Q6_K | 3 | 8 | 0.375 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | 3 | 9 | 0.333333 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | 3 | 9 | 0.333333 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | 3 | 9 | 0.333333 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | 3 | 9 | 0.333333 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | 3 | 9 | 0.333333 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | 3 | 9 | 0.333333 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | 3 | 9 | 0.333333 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | 3 | 9 | 0.333333 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | 2.8 | 9 | 0.311111 | 5 |
| code-llama-instruct:7:ggufv2:Q2_K | 2 | 8 | 0.25 | 5 |
| code-llama-instruct:34:ggufv2:Q8_0 | 2 | 8 | 0.25 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | 2 | 9 | 0.222222 | 5 |
| code-llama-instruct:34:ggufv2:Q6_K | 1 | 8 | 0.125 | 5 |
| code-llama-instruct:34:ggufv2:Q5_K_M | 1 | 8 | 0.125 | 5 |
| code-llama-instruct:7:ggufv2:Q5_K_M | 1 | 9 | 0.111111 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | 0 | 9 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q6_K | 0 | 8 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q4_K_M | 0 | 8 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q5_K_M | 0 | 8 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q6_K | 0 | 8 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q8_0 | 0 | 9 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | 0 | 9 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | 0 | 9 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q2_K | 0 | 8 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | 0 | 9 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | 0 | 9 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q6_K | 0 | 8 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q8_0 | 0 | 8 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q2_K | 0 | 8 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q3_K_M | 0 | 8 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q4_K_M | 0 | 8 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | 0 | 9 | 0 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | 0 | 9 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | 0 | 9 | 0 | 5 |
| Full model name | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|
| openhermes-2.5:7:ggufv2:Q8_0 | 12 | 12 | 1 | 5 |
| gpt-3.5-turbo-0125 | 12 | 12 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | 12 | 12 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | 12 | 12 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | 12 | 12 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | 12 | 12 | 1 | 5 |
| gpt-4-0125-preview | 9 | 12 | 0.75 | 5 |
| gpt-4-0613 | 7.8 | 12 | 0.65 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | 6 | 12 | 0.5 | 5 |
| code-llama-instruct:34:ggufv2:Q2_K | 6 | 12 | 0.5 | 5 |
| gpt-3.5-turbo-0613 | 6 | 12 | 0.5 | 5 |
| chatglm3:6:ggmlv3:q4_0 | 4.8 | 12 | 0.4 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | 3 | 12 | 0.25 | 5 |
| code-llama-instruct:7:ggufv2:Q3_K_M | 3 | 12 | 0.25 | 5 |
| code-llama-instruct:7:ggufv2:Q2_K | 3 | 12 | 0.25 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | 3 | 12 | 0.25 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | 3 | 12 | 0.25 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | 3 | 12 | 0.25 | 5 |
| code-llama-instruct:34:ggufv2:Q3_K_M | 3 | 12 | 0.25 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | 3 | 12 | 0.25 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | 3 | 12 | 0.25 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | 0 | 12 | 0 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | 0 | 12 | 0 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | 0 | 12 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | 0 | 12 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | 0 | 12 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | 0 | 12 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | 0 | 12 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q8_0 | 0 | 12 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | 0 | 12 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | 0 | 12 | 0 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | 0 | 12 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | 0 | 12 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q8_0 | 0 | 12 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q6_K | 0 | 12 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q5_K_M | 0 | 12 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q4_K_M | 0 | 12 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q3_K_M | 0 | 12 | 0 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | 0 | 12 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | 0 | 12 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q8_0 | 0 | 12 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q6_K | 0 | 12 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | 0 | 12 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q5_K_M | 0 | 12 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q6_K | 0 | 12 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q8_0 | 0 | 12 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q6_K | 0 | 12 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q5_K_M | 0 | 12 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q4_K_M | 0 | 12 | 0 | 5 |
| gpt-4o-2024-05-13 | 0 | 12 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | 0 | 12 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | 0 | 12 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | 0 | 12 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | 0 | 12 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q6_K | 0 | 12 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | 0 | 12 | 0 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | 0 | 12 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q2_K | 0 | 12 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | 0 | 12 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | 0 | 12 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | 0 | 12 | 0 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | 0 | 12 | 0 | 5 |
| Full model name | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|
| gpt-3.5-turbo-0613 | 23.2 | 64 | 0.3625 | 5 |
| gpt-4-0613 | 23 | 64 | 0.359375 | 5 |
| gpt-3.5-turbo-0125 | 22.8 | 64 | 0.35625 | 5 |
| chatglm3:6:ggmlv3:q4_0 | 18.4 | 64 | 0.2875 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | 18 | 64 | 0.28125 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | 18 | 64 | 0.28125 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | 12 | 64 | 0.1875 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | 11 | 64 | 0.171875 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | 10.4 | 64 | 0.1625 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | 8 | 64 | 0.125 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | 8 | 64 | 0.125 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | 8 | 64 | 0.125 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | 7 | 64 | 0.109375 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | 6.4 | 64 | 0.1 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | 4.2 | 64 | 0.065625 | 5 |
| code-llama-instruct:7:ggufv2:Q2_K | 4 | 64 | 0.0625 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | 3 | 64 | 0.046875 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | 3 | 64 | 0.046875 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | 3 | 64 | 0.046875 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | 3 | 64 | 0.046875 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | 2.4 | 64 | 0.0375 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | 2.4 | 64 | 0.0375 | 5 |
| code-llama-instruct:13:ggufv2:Q5_K_M | 0 | 64 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q4_K_M | 0 | 64 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q3_K_M | 0 | 64 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | 0 | 64 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | 0 | 64 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | 0 | 64 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q3_K_M | 0 | 64 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q8_0 | 0 | 64 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q8_0 | 0 | 64 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | 0 | 64 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | 0 | 64 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | 0 | 64 | 0 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | 0 | 64 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q6_K | 0 | 64 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q5_K_M | 0 | 64 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | 0 | 64 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q6_K | 0 | 64 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | 0 | 64 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | 0 | 64 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q5_K_M | 0 | 64 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q6_K | 0 | 64 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q8_0 | 0 | 64 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q2_K | 0 | 64 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q8_0 | 0 | 64 | 0 | 5 |
| gpt-4-0125-preview | 0 | 64 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q6_K | 0 | 64 | 0 | 5 |
| gpt-4o-2024-05-13 | 0 | 64 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | 0 | 64 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | 0 | 64 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | 0 | 64 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | 0 | 64 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q6_K | 0 | 64 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | 0 | 64 | 0 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | 0 | 64 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q2_K | 0 | 64 | 0 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | 0 | 64 | 0 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | 0 | 64 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | 0 | 64 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q4_K_M | 0 | 64 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q3_K_M | 0 | 64 | 0 | 5 |
| Full model name | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|
| code-llama-instruct:34:ggufv2:Q4_K_M | 7.8 | 8 | 0.975 | 5 |
| code-llama-instruct:34:ggufv2:Q5_K_M | 7.6 | 8 | 0.95 | 5 |
| code-llama-instruct:34:ggufv2:Q8_0 | 7.4 | 8 | 0.925 | 5 |
| code-llama-instruct:34:ggufv2:Q6_K | 7.2 | 8 | 0.9 | 5 |
| gpt-4-0613 | 8 | 9 | 0.888889 | 5 |
| code-llama-instruct:13:ggufv2:Q2_K | 7 | 8 | 0.875 | 5 |
| code-llama-instruct:34:ggufv2:Q3_K_M | 7 | 8 | 0.875 | 5 |
| gpt-3.5-turbo-0125 | 7.8 | 9 | 0.866667 | 5 |
| code-llama-instruct:13:ggufv2:Q3_K_M | 6.8 | 8 | 0.85 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | 6.8 | 8 | 0.85 | 5 |
| gpt-4o-2024-05-13 | 6.8 | 8 | 0.85 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | 7.6 | 9 | 0.844444 | 5 |
| code-llama-instruct:13:ggufv2:Q6_K | 6.6 | 8 | 0.825 | 5 |
| code-llama-instruct:7:ggufv2:Q3_K_M | 7.2 | 9 | 0.8 | 5 |
| code-llama-instruct:7:ggufv2:Q2_K | 6.4 | 8 | 0.8 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | 7 | 9 | 0.777778 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | 7 | 9 | 0.777778 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | 7 | 9 | 0.777778 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | 7 | 9 | 0.777778 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | 7 | 9 | 0.777778 | 5 |
| code-llama-instruct:7:ggufv2:Q6_K | 6.2 | 8 | 0.775 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | 6.2 | 8 | 0.775 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | 6.2 | 8 | 0.775 | 5 |
| code-llama-instruct:13:ggufv2:Q5_K_M | 6.2 | 8 | 0.775 | 5 |
| code-llama-instruct:13:ggufv2:Q4_K_M | 6.2 | 8 | 0.775 | 5 |
| llama-2-chat:13:ggufv2:Q6_K | 6.2 | 8 | 0.775 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | 6.8 | 9 | 0.755556 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | 6.8 | 9 | 0.755556 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | 6.8 | 9 | 0.755556 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | 6.8 | 9 | 0.755556 | 5 |
| gpt-3.5-turbo-0613 | 6.8 | 9 | 0.755556 | 5 |
| code-llama-instruct:13:ggufv2:Q8_0 | 6 | 8 | 0.75 | 5 |
| code-llama-instruct:34:ggufv2:Q2_K | 6 | 8 | 0.75 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | 6.6 | 9 | 0.733333 | 5 |
| gpt-4-0125-preview | 6.6 | 9 | 0.733333 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | 6.6 | 9 | 0.733333 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | 6.6 | 9 | 0.733333 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | 5.8 | 8 | 0.725 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | 7.2 | 10 | 0.72 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | 6.4 | 9 | 0.711111 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | 6.4 | 9 | 0.711111 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | 6.2 | 9 | 0.688889 | 5 |
| code-llama-instruct:7:ggufv2:Q5_K_M | 6.2 | 9 | 0.688889 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | 6.2 | 9 | 0.688889 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | 6.2 | 9 | 0.688889 | 5 |
| code-llama-instruct:7:ggufv2:Q8_0 | 6 | 9 | 0.666667 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | 6 | 9 | 0.666667 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | 6 | 9 | 0.666667 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | 6 | 9 | 0.666667 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | 5.2 | 8 | 0.65 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | 5.2 | 8 | 0.65 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | 5.8 | 9 | 0.644444 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | 5.8 | 9 | 0.644444 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | 5.4 | 9 | 0.6 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | 5.4 | 9 | 0.6 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | 4.4 | 9 | 0.488889 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | 4.2 | 9 | 0.466667 | 5 |
| llama-2-chat:7:ggufv2:Q8_0 | 3.2 | 9 | 0.355556 | 5 |
| llama-2-chat:7:ggufv2:Q6_K | 3 | 9 | 0.333333 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | 2.6 | 9 | 0.288889 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | 2.6 | 9 | 0.288889 | 5 |
| chatglm3:6:ggmlv3:q4_0 | 2.2 | 8 | 0.275 | 5 |
| Full model name | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|
| gpt-4-0613 | 20.4 | 30 | 0.68 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | 20 | 30 | 0.666667 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | 20 | 30 | 0.666667 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | 20 | 30 | 0.666667 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | 19.6 | 30 | 0.653333 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | 18 | 30 | 0.6 | 5 |
| code-llama-instruct:34:ggufv2:Q3_K_M | 18 | 30 | 0.6 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | 17.6 | 30 | 0.586667 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | 17.2 | 30 | 0.573333 | 5 |
| code-llama-instruct:13:ggufv2:Q8_0 | 17 | 30 | 0.566667 | 5 |
| code-llama-instruct:34:ggufv2:Q2_K | 17 | 30 | 0.566667 | 5 |
| code-llama-instruct:13:ggufv2:Q2_K | 17 | 30 | 0.566667 | 5 |
| code-llama-instruct:13:ggufv2:Q5_K_M | 17 | 30 | 0.566667 | 5 |
| code-llama-instruct:13:ggufv2:Q6_K | 16.2 | 30 | 0.54 | 5 |
| code-llama-instruct:7:ggufv2:Q2_K | 16 | 30 | 0.533333 | 5 |
| gpt-4o-2024-05-13 | 16 | 30 | 0.533333 | 5 |
| code-llama-instruct:13:ggufv2:Q4_K_M | 16 | 30 | 0.533333 | 5 |
| code-llama-instruct:13:ggufv2:Q3_K_M | 16 | 30 | 0.533333 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | 16 | 30 | 0.533333 | 5 |
| gpt-3.5-turbo-0613 | 15 | 30 | 0.5 | 5 |
| gpt-3.5-turbo-0125 | 14.6 | 30 | 0.486667 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | 14.4 | 30 | 0.48 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | 14.4 | 30 | 0.48 | 5 |
| chatglm3:6:ggmlv3:q4_0 | 14.4 | 30 | 0.48 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | 14.4 | 30 | 0.48 | 5 |
| code-llama-instruct:34:ggufv2:Q6_K | 14.2 | 30 | 0.473333 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | 14.2 | 30 | 0.473333 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | 14 | 30 | 0.466667 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | 14 | 30 | 0.466667 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | 14 | 30 | 0.466667 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | 14 | 30 | 0.466667 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | 14 | 30 | 0.466667 | 5 |
| code-llama-instruct:34:ggufv2:Q4_K_M | 14 | 30 | 0.466667 | 5 |
| code-llama-instruct:34:ggufv2:Q5_K_M | 14 | 30 | 0.466667 | 5 |
| code-llama-instruct:34:ggufv2:Q8_0 | 14 | 30 | 0.466667 | 5 |
| gpt-4-0125-preview | 13.2 | 30 | 0.44 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | 13 | 30 | 0.433333 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | 13 | 30 | 0.433333 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | 13 | 30 | 0.433333 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | 13 | 30 | 0.433333 | 5 |
| code-llama-instruct:7:ggufv2:Q3_K_M | 12.8 | 30 | 0.426667 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | 12.8 | 30 | 0.426667 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | 12.6 | 30 | 0.42 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | 12.4 | 30 | 0.413333 | 5 |
| code-llama-instruct:7:ggufv2:Q5_K_M | 12 | 30 | 0.4 | 5 |
| code-llama-instruct:7:ggufv2:Q8_0 | 12 | 30 | 0.4 | 5 |
| llama-2-chat:13:ggufv2:Q6_K | 11.6 | 30 | 0.386667 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | 11.6 | 30 | 0.386667 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | 11.4 | 30 | 0.38 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | 11 | 30 | 0.366667 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | 11 | 30 | 0.366667 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | 11 | 30 | 0.366667 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | 10.8 | 30 | 0.36 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | 10 | 30 | 0.333333 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | 10 | 30 | 0.333333 | 5 |
| code-llama-instruct:7:ggufv2:Q6_K | 10 | 30 | 0.333333 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | 8.8 | 30 | 0.293333 | 5 |
| llama-2-chat:7:ggufv2:Q8_0 | 8 | 30 | 0.266667 | 5 |
| llama-2-chat:7:ggufv2:Q6_K | 8 | 30 | 0.266667 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | 7.2 | 30 | 0.24 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | 7 | 30 | 0.233333 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | 3 | 30 | 0.1 | 5 |
| Full model name | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|
| gpt-3.5-turbo-0125 | 29 | 30 | 0.966667 | 5 |
| gpt-4-0613 | 29 | 30 | 0.966667 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | 29 | 30 | 0.966667 | 5 |
| code-llama-instruct:7:ggufv2:Q5_K_M | 28.8 | 30 | 0.96 | 5 |
| code-llama-instruct:7:ggufv2:Q8_0 | 28.8 | 30 | 0.96 | 5 |
| code-llama-instruct:7:ggufv2:Q6_K | 28.8 | 30 | 0.96 | 5 |
| gpt-3.5-turbo-0613 | 28.4 | 30 | 0.946667 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | 28.2 | 30 | 0.94 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | 28.2 | 30 | 0.94 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | 27.8 | 30 | 0.926667 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | 27.8 | 30 | 0.926667 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | 27.6 | 30 | 0.92 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | 27.6 | 30 | 0.92 | 5 |
| code-llama-instruct:7:ggufv2:Q2_K | 27.6 | 30 | 0.92 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | 27.6 | 30 | 0.92 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | 27.4 | 30 | 0.913333 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | 27.2 | 30 | 0.906667 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | 27.2 | 30 | 0.906667 | 5 |
| code-llama-instruct:34:ggufv2:Q4_K_M | 27.2 | 30 | 0.906667 | 5 |
| code-llama-instruct:34:ggufv2:Q5_K_M | 27 | 30 | 0.9 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | 27 | 30 | 0.9 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | 26.8 | 30 | 0.893333 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | 26.4 | 30 | 0.88 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | 26.2 | 30 | 0.873333 | 5 |
| code-llama-instruct:7:ggufv2:Q3_K_M | 26.2 | 30 | 0.873333 | 5 |
| code-llama-instruct:34:ggufv2:Q8_0 | 25.8 | 30 | 0.86 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | 25.8 | 30 | 0.86 | 5 |
| code-llama-instruct:34:ggufv2:Q6_K | 25.6 | 30 | 0.853333 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | 25.4 | 30 | 0.846667 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | 25.4 | 30 | 0.846667 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | 25.2 | 30 | 0.84 | 5 |
| gpt-4-0125-preview | 25 | 30 | 0.833333 | 5 |
| code-llama-instruct:13:ggufv2:Q4_K_M | 25 | 30 | 0.833333 | 5 |
| code-llama-instruct:13:ggufv2:Q3_K_M | 25 | 30 | 0.833333 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | 25 | 30 | 0.833333 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | 24.8 | 30 | 0.826667 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | 24.8 | 30 | 0.826667 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | 24.8 | 30 | 0.826667 | 5 |
| code-llama-instruct:13:ggufv2:Q2_K | 24.6 | 30 | 0.82 | 5 |
| llama-2-chat:13:ggufv2:Q6_K | 24.4 | 30 | 0.813333 | 5 |
| gpt-4o-2024-05-13 | 24 | 30 | 0.8 | 5 |
| code-llama-instruct:13:ggufv2:Q6_K | 23.8 | 30 | 0.793333 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | 23.6 | 30 | 0.786667 | 5 |
| code-llama-instruct:34:ggufv2:Q3_K_M | 23.6 | 30 | 0.786667 | 5 |
| code-llama-instruct:13:ggufv2:Q5_K_M | 23.4 | 30 | 0.78 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | 23.2 | 30 | 0.773333 | 5 |
| code-llama-instruct:13:ggufv2:Q8_0 | 23 | 30 | 0.766667 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | 22.8 | 30 | 0.76 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | 22.8 | 30 | 0.76 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | 22.4 | 30 | 0.746667 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | 21.8 | 30 | 0.726667 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | 20.8 | 30 | 0.693333 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | 20.8 | 30 | 0.693333 | 5 |
| code-llama-instruct:34:ggufv2:Q2_K | 20.6 | 30 | 0.686667 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | 20.6 | 30 | 0.686667 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | 20.4 | 30 | 0.68 | 5 |
| llama-2-chat:7:ggufv2:Q6_K | 19.8 | 30 | 0.66 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | 19.4 | 30 | 0.646667 | 5 |
| llama-2-chat:7:ggufv2:Q8_0 | 19.2 | 30 | 0.64 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | 19 | 30 | 0.633333 | 5 |
| chatglm3:6:ggmlv3:q4_0 | 16.6 | 30 | 0.553333 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | 13 | 30 | 0.433333 | 5 |
| Full model name | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|
| gpt-3.5-turbo-0125 | 27.8 | 30 | 0.926667 | 5 |
| gpt-4-0613 | 26.4 | 30 | 0.88 | 5 |
| gpt-3.5-turbo-0613 | 25 | 30 | 0.833333 | 5 |
| chatglm3:6:ggmlv3:q4_0 | 0 | 30 | 0 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | 0 | 30 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | 0 | 30 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | 0 | 30 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | 0 | 30 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q6_K | 0 | 30 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q8_0 | 0 | 30 | 0 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | 0 | 30 | 0 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | 0 | 30 | 0 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | 0 | 30 | 0 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | 0 | 30 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | 0 | 30 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | 0 | 30 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | 0 | 30 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | 0 | 30 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | 0 | 30 | 0 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | 0 | 30 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | 0 | 30 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | 0 | 30 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | 0 | 30 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | 0 | 30 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | 0 | 30 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | 0 | 30 | 0 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | 0 | 30 | 0 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | 0 | 30 | 0 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | 0 | 30 | 0 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | 0 | 30 | 0 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | 0 | 30 | 0 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | 0 | 30 | 0 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | 0 | 30 | 0 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | 0 | 30 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q3_K_M | 0 | 30 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q3_K_M | 0 | 30 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q4_K_M | 0 | 30 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q5_K_M | 0 | 30 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q6_K | 0 | 30 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q8_0 | 0 | 30 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q2_K | 0 | 30 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q3_K_M | 0 | 30 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q4_K_M | 0 | 30 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q5_K_M | 0 | 30 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q6_K | 0 | 30 | 0 | 5 |
| code-llama-instruct:34:ggufv2:Q8_0 | 0 | 30 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q2_K | 0 | 30 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | 0 | 30 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q2_K | 0 | 30 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q5_K_M | 0 | 30 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q6_K | 0 | 30 | 0 | 5 |
| code-llama-instruct:7:ggufv2:Q8_0 | 0 | 30 | 0 | 5 |
| gpt-4-0125-preview | 0 | 30 | 0 | 5 |
| gpt-4o-2024-05-13 | 0 | 30 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | 0 | 30 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | 0 | 30 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | 0 | 30 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | 0 | 30 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q6_K | 0 | 30 | 0 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | 0 | 30 | 0 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | 0 | 30 | 0 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | 0 | 30 | 0 | 5 |
Retrieval-Augmented Generation (RAG)
In this set of tasks, we test LLM abilities to generate answers to a given question using a RAG agent, or to judge the relevance of a RAG fragment to a given question. Instructions can be explicit ("is this fragment relevant to the question?") or implicit (just asking the question without instructions and evaluating whether the model responds with 'not enough information given').
| Full model name | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|
| llama-2-chat:70:ggufv2:Q3_K_M | 6 | 6 | 1 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | 6 | 6 | 1 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | 6 | 6 | 1 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | 6 | 6 | 1 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | 6 | 6 | 1 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | 6 | 6 | 1 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | 6 | 6 | 1 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | 6 | 6 | 1 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | 6 | 6 | 1 | 5 |
| llama-2-chat:7:ggufv2:Q6_K | 6 | 6 | 1 | 5 |
| llama-2-chat:7:ggufv2:Q8_0 | 6 | 6 | 1 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | 6 | 6 | 1 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | 6 | 6 | 1 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | 6 | 6 | 1 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | 6 | 6 | 1 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | 6 | 6 | 1 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | 6 | 6 | 1 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | 6 | 6 | 1 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | 6 | 6 | 1 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | 6 | 6 | 1 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | 6 | 6 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | 6 | 6 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | 6 | 6 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | 6 | 6 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | 6 | 6 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | 6 | 6 | 1 | 5 |
| llama-2-chat:13:ggufv2:Q6_K | 6 | 6 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | 6 | 6 | 1 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | 6 | 6 | 1 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | 6 | 6 | 1 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | 6 | 6 | 1 | 5 |
| gpt-4o-2024-05-13 | 6 | 6 | 1 | 5 |
| gpt-4-0613 | 6 | 6 | 1 | 5 |
| gpt-4-0125-preview | 6 | 6 | 1 | 5 |
| gpt-3.5-turbo-0613 | 6 | 6 | 1 | 5 |
| gpt-3.5-turbo-0125 | 6 | 6 | 1 | 5 |
| code-llama-instruct:7:ggufv2:Q8_0 | 6 | 6 | 1 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | 6 | 6 | 1 | 5 |
| code-llama-instruct:13:ggufv2:Q6_K | 5 | 6 | 0.833333 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | 5 | 6 | 0.833333 | 5 |
| code-llama-instruct:7:ggufv2:Q6_K | 5 | 6 | 0.833333 | 5 |
| code-llama-instruct:7:ggufv2:Q5_K_M | 5 | 6 | 0.833333 | 5 |
| code-llama-instruct:7:ggufv2:Q3_K_M | 5 | 6 | 0.833333 | 5 |
| code-llama-instruct:13:ggufv2:Q8_0 | 5 | 6 | 0.833333 | 5 |
| chatglm3:6:ggmlv3:q4_0 | 4.4 | 6 | 0.733333 | 5 |
| code-llama-instruct:13:ggufv2:Q5_K_M | 4 | 6 | 0.666667 | 5 |
| code-llama-instruct:34:ggufv2:Q4_K_M | 3 | 6 | 0.5 | 5 |
| code-llama-instruct:34:ggufv2:Q3_K_M | 3 | 6 | 0.5 | 5 |
| code-llama-instruct:34:ggufv2:Q2_K | 3 | 6 | 0.5 | 5 |
| code-llama-instruct:34:ggufv2:Q8_0 | 2 | 6 | 0.333333 | 5 |
| code-llama-instruct:34:ggufv2:Q5_K_M | 2 | 6 | 0.333333 | 5 |
| code-llama-instruct:13:ggufv2:Q4_K_M | 2 | 6 | 0.333333 | 5 |
| code-llama-instruct:7:ggufv2:Q2_K | 2 | 6 | 0.333333 | 5 |
| code-llama-instruct:34:ggufv2:Q6_K | 2 | 6 | 0.333333 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | 2 | 6 | 0.333333 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | 1 | 6 | 0.166667 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | 0.8 | 6 | 0.133333 | 5 |
| code-llama-instruct:13:ggufv2:Q2_K | 0.2 | 6 | 0.0333333 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | 0 | 6 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | 0 | 6 | 0 | 5 |
| code-llama-instruct:13:ggufv2:Q3_K_M | 0 | 6 | 0 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | 0 | 6 | 0 | 5 |
| Full model name | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|
| chatglm3:6:ggmlv3:q4_0 | 2 | 2 | 1 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | 2 | 2 | 1 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | 2 | 2 | 1 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | 2 | 2 | 1 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | 2 | 2 | 1 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | 2 | 2 | 1 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | 2 | 2 | 1 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | 2 | 2 | 1 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | 2 | 2 | 1 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | 2 | 2 | 1 | 5 |
| gpt-3.5-turbo-0613 | 2 | 2 | 1 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | 2 | 2 | 1 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | 2 | 2 | 1 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | 2 | 2 | 1 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | 2 | 2 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | 2 | 2 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | 2 | 2 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | 2 | 2 | 1 | 5 |
| gpt-4-0613 | 2 | 2 | 1 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | 2 | 2 | 1 | 5 |
| code-llama-instruct:34:ggufv2:Q2_K | 2 | 2 | 1 | 5 |
| code-llama-instruct:34:ggufv2:Q5_K_M | 2 | 2 | 1 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | 2 | 2 | 1 | 5 |
| gpt-3.5-turbo-0125 | 1.8 | 2 | 0.9 | 5 |
| code-llama-instruct:7:ggufv2:Q6_K | 1.8 | 2 | 0.9 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | 1.8 | 2 | 0.9 | 5 |
| code-llama-instruct:34:ggufv2:Q6_K | 1.8 | 2 | 0.9 | 5 |
| code-llama-instruct:34:ggufv2:Q8_0 | 1.8 | 2 | 0.9 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | 1.8 | 2 | 0.9 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | 1.4 | 2 | 0.7 | 5 |
| code-llama-instruct:7:ggufv2:Q3_K_M | 1.4 | 2 | 0.7 | 5 |
| code-llama-instruct:7:ggufv2:Q2_K | 1.4 | 2 | 0.7 | 5 |
| gpt-4o-2024-05-13 | 1.4 | 2 | 0.7 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | 1.2 | 2 | 0.6 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | 1.2 | 2 | 0.6 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | 1.2 | 2 | 0.6 | 5 |
| code-llama-instruct:34:ggufv2:Q3_K_M | 1 | 2 | 0.5 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | 1 | 2 | 0.5 | 5 |
| code-llama-instruct:13:ggufv2:Q4_K_M | 1 | 2 | 0.5 | 5 |
| code-llama-instruct:13:ggufv2:Q5_K_M | 1 | 2 | 0.5 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | 1 | 2 | 0.5 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | 1 | 2 | 0.5 | 5 |
| code-llama-instruct:7:ggufv2:Q8_0 | 1 | 2 | 0.5 | 5 |
| code-llama-instruct:13:ggufv2:Q6_K | 1 | 2 | 0.5 | 5 |
| code-llama-instruct:13:ggufv2:Q8_0 | 1 | 2 | 0.5 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | 1 | 2 | 0.5 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | 1 | 2 | 0.5 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | 1 | 2 | 0.5 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | 1 | 2 | 0.5 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | 1 | 2 | 0.5 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | 1 | 2 | 0.5 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | 1 | 2 | 0.5 | 5 |
| gpt-4-0125-preview | 1 | 2 | 0.5 | 5 |
| llama-2-chat:13:ggufv2:Q6_K | 1 | 2 | 0.5 | 5 |
| llama-2-chat:7:ggufv2:Q8_0 | 1 | 2 | 0.5 | 5 |
| llama-2-chat:7:ggufv2:Q6_K | 1 | 2 | 0.5 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | 1 | 2 | 0.5 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | 1 | 2 | 0.5 | 5 |
| code-llama-instruct:7:ggufv2:Q5_K_M | 1 | 2 | 0.5 | 5 |
| code-llama-instruct:34:ggufv2:Q4_K_M | 0.8 | 2 | 0.4 | 5 |
| code-llama-instruct:13:ggufv2:Q2_K | 0.8 | 2 | 0.4 | 5 |
| code-llama-instruct:13:ggufv2:Q3_K_M | 0 | 2 | 0 | 5 |
Text Extraction
In this set of tasks, we test LLM abilities to extract text from a given document.
| Full model name | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|
| gpt-4-0125-preview | 68.2808 | 99 | 0.689705 | 5 |
| gpt-4-0613 | 66.2214 | 99 | 0.668903 | 5 |
| gpt-4o-2024-05-13 | 64.7407 | 99 | 0.653946 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | 61.2976 | 99 | 0.619167 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | 59.482 | 99 | 0.600829 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | 59.1309 | 99 | 0.597281 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | 57.4117 | 99 | 0.579916 | 5 |
| gpt-3.5-turbo-0613 | 56.9628 | 99 | 0.575381 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | 54.8943 | 99 | 0.554488 | 5 |
| gpt-3.5-turbo-0125 | 50.4932 | 99 | 0.510032 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | 43.9613 | 99 | 0.444054 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | 38.1897 | 99 | 0.385754 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | 36.5284 | 99 | 0.368974 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | 36.3738 | 99 | 0.367412 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | 34.8167 | 99 | 0.351684 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | 34.3555 | 99 | 0.347025 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | 32.7949 | 99 | 0.331261 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | 23.8526 | 99 | 0.240936 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | 23.3302 | 99 | 0.235659 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | 22.7326 | 99 | 0.229622 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | 22.3269 | 99 | 0.225524 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | 21.2897 | 99 | 0.215047 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | 20.8064 | 99 | 0.210166 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | 19.5919 | 99 | 0.197898 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | 19.1849 | 99 | 0.193786 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | 18.7286 | 99 | 0.189177 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | 18.6669 | 99 | 0.188555 | 5 |
| chatglm3:6:ggmlv3:q4_0 | 18.6402 | 99 | 0.188284 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | 16.4769 | 99 | 0.166434 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | 16.103 | 99 | 0.162657 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | 15.5939 | 99 | 0.157514 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | 13.7345 | 99 | 0.138732 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | 11.5703 | 99 | 0.116871 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | 11.1504 | 99 | 0.112631 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | 8.79788 | 99 | 0.0888675 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | 8.43969 | 99 | 0.0852494 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | 7.58505 | 99 | 0.0766167 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | 7.54833 | 99 | 0.0762457 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | 6.90615 | 99 | 0.0697591 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | 6.4421 | 99 | 0.0650717 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | 6.42895 | 99 | 0.0649389 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | 3.58247 | 99 | 0.0361865 | 5 |
| Full model name | Subtask | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|---|
| gpt-4o-2024-05-13 | assay | 6.67307 | 9 | 0.741453 | 5 |
| gpt-4-0125-preview | assay | 6.60264 | 9 | 0.733627 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | assay | 6.45354 | 9 | 0.717059 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | assay | 6.42156 | 9 | 0.713507 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | assay | 6.24141 | 9 | 0.69349 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | assay | 5.8662 | 9 | 0.6518 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | assay | 5.84165 | 9 | 0.649072 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | assay | 5.83272 | 9 | 0.64808 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | assay | 5.77475 | 9 | 0.641639 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | assay | 5.72421 | 9 | 0.636024 | 5 |
| gpt-3.5-turbo-0613 | assay | 5.71717 | 9 | 0.635241 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | assay | 5.66084 | 9 | 0.628983 | 5 |
| gpt-3.5-turbo-0125 | assay | 5.48324 | 9 | 0.609249 | 5 |
| gpt-4-0613 | assay | 5.47238 | 9 | 0.608042 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | assay | 5.40473 | 9 | 0.600526 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | assay | 4.99329 | 9 | 0.55481 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | assay | 4.35689 | 9 | 0.484099 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | assay | 3.17543 | 9 | 0.352825 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | assay | 1.8509 | 9 | 0.205655 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | assay | 1.81844 | 9 | 0.202049 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | assay | 1.68419 | 9 | 0.187133 | 5 |
| chatglm3:6:ggmlv3:q4_0 | assay | 1.61672 | 9 | 0.179636 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | assay | 1.53778 | 9 | 0.170864 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | assay | 1.48103 | 9 | 0.164559 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | assay | 1.37088 | 9 | 0.15232 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | assay | 1.16327 | 9 | 0.129252 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | assay | 1.15926 | 9 | 0.128806 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | assay | 1.15095 | 9 | 0.127884 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | assay | 1.07788 | 9 | 0.119765 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | assay | 1.05347 | 9 | 0.117052 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | assay | 1.02909 | 9 | 0.114343 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | assay | 0.974441 | 9 | 0.108271 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | assay | 0.922706 | 9 | 0.102523 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | assay | 0.919259 | 9 | 0.10214 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | assay | 0.836349 | 9 | 0.0929276 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | assay | 0.756302 | 9 | 0.0840336 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | assay | 0.750557 | 9 | 0.0833952 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | assay | 0.647223 | 9 | 0.0719137 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | assay | 0.604799 | 9 | 0.0671999 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | assay | 0.522273 | 9 | 0.0580303 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | assay | 0.455699 | 9 | 0.0506332 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | assay | 0.233824 | 9 | 0.0259804 | 5 |
| Full model name | Subtask | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|---|
| gpt-4-0613 | chemical | 6.38889 | 9 | 0.709877 | 5 |
| gpt-4-0125-preview | chemical | 6.22222 | 9 | 0.691358 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | chemical | 6.16667 | 9 | 0.685185 | 5 |
| gpt-4o-2024-05-13 | chemical | 5.55556 | 9 | 0.617284 | 5 |
| gpt-3.5-turbo-0613 | chemical | 5.44444 | 9 | 0.604938 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | chemical | 5.23309 | 9 | 0.581455 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | chemical | 5.16667 | 9 | 0.574074 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | chemical | 5.06667 | 9 | 0.562963 | 5 |
| gpt-3.5-turbo-0125 | chemical | 5.06444 | 9 | 0.562716 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | chemical | 4.95556 | 9 | 0.550617 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | chemical | 4.66667 | 9 | 0.518519 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | chemical | 4.02332 | 9 | 0.447036 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | chemical | 3.69824 | 9 | 0.410916 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | chemical | 3.5588 | 9 | 0.395422 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | chemical | 3.23175 | 9 | 0.359083 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | chemical | 2.9648 | 9 | 0.329422 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | chemical | 2.85926 | 9 | 0.317696 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | chemical | 2.80214 | 9 | 0.311349 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | chemical | 2.28839 | 9 | 0.254265 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | chemical | 1.99259 | 9 | 0.221399 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | chemical | 1.98451 | 9 | 0.220501 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | chemical | 1.98451 | 9 | 0.220501 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | chemical | 1.92687 | 9 | 0.214097 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | chemical | 1.92403 | 9 | 0.213781 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | chemical | 1.86594 | 9 | 0.207326 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | chemical | 1.7972 | 9 | 0.199689 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | chemical | 1.65417 | 9 | 0.183796 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | chemical | 1.60885 | 9 | 0.178761 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | chemical | 1.37178 | 9 | 0.15242 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | chemical | 1.02473 | 9 | 0.113859 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | chemical | 0.993896 | 9 | 0.110433 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | chemical | 0.920791 | 9 | 0.10231 | 5 |
| chatglm3:6:ggmlv3:q4_0 | chemical | 0.839293 | 9 | 0.0932548 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | chemical | 0.580952 | 9 | 0.0645503 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | chemical | 0.473978 | 9 | 0.0526642 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | chemical | 0.473978 | 9 | 0.0526642 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | chemical | 0.447004 | 9 | 0.0496671 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | chemical | 0.44189 | 9 | 0.0490989 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | chemical | 0.429118 | 9 | 0.0476798 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | chemical | 0.416702 | 9 | 0.0463002 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | chemical | 0.270151 | 9 | 0.0300168 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | chemical | 0.264943 | 9 | 0.0294381 | 5 |
| Full model name | Subtask | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|---|
| gpt-4-0613 | context | 7.90663 | 9 | 0.878514 | 5 |
| gpt-4-0125-preview | context | 7.85253 | 9 | 0.872503 | 5 |
| gpt-4o-2024-05-13 | context | 7.82965 | 9 | 0.869961 | 5 |
| gpt-3.5-turbo-0125 | context | 6.89247 | 9 | 0.76583 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | context | 6.89055 | 9 | 0.765616 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | context | 6.79989 | 9 | 0.755543 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | context | 6.77271 | 9 | 0.752524 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | context | 6.67749 | 9 | 0.741944 | 5 |
| gpt-3.5-turbo-0613 | context | 6.50472 | 9 | 0.722747 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | context | 6.44769 | 9 | 0.71641 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | context | 5.16754 | 9 | 0.574171 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | context | 5.12599 | 9 | 0.569555 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | context | 5.02844 | 9 | 0.558716 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | context | 5.0158 | 9 | 0.557311 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | context | 4.99362 | 9 | 0.554846 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | context | 4.51314 | 9 | 0.50146 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | context | 4.22332 | 9 | 0.469258 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | context | 4.10284 | 9 | 0.455871 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | context | 4.08979 | 9 | 0.454421 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | context | 4.06318 | 9 | 0.451465 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | context | 4.01117 | 9 | 0.445686 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | context | 3.90982 | 9 | 0.434425 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | context | 3.86897 | 9 | 0.429886 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | context | 3.79416 | 9 | 0.421573 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | context | 3.74591 | 9 | 0.416212 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | context | 3.70126 | 9 | 0.411251 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | context | 3.32657 | 9 | 0.369619 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | context | 3.1452 | 9 | 0.349466 | 5 |
| chatglm3:6:ggmlv3:q4_0 | context | 2.85636 | 9 | 0.317373 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | context | 2.10857 | 9 | 0.234285 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | context | 1.89605 | 9 | 0.210672 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | context | 1.78868 | 9 | 0.198742 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | context | 1.78618 | 9 | 0.198464 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | context | 1.77351 | 9 | 0.197056 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | context | 1.67334 | 9 | 0.185926 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | context | 1.64821 | 9 | 0.183134 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | context | 1.58821 | 9 | 0.176468 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | context | 1.57169 | 9 | 0.174632 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | context | 1.34289 | 9 | 0.14921 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | context | 1.23881 | 9 | 0.137645 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | context | 1.12335 | 9 | 0.124816 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | context | 1.10292 | 9 | 0.122547 | 5 |
| Full model name | Subtask | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|---|
| openhermes-2.5:7:ggufv2:Q8_0 | disease | 6.46667 | 9 | 0.718519 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | disease | 6.46667 | 9 | 0.718519 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | disease | 6.46667 | 9 | 0.718519 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | disease | 6.46667 | 9 | 0.718519 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | disease | 6.46667 | 9 | 0.718519 | 5 |
| gpt-4-0125-preview | disease | 6.21333 | 9 | 0.69037 | 5 |
| gpt-4o-2024-05-13 | disease | 6.2 | 9 | 0.688889 | 5 |
| gpt-4-0613 | disease | 6.13333 | 9 | 0.681481 | 5 |
| gpt-3.5-turbo-0613 | disease | 6.06667 | 9 | 0.674074 | 5 |
| gpt-3.5-turbo-0125 | disease | 4.75238 | 9 | 0.528042 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | disease | 4.32493 | 9 | 0.480548 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | disease | 4.20708 | 9 | 0.467453 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | disease | 4.14674 | 9 | 0.460748 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | disease | 4.02927 | 9 | 0.447696 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | disease | 4.01581 | 9 | 0.446201 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | disease | 3.47244 | 9 | 0.385827 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | disease | 3.04532 | 9 | 0.338369 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | disease | 2.92854 | 9 | 0.325393 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | disease | 2.65437 | 9 | 0.294929 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | disease | 2.57657 | 9 | 0.286285 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | disease | 2.44785 | 9 | 0.271983 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | disease | 2.29171 | 9 | 0.254634 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | disease | 2.29094 | 9 | 0.254549 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | disease | 1.73452 | 9 | 0.192725 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | disease | 1.73452 | 9 | 0.192725 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | disease | 1.73452 | 9 | 0.192725 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | disease | 1.33093 | 9 | 0.147881 | 5 |
| chatglm3:6:ggmlv3:q4_0 | disease | 1.21669 | 9 | 0.135188 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | disease | 0.995894 | 9 | 0.110655 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | disease | 0.306386 | 9 | 0.0340429 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | disease | 0.26663 | 9 | 0.0296256 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | disease | 0.250053 | 9 | 0.0277836 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | disease | 0.235648 | 9 | 0.0261831 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | disease | 0.185035 | 9 | 0.0205595 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | disease | 0.182046 | 9 | 0.0202274 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | disease | 0.179398 | 9 | 0.0199331 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | disease | 0.150208 | 9 | 0.0166897 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | disease | 0.142957 | 9 | 0.0158841 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | disease | 0.103277 | 9 | 0.0114752 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | disease | 0.0898052 | 9 | 0.00997835 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | disease | 0.0874203 | 9 | 0.00971337 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | disease | 0.0587138 | 9 | 0.00652375 | 5 |
| Full model name | Subtask | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|---|
| gpt-4o-2024-05-13 | entity | 5.9909 | 9 | 0.665655 | 5 |
| gpt-4-0125-preview | entity | 4.59502 | 9 | 0.510558 | 5 |
| gpt-3.5-turbo-0613 | entity | 4.57972 | 9 | 0.508858 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | entity | 4.22461 | 9 | 0.469401 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | entity | 4.1344 | 9 | 0.459377 | 5 |
| gpt-4-0613 | entity | 4.12852 | 9 | 0.458724 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | entity | 4.09333 | 9 | 0.454814 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | entity | 4.02016 | 9 | 0.446685 | 5 |
| gpt-3.5-turbo-0125 | entity | 3.71195 | 9 | 0.412439 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | entity | 3.65819 | 9 | 0.406466 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | entity | 2.42313 | 9 | 0.269236 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | entity | 2.33413 | 9 | 0.259348 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | entity | 2.30597 | 9 | 0.256219 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | entity | 2.20283 | 9 | 0.244759 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | entity | 2.10077 | 9 | 0.233419 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | entity | 2.0607 | 9 | 0.228967 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | entity | 2.00802 | 9 | 0.223113 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | entity | 1.99809 | 9 | 0.22201 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | entity | 1.99214 | 9 | 0.221349 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | entity | 1.79999 | 9 | 0.199999 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | entity | 1.77563 | 9 | 0.197292 | 5 |
| chatglm3:6:ggmlv3:q4_0 | entity | 1.22227 | 9 | 0.135808 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | entity | 1.20851 | 9 | 0.134279 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | entity | 1.16189 | 9 | 0.129099 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | entity | 1.10007 | 9 | 0.12223 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | entity | 1.01555 | 9 | 0.112839 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | entity | 0.948961 | 9 | 0.10544 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | entity | 0.903324 | 9 | 0.100369 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | entity | 0.807379 | 9 | 0.0897088 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | entity | 0.785233 | 9 | 0.0872482 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | entity | 0.75253 | 9 | 0.0836144 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | entity | 0.749495 | 9 | 0.0832772 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | entity | 0.699988 | 9 | 0.0777764 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | entity | 0.695524 | 9 | 0.0772805 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | entity | 0.694377 | 9 | 0.077153 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | entity | 0.685368 | 9 | 0.076152 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | entity | 0.685027 | 9 | 0.0761141 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | entity | 0.629764 | 9 | 0.0699737 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | entity | 0.623851 | 9 | 0.0693168 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | entity | 0.623813 | 9 | 0.0693126 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | entity | 0.56502 | 9 | 0.06278 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | entity | 0.318196 | 9 | 0.0353551 | 5 |
| Full model name | Subtask | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|---|
| openhermes-2.5:7:ggufv2:Q2_K | experiment_yes_or_no | 9 | 9 | 1 | 5 |
| gpt-4-0125-preview | experiment_yes_or_no | 9 | 9 | 1 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | experiment_yes_or_no | 9 | 9 | 1 | 5 |
| chatglm3:6:ggmlv3:q4_0 | experiment_yes_or_no | 8.6 | 9 | 0.955556 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | experiment_yes_or_no | 8.33333 | 9 | 0.925926 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | experiment_yes_or_no | 8.33333 | 9 | 0.925926 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | experiment_yes_or_no | 8.33333 | 9 | 0.925926 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | experiment_yes_or_no | 8.025 | 9 | 0.891667 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | experiment_yes_or_no | 8 | 9 | 0.888889 | 5 |
| gpt-4-0613 | experiment_yes_or_no | 8 | 9 | 0.888889 | 5 |
| gpt-4o-2024-05-13 | experiment_yes_or_no | 8 | 9 | 0.888889 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | experiment_yes_or_no | 8 | 9 | 0.888889 | 5 |
| gpt-3.5-turbo-0613 | experiment_yes_or_no | 8 | 9 | 0.888889 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | experiment_yes_or_no | 7.05061 | 9 | 0.783401 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | experiment_yes_or_no | 6.07336 | 9 | 0.674818 | 5 |
| gpt-3.5-turbo-0125 | experiment_yes_or_no | 6.03333 | 9 | 0.67037 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | experiment_yes_or_no | 5.23564 | 9 | 0.581738 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | experiment_yes_or_no | 5.16593 | 9 | 0.573993 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | experiment_yes_or_no | 3.7 | 9 | 0.411111 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | experiment_yes_or_no | 3.68182 | 9 | 0.409091 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | experiment_yes_or_no | 3.32028 | 9 | 0.36892 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | experiment_yes_or_no | 3.26963 | 9 | 0.363292 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | experiment_yes_or_no | 3.0913 | 9 | 0.343478 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | experiment_yes_or_no | 2.36364 | 9 | 0.262626 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | experiment_yes_or_no | 2.36015 | 9 | 0.262239 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | experiment_yes_or_no | 2.2851 | 9 | 0.2539 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | experiment_yes_or_no | 2.2802 | 9 | 0.253355 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | experiment_yes_or_no | 2.06817 | 9 | 0.229797 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | experiment_yes_or_no | 1.89935 | 9 | 0.211039 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | experiment_yes_or_no | 1.45686 | 9 | 0.161873 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | experiment_yes_or_no | 1.29991 | 9 | 0.144434 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | experiment_yes_or_no | 1.1661 | 9 | 0.129567 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | experiment_yes_or_no | 1.15184 | 9 | 0.127982 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | experiment_yes_or_no | 1.06643 | 9 | 0.118492 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | experiment_yes_or_no | 1.03147 | 9 | 0.114607 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | experiment_yes_or_no | 0.785587 | 9 | 0.0872874 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | experiment_yes_or_no | 0.726745 | 9 | 0.0807495 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | experiment_yes_or_no | 0.618798 | 9 | 0.0687554 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | experiment_yes_or_no | 0.468722 | 9 | 0.0520802 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | experiment_yes_or_no | 0.267272 | 9 | 0.0296969 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | experiment_yes_or_no | 0.201489 | 9 | 0.0223876 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | experiment_yes_or_no | 0.130285 | 9 | 0.0144761 | 5 |
| Full model name | Subtask | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|---|
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | hypothesis | 3.67339 | 9 | 0.408154 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | hypothesis | 3.33681 | 9 | 0.370756 | 5 |
| gpt-4-0613 | hypothesis | 3.29696 | 9 | 0.366328 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | hypothesis | 2.9272 | 9 | 0.325244 | 5 |
| gpt-4o-2024-05-13 | hypothesis | 2.89512 | 9 | 0.32168 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | hypothesis | 2.75585 | 9 | 0.306206 | 5 |
| gpt-3.5-turbo-0125 | hypothesis | 2.72775 | 9 | 0.303084 | 5 |
| gpt-3.5-turbo-0613 | hypothesis | 2.64497 | 9 | 0.293885 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | hypothesis | 2.57382 | 9 | 0.28598 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | hypothesis | 2.47292 | 9 | 0.274769 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | hypothesis | 2.37196 | 9 | 0.263551 | 5 |
| gpt-4-0125-preview | hypothesis | 2.33518 | 9 | 0.259465 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | hypothesis | 2.29085 | 9 | 0.254539 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | hypothesis | 2.23255 | 9 | 0.248061 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | hypothesis | 2.09626 | 9 | 0.232917 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | hypothesis | 2.05375 | 9 | 0.228195 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | hypothesis | 1.87442 | 9 | 0.208269 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | hypothesis | 1.83735 | 9 | 0.20415 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | hypothesis | 1.71557 | 9 | 0.190619 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | hypothesis | 1.52181 | 9 | 0.16909 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | hypothesis | 1.4915 | 9 | 0.165722 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | hypothesis | 1.44143 | 9 | 0.160158 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | hypothesis | 1.44009 | 9 | 0.16001 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | hypothesis | 1.4389 | 9 | 0.159878 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | hypothesis | 1.41421 | 9 | 0.157134 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | hypothesis | 1.39565 | 9 | 0.155072 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | hypothesis | 1.13596 | 9 | 0.126218 | 5 |
| chatglm3:6:ggmlv3:q4_0 | hypothesis | 0.98676 | 9 | 0.10964 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | hypothesis | 0.878406 | 9 | 0.0976006 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | hypothesis | 0.876219 | 9 | 0.0973577 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | hypothesis | 0.68638 | 9 | 0.0762645 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | hypothesis | 0.623758 | 9 | 0.0693064 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | hypothesis | 0.62053 | 9 | 0.0689478 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | hypothesis | 0.604423 | 9 | 0.0671582 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | hypothesis | 0.572369 | 9 | 0.0635965 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | hypothesis | 0.55524 | 9 | 0.0616934 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | hypothesis | 0.520453 | 9 | 0.0578281 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | hypothesis | 0.49279 | 9 | 0.0547544 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | hypothesis | 0.424638 | 9 | 0.0471819 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | hypothesis | 0.408017 | 9 | 0.0453352 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | hypothesis | 0.402337 | 9 | 0.0447041 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | hypothesis | 0.366299 | 9 | 0.0406999 | 5 |
| Full model name | Subtask | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|---|
| gpt-4o-2024-05-13 | intervention | 5.34631 | 9 | 0.594034 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | intervention | 4.9841 | 9 | 0.553789 | 5 |
| gpt-4-0125-preview | intervention | 4.92171 | 9 | 0.546857 | 5 |
| gpt-4-0613 | intervention | 4.72253 | 9 | 0.524725 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | intervention | 4.71449 | 9 | 0.523833 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | intervention | 4.44465 | 9 | 0.49385 | 5 |
| gpt-3.5-turbo-0613 | intervention | 4.27143 | 9 | 0.474603 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | intervention | 4.00021 | 9 | 0.444467 | 5 |
| gpt-3.5-turbo-0125 | intervention | 3.75141 | 9 | 0.416823 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | intervention | 3.55238 | 9 | 0.394709 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | intervention | 2.92766 | 9 | 0.325296 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | intervention | 2.23683 | 9 | 0.248536 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | intervention | 2.23319 | 9 | 0.248132 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | intervention | 1.66677 | 9 | 0.185196 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | intervention | 1.23412 | 9 | 0.137124 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | intervention | 1.17173 | 9 | 0.130192 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | intervention | 1.15754 | 9 | 0.128615 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | intervention | 1.02157 | 9 | 0.113507 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | intervention | 0.987919 | 9 | 0.109769 | 5 |
| chatglm3:6:ggmlv3:q4_0 | intervention | 0.881806 | 9 | 0.0979784 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | intervention | 0.879646 | 9 | 0.0977385 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | intervention | 0.723791 | 9 | 0.0804212 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | intervention | 0.680182 | 9 | 0.0755758 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | intervention | 0.668995 | 9 | 0.0743328 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | intervention | 0.640258 | 9 | 0.0711397 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | intervention | 0.550643 | 9 | 0.0611826 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | intervention | 0.542302 | 9 | 0.0602558 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | intervention | 0.502722 | 9 | 0.055858 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | intervention | 0.417501 | 9 | 0.046389 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | intervention | 0.416756 | 9 | 0.0463062 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | intervention | 0.410888 | 9 | 0.0456543 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | intervention | 0.402319 | 9 | 0.0447021 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | intervention | 0.37923 | 9 | 0.0421366 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | intervention | 0.339683 | 9 | 0.0377425 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | intervention | 0.327257 | 9 | 0.0363619 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | intervention | 0.319187 | 9 | 0.0354652 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | intervention | 0.265476 | 9 | 0.0294974 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | intervention | 0.24986 | 9 | 0.0277622 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | intervention | 0.244444 | 9 | 0.0271605 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | intervention | 0.2273 | 9 | 0.0252555 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | intervention | 0.118691 | 9 | 0.0131879 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | intervention | 0.0769231 | 9 | 0.00854701 | 5 |
| Full model name | Subtask | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|---|
| gpt-4-0125-preview | ncbi_link | 6.48768 | 9 | 0.720853 | 5 |
| gpt-4-0613 | ncbi_link | 6.05933 | 9 | 0.673259 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | ncbi_link | 3.5303 | 9 | 0.392256 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | ncbi_link | 3.5303 | 9 | 0.392256 | 5 |
| gpt-4o-2024-05-13 | ncbi_link | 3.51302 | 9 | 0.390335 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | ncbi_link | 3.47436 | 9 | 0.38604 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | ncbi_link | 3.11111 | 9 | 0.345679 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | ncbi_link | 2.37436 | 9 | 0.263818 | 5 |
| gpt-3.5-turbo-0613 | ncbi_link | 2.16667 | 9 | 0.240741 | 5 |
| gpt-3.5-turbo-0125 | ncbi_link | 1.42925 | 9 | 0.158805 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | ncbi_link | 1.03429 | 9 | 0.114921 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | ncbi_link | 0.884957 | 9 | 0.0983286 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | ncbi_link | 0.881705 | 9 | 0.0979672 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | ncbi_link | 0.710989 | 9 | 0.0789988 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | ncbi_link | 0.656812 | 9 | 0.0729791 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | ncbi_link | 0.615714 | 9 | 0.0684127 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | ncbi_link | 0.596131 | 9 | 0.0662368 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | ncbi_link | 0.574422 | 9 | 0.0638246 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | ncbi_link | 0.558824 | 9 | 0.0620915 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | ncbi_link | 0.505458 | 9 | 0.0561621 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | ncbi_link | 0.429927 | 9 | 0.0477697 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | ncbi_link | 0.328564 | 9 | 0.0365071 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | ncbi_link | 0.271548 | 9 | 0.030172 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | ncbi_link | 0.255217 | 9 | 0.0283574 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | ncbi_link | 0.253735 | 9 | 0.0281928 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | ncbi_link | 0.246231 | 9 | 0.027359 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | ncbi_link | 0.241357 | 9 | 0.0268174 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | ncbi_link | 0.236802 | 9 | 0.0263113 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | ncbi_link | 0.233815 | 9 | 0.0259795 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | ncbi_link | 0.230909 | 9 | 0.0256566 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | ncbi_link | 0.216341 | 9 | 0.0240378 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | ncbi_link | 0.196981 | 9 | 0.0218868 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | ncbi_link | 0.192574 | 9 | 0.0213971 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | ncbi_link | 0.179211 | 9 | 0.0199123 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | ncbi_link | 0.177339 | 9 | 0.0197044 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | ncbi_link | 0.173014 | 9 | 0.0192238 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | ncbi_link | 0.170952 | 9 | 0.0189947 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | ncbi_link | 0.166777 | 9 | 0.0185308 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | ncbi_link | 0.166614 | 9 | 0.0185127 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | ncbi_link | 0.15271 | 9 | 0.0169677 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | ncbi_link | 0.150011 | 9 | 0.0166679 | 5 |
| chatglm3:6:ggmlv3:q4_0 | ncbi_link | 0.122857 | 9 | 0.0136508 | 5 |
| Full model name | Subtask | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|---|
| gpt-4-0613 | significance | 5.6 | 9 | 0.622222 | 5 |
| gpt-4-0125-preview | significance | 5.18384 | 9 | 0.575982 | 5 |
| gpt-4o-2024-05-13 | significance | 4.22424 | 9 | 0.46936 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | significance | 3.92996 | 9 | 0.436662 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | significance | 3.78182 | 9 | 0.420202 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | significance | 3.78182 | 9 | 0.420202 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | significance | 3.77787 | 9 | 0.419763 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | significance | 3.69091 | 9 | 0.410101 | 5 |
| gpt-3.5-turbo-0613 | significance | 3.58562 | 9 | 0.398402 | 5 |
| gpt-3.5-turbo-0125 | significance | 3.51717 | 9 | 0.390797 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | significance | 2.93833 | 9 | 0.326481 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | significance | 2.87928 | 9 | 0.31992 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | significance | 2.79423 | 9 | 0.31047 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | significance | 2.62296 | 9 | 0.29144 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | significance | 2.56724 | 9 | 0.285249 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | significance | 2.48514 | 9 | 0.276127 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | significance | 2.4813 | 9 | 0.2757 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | significance | 1.50696 | 9 | 0.16744 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | significance | 1.34869 | 9 | 0.149854 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | significance | 1.31454 | 9 | 0.14606 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | significance | 1.2312 | 9 | 0.1368 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | significance | 1.01129 | 9 | 0.112366 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | significance | 0.994971 | 9 | 0.110552 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | significance | 0.957259 | 9 | 0.106362 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | significance | 0.758379 | 9 | 0.0842644 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | significance | 0.716547 | 9 | 0.0796163 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | significance | 0.68386 | 9 | 0.0759845 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | significance | 0.636128 | 9 | 0.0706809 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | significance | 0.518572 | 9 | 0.0576191 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | significance | 0.329457 | 9 | 0.0366064 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | significance | 0.326026 | 9 | 0.0362251 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | significance | 0.281188 | 9 | 0.0312431 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | significance | 0.228461 | 9 | 0.0253845 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | significance | 0.213246 | 9 | 0.023694 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | significance | 0.207957 | 9 | 0.0231063 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | significance | 0.205271 | 9 | 0.0228079 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | significance | 0.194946 | 9 | 0.0216607 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | significance | 0.178078 | 9 | 0.0197864 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | significance | 0.131484 | 9 | 0.0146093 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | significance | 0.123914 | 9 | 0.0137682 | 5 |
| chatglm3:6:ggmlv3:q4_0 | significance | 0.118153 | 9 | 0.0131281 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | significance | 0.103278 | 9 | 0.0114753 | 5 |
| Full model name | Subtask | Score achieved | Score possible | Accuracy | Iterations |
|---|---|---|---|---|---|
| gpt-4-0125-preview | stats | 8.86667 | 9 | 0.985185 | 5 |
| openhermes-2.5:7:ggufv2:Q8_0 | stats | 8.66667 | 9 | 0.962963 | 5 |
| openhermes-2.5:7:ggufv2:Q6_K | stats | 8.66667 | 9 | 0.962963 | 5 |
| openhermes-2.5:7:ggufv2:Q5_K_M | stats | 8.52821 | 9 | 0.947578 | 5 |
| gpt-4o-2024-05-13 | stats | 8.51282 | 9 | 0.945869 | 5 |
| gpt-4-0613 | stats | 8.51282 | 9 | 0.945869 | 5 |
| openhermes-2.5:7:ggufv2:Q4_K_M | stats | 8.25641 | 9 | 0.917379 | 5 |
| openhermes-2.5:7:ggufv2:Q3_K_M | stats | 8.05641 | 9 | 0.895157 | 5 |
| openhermes-2.5:7:ggufv2:Q2_K | stats | 8 | 9 | 0.888889 | 5 |
| gpt-3.5-turbo-0613 | stats | 7.98135 | 9 | 0.886817 | 5 |
| gpt-3.5-turbo-0125 | stats | 7.12976 | 9 | 0.792195 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q5_K_M | stats | 6.89091 | 9 | 0.765657 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q4_K_M | stats | 6.29908 | 9 | 0.699898 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q6_K | stats | 6.18322 | 9 | 0.687024 | 5 |
| llama-3-instruct:8:ggufv2:Q8_0 | stats | 5.17406 | 9 | 0.574895 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q3_K_M | stats | 5.1041 | 9 | 0.567122 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q8_0 | stats | 5.04591 | 9 | 0.560657 | 5 |
| mistral-instruct-v0.2:7:ggufv2:Q2_K | stats | 4.63496 | 9 | 0.514995 | 5 |
| llama-3-instruct:8:ggufv2:Q6_K | stats | 4.30739 | 9 | 0.478599 | 5 |
| llama-3-instruct:8:ggufv2:Q5_K_M | stats | 3.9346 | 9 | 0.437178 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q8_0 | stats | 3.60737 | 9 | 0.400819 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q5_K_M | stats | 3.58841 | 9 | 0.398712 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q6_K | stats | 3.21213 | 9 | 0.356904 | 5 |
| llama-2-chat:70:ggufv2:Q4_K_M | stats | 3.08109 | 9 | 0.342344 | 5 |
| llama-3-instruct:8:ggufv2:Q4_K_M | stats | 2.98843 | 9 | 0.332048 | 5 |
| llama-2-chat:70:ggufv2:Q2_K | stats | 2.65216 | 9 | 0.294684 | 5 |
| llama-2-chat:70:ggufv2:Q3_K_M | stats | 2.44276 | 9 | 0.271418 | 5 |
| llama-2-chat:70:ggufv2:Q5_K_M | stats | 2.3993 | 9 | 0.266589 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q4_K_M | stats | 2.21549 | 9 | 0.246165 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q3_K_M | stats | 1.96241 | 9 | 0.218045 | 5 |
| mixtral-instruct-v0.1:46_7:ggufv2:Q2_K | stats | 1.76057 | 9 | 0.195619 | 5 |
| llama-2-chat:7:ggufv2:Q4_K_M | stats | 1.43589 | 9 | 0.159543 | 5 |
| llama-2-chat:13:ggufv2:Q4_K_M | stats | 1.41695 | 9 | 0.157439 | 5 |
| llama-2-chat:13:ggufv2:Q8_0 | stats | 1.38608 | 9 | 0.154009 | 5 |
| llama-2-chat:7:ggufv2:Q5_K_M | stats | 1.3859 | 9 | 0.153989 | 5 |
| llama-2-chat:13:ggufv2:Q3_K_M | stats | 1.35834 | 9 | 0.150927 | 5 |
| llama-2-chat:13:ggufv2:Q5_K_M | stats | 1.3371 | 9 | 0.148567 | 5 |
| llama-2-chat:13:ggufv2:Q2_K | stats | 1.12439 | 9 | 0.124932 | 5 |
| code-llama-instruct:7:ggufv2:Q4_K_M | stats | 0.860471 | 9 | 0.0956079 | 5 |
| llama-2-chat:7:ggufv2:Q3_K_M | stats | 0.804538 | 9 | 0.0893931 | 5 |
| llama-2-chat:7:ggufv2:Q2_K | stats | 0.558031 | 9 | 0.0620034 | 5 |
| chatglm3:6:ggmlv3:q4_0 | stats | 0.17925 | 9 | 0.0199166 | 5 |
