NILEAGI-SUB · Available
Swahili Language Understanding
Sector-by-sector Swahili understanding under one controlled protocol. First report: education, SUB-MCQ EDUCATION with Level I results on compact open models.
2,569
Validated items
8
Open models (Level I)
27 pts
Accuracy spread (~2-5B)
Highlight · August 2026
SUB-MCQ EDUCATION
2,569 expert-reviewed Swahili multiple-choice items from the Swahili-subject curriculum (Standard 3-7 and Form 2-4). Featured in the Cohere Labs Open Science Community.
Items by grade
| Grade band | Items | Share |
|---|---|---|
| Standard 3 | 66 | 2.6% |
| Standard 4 | 116 | 4.5% |
| Standard 5 | 274 | 10.7% |
| Standard 6 | 651 | 25.3% |
| Standard 7 | 517 | 20.1% |
| Form 2 | 399 | 15.5% |
| Form 3 | 125 | 4.9% |
| Form 4 | 421 | 16.4% |
| Total | 2,569 | 100% |
Level I results (~2-5B open models)
Eight open or openly accessible models evaluated on identical items with deterministic decoding and unified exact-match scoring. Accuracy spans about 27 points at a shared parameter budget. Chance baseline ≈ 22.4%.
| # | Model | Params | Accuracy |
|---|---|---|---|
| 1 | Gemma 4 E4B | 4.5B | 54.8% |
| 2 | AfriqueQwen3.5-4B | 4.0B | 50.3% |
| 3 | AfriqueGemma-4B | 4.0B | 43.4% |
| 4 | Tiny Aya Earth | 3.35B | 41.4% |
| 5 | Tiny Aya Global | 3.35B | 40.7% |
| 6 | Qwen3.5-4B | 4.0B | 35.4% |
| 7 | Qwen3.5-2B | 2.0B | 29.1% |
| 8 | Llama 3.2 3B | 3.2B | 27.8% |
Resources
Item preview
Format preview and sample items from the education release.
Dataset page
Methods, grade bands, and access for SUB-MCQ EDUCATION.
Full technical report
Methods, limitations, and grade-stratified results.