← Back to Job Board

Consultancy for the Independent Evaluation of the FarajaMH Generative AI Screening Model Before and After Fine-Tuning

African Population and Health Research Center (APHRC)

Nairobi, Kenya · contract

Posted 19 Nov 2026 · Apply by 10 Dec 2026

Apply

About

APHRC seeks an individual consultant or small team to design, deploy on APHRC premises and run an independent, reproducible evaluation of the FarajaMH generative AI screening model before and after fine-tuning. The consultancy will support model selection, evaluate accuracy, safety, language understanding, cultural fit and fairness, and hand over a reusable evaluation harness by 31 August 2027. The lead consultant requires advanced relevant education or equivalent experience and at least five years' machine-learning experience, including evaluation of language or speech models. Duty station: APHRC servers in Nairobi (and, for Tanzanian data, the CUHAS node in Mwanza).

How to apply

Create a free account to apply

Sign up and add a short profile, then see how to apply and open the official advert. We'll track every application for you, alert you to jobs like this one, and you'll apply faster next time.

About two minutes, and you come straight back to this job. Cazini never charges to apply.

Responsibilities

  • Review the work package three (WP3) Technical Methodology, the work package (WP3 Model Evaluation Strategy, the FarajaMH study protocols and the work package four (WP4) design and agree the evaluation plan with the principal investigator (PI) and Work package three (WP3) team.
  • Finalise and lock the evaluation protocol before any model is run. The protocol is hash-pinned (SHA-256) and not changed once results are seen.
  • Map each evaluation measure to the governance decision it informs base-model selection, the per-round PROMOTE, RETRAIN or ROLLBACK gate, and the Track A freeze trigger.
  • Onboard on the existing evaluation platform or install, configure and document their evaluation platform on APHRC servers.
  • Work with clinicians, the WP2 team, linguists and Lived Experience Advisory Committee (LEAC) members to complete, quality-check and freeze the Testing Set v1.0: core cases (B1), language (B2), idioms of distress (B3), safety (B4), audio (B5), the FarajaMH MCQ set and the public benchmarks.
  • Validate the Golden Set (500 to 1,000 clinician-adjudicated examples) for coverage of all conditions, languages, sites and WP4 demographic strata, and confirm that it never enters training.
  • Calibrate the judge models against clinician ratings for each language variety (accepted at Krippendorf’s alpha ≥ 0.67; otherwise, all replies in that variety go to human raters).
  • Evaluate every candidate model, as released by its developer, under identical conditions: hard gates (native audio, Swahili smoke test, LoRA dry run on one node, no critical safety failure) and the nine test areas.
  • Score and rank the candidates with 95% bootstrap confidence intervals, DeLong tests for paired AUROC differences and a rank-stability analysis; shortlist the finalists.
  • Re-test the finalists after the APHRC team's short LoRA fine-tuning probe and prepare the base-model decision pack for tri-committee sign-off.
  • Implement and validate the automated per-round Golden Set gate, with every PROMOTE, RETRAIN or ROLLBACK decision written to the audit trail.
  • At the end of each data wave within the contract period, re-run the full locked evaluation on the merged SFT adapter and on the DPO-aligned adapter, and report the change from the pre-fine-tuning baseline for every area.
  • Report classification performance and generative quality by site and by country, including the held-out external test set.
  • Check for catastrophic forgetting across waves, registers and modalities.
  • Evaluate the safety layer: single-turn and multi-turn red-teaming, HarmBench and XSTest in English and Swahili, refusal scripts, content filters, the suicide-risk escalation pathway and over-refusal.

Qualifications

  • A PhD, or a master's degree with substantial equivalent experience, in machine learning, computer science, biostatistics, health data science or a closely related field.
  • At least five years' experience in machine learning, with a demonstrated record of evaluating large language or speech models (for example with lm-evaluation-harness, vLLM or Hugging Face tooling).
  • Experience evaluating AI for health, preferably mental health screening or clinical decision support, including safety and red-teaming.
  • Strong statistical skills: AUROC and calibration, bootstrap confidence intervals, inter-rater agreement and subgroup fairness analysis.
  • Experience deploying containerized GPU workloads on on-premises or institutional servers, with no reliance on cloud services.
  • Experience designing human evaluation with clinicians and community members, ideally including people with lived experience.
  • Knowledge of data protection law and research ethics in Kenya or Tanzania.
  • Working knowledge of Swahili; experience with speech or audio model evaluation; experience with federated learning; and peer-reviewed publications in AI evaluation are an added advantage.

More jobs in Nairobi

Get new jobs by email

Free, one email a day at most, unsubscribe any time.