Priya Raman
AI Trainer & RLHF Specialist
Bengaluru, India
Summary
AI trainer with 5 years producing the human feedback that shapes model behaviour. Wrote the rating guidelines used by a 120-person annotation team and raised inter-rater agreement from 0.61 to 0.87. Progressed from rater to guideline author to quality lead.
Experience
RLHF Quality Lead · Lumen Data Services
Jan 2024 – Present
- Author rating guidelines for preference and safety tasks used daily by 120 raters across three time zones.
- Raised inter-rater agreement from 0.61 to 0.87 by rewriting ambiguous criteria and adding worked examples.
- Designed the gold-set sampling process used to audit rater quality weekly, catching drift within days rather than at project end.
- Run calibration sessions and adjudicate contested cases; built the escalation path for genuine disagreements.
Senior AI Rater · Lumen Data Services
Mar 2022 – Dec 2023
- Rated model outputs for helpfulness, factual accuracy and safety across text and multimodal tasks.
- Flagged a systematic mislabel pattern in the guidelines that had affected roughly 4,000 prior ratings.
- Mentored 15 new raters through onboarding and became the reference point for hard safety cases.
Education
MA English Literature
Christ University · 2018 – 2020
BA English
University of Delhi · 2015 – 2018
Certifications
- Certified Quality Analyst (CQA) — QAI
Skills
Human Feedback: RLHF · Preference Ranking · Annotation Guidelines · Inter-Rater Reliability · Calibration
Quality: Gold Sets · Sampling Design · Audit Workflows · Rater Training · Drift Detection
Judgement Areas: Factual Accuracy · Safety Policy · Instruction Following · Tone & Helpfulness