LLM数据评测专家

Job Description

主要职责：

1. 评估体系设计：针对客服对话场景，设计多轮对话评估维度与打分细则（如上下文一致性、角色稳定性、是否跑题、工具/知识使用是否合理等）。编写与迭代 标注指南（SOP）、典型案例库与边界案例说明，降低歧义与口径漂移。推进LLM相关标注及评测需求，包括不仅限于对话流程/function/prompt工程等。

2. 数据质量与评测执行：对多轮对话样本做人工评估、对比评测（A/B、side-by-side） 及 Bad case归因（是数据问题、模型问题还是评测标准问题）。组织或参与 抽样质检、交叉审核、跟踪一致率、分歧原因，推动标准修订。管理并赋能标注团队，严格把控标注质量及项目交付周期。

3. 与模型/业务/平台对齐： 协同算法、研发等团队，推进模型能力的迭代提升，解决模型训练及效果过程中的数据问题。将业务目标 翻译为可标注、可统计的指标，配合构建 对话评测集 与 回归测试集，支撑版本迭代与问题定位。协助平台工具侧完成相关流程及功能测试，维护标注流程稳定性。

Requirements

任职要求：

1. 本科及以上学历及CET 6以上；语言学、心理学、传播学、人机交互、计算机、数据科学等相关专业或同等经验。2年以上 与 NLP / 大模型 / 对话系统相关的数据标注、质量评估、评测设计经验；有多轮对话项目经验优先。

2. 具备较强的 文本细读与逻辑判断能力，能清晰写出 可执行的标注规则。

3. 有 Python / SQL 做简单数据分析者加分，会使用 Linux优先。

4. 责任心强，抗压能力强，能适应 标准迭代快、需要大量沟通的工作节奏。

加分项：

1. 参与过 RLHF / 偏好数据 / 对话安全与合规 相关评估。

2. 了解对话状态、指代消解、幻觉、越狱、角色设定漂移 等多轮场景典型问题。

3. 有英语听说读写能力，可阅读技术文档或参与双语评测。

软性素质：

1. 表达清晰，善于把模糊需求变成可落地的 rubric。

2. 对边界案例 敏感，愿意推动 标准文档化 而非仅靠口头共识。

3. 具备数据与直觉的平衡：既相信统计，也能从个案中提炼规律。

About AI Rudder

AI Rudder is an AI product company transforming how business communicate to their customers.

We launched in 2019 with one of the world’s first enterprise-grade AI voice agents: built to hold real, intelligent conversations, not just play back scripts. Today, we serve over 500 enterprises across 20+ countries, processing 50 million calls every day and having delivered more than 8 billion calls to date. Our customers range from fast-growing startups to large enterprises.

When AI talks to people, it has to feel human. That’s what we build.

How We Work

AI-first: AI isn’t a tool we reach for. It’s the default we start from. Every team, every workflow: AI-first.

Global Collaboration: We operate globally and work as one team. Distance is a logistics problem, not a cultural one. We build the habits that make distributed teams actually work

Communication Principle: Simple, Trust, Transparent. We remove unnecessary complexity so people can focus on what matters most.

What We Offer

• Learning & Development: Continuous learning through real projects, knowledge sharing session, mentorship, and cross-functional collaboration.
• Performance Recognition: Regular performance reviews, feedback, and opportunities for career progression.
• Global Exposure: Collaborate with international teams, global rotations, and work with global enterprise customers.
• Meaningful Impact: Build AI solutions that create real value for businesses around the world.
• Ownership & Autonomy: Freedom to take initiative and drive outcomes.
• Team Engagement: Team Building, Local & Global Gatherings & Local Events Celebration

AI Rudder is an Equal Opportunity Employer. We celebrate diversity and are committed to creating an inclusive environment for all employees and applicants.