Senior Data Scientist AI Evaluation

Accountabilities

• AI Evaluation Design:Define ground truth, quality metrics, scoring methodologies, and evaluation frameworks to assess the accuracy, consistency, and reliability of AI models and agents.

• Evaluation Pipeline Development:Build repeatable evaluation loops that monitor model quality over time, identify performance regressions, and support reliable release decisions.

• Statistical Measurement and Validation:Apply rigorous statistical methods to evaluation design, including sample sizing, confidence intervals, significance testing, and techniques for handling non-deterministic model outputs.

• Automated Grader Validation:Compare automated evaluation methods with human assessments to establish reliability, identify limitations, and improve scoring accuracy.

• Evaluation Infrastructure Collaboration:Partner with Engineering and Analytics Engineering to operationalize evaluation frameworks, integrate testing into development workflows, and support scalable evaluation infrastructure.

• Performance Analysis and Improvement:Interpret evaluation results, identify weaknesses and quality gaps, and provide actionable recommendations to improve model and agent performance.

• Quality Standards and Documentation:Establish evaluation guidelines, documentation practices, review processes, and consistent quality standards across AI development initiatives.

• Cross-Functional Partnership:Collaborate with Product, Engineering, Analytics Engineering, and business stakeholders to define success criteria and align evaluation priorities with business needs.

• Continuous Improvement and Mentorship:Promote evaluation best practices, share technical insights, and help foster a culture of measurable AI quality, accountability, and evidence-based decision-making.

• Safe AI Deployment:Support teams in making informed release decisions through independent, reliable assessments of model performance and readiness.

Requirements:

• Approximately 6–10 years of experience in quantitative data science, machine learning, or a related technical discipline, with focused experience in measurement, evaluation, experimentation, or model validation.

• Strong quantitative and statistical foundations, including experience designing rigorous experiments and interpreting results with appropriate uncertainty measures.

• Demonstrated experience defining meaningful evaluation metrics, establishing ground truth, and assessing the quality of complex or ambiguous model outputs.

• Proficiency in Python and SQL, with experience applying these skills to data analysis, model assessment, and evaluation workflows.

• Experience evaluating machine learning models in production environments and translating findings into practical improvements.

• Ability to validate automated graders against human judgments and account for variability, non-determinism, and potential measurement bias.

• Strong problem-solving and analytical judgment, particularly when developing evaluation approaches for new or evolving AI systems.

• Excellent communication skills, with the ability to explain evaluation results, trade-offs, and recommendations to technical teams and business stakeholders.

• Proven ability to collaborate across functions while independently owning complex analytical projects in a fast-paced environment.

• A quantitative degree in data science, statistics, mathematics, computer science, or a related field is an advantage; equivalent practical experience is also welcome.

• Hands-on experience evaluating large language models (LLMs) or AI agents in production, including evaluation harnesses, LLM-as-judge calibration, and continuous integration regression gates, is a plus.

• Experience evaluating text-to-SQL systems, analytics agents, or other AI applications where outputs can be verified against underlying data is an advantage.

• Background in fintech, brokerage, financial services, or other domains where incorrect outputs can create significant business or risk consequences is desirable.

• Familiarity with AI tools used in research, analysis, and engineering workflows is beneficial.

Benefits:

• Competitive compensation:Salary package designed to reflect experience and expertise.

• Stock options:Opportunity to participate in the company's long-term growth.

• Health benefits:Benefits designed to support employee health and well-being.

• Home-office setup allowance:One-time allowance of USD $500 to support your remote workspace.

• Monthly stipend:USD $150 per month provided through a Brex card.

• Remote work environment:Opportunity to work remotely within the Americas, with Canada as the target location for this listing.

• Technical ownership:Lead the development of evaluation practices and establish standards that influence AI quality across the organization.

• Cross-functional impact:Work closely with product, engineering, analytics, and business teams to turn rigorous measurement into better AI systems.

• Professional development:Build expertise in AI evaluation, model validation, and the responsible deployment of intelligent systems in a growing technology environment.