Job Description

About the Company

We are a seed-stage AI company building the industry standard for evaluating and benchmarking large language models on real enterprise tasks.

About the Role

As a Research Scientist, you will develop new benchmarks, methodologies, and evaluation pipelines that shape how cutting-edge models are assessed, compared, and deployed in production environments. Your work will directly influence model selection and safety decisions across foundation model labs, high-growth AI product companies, and Fortune-scale enterprises.

Responsibilities

Benchmarking & Model Analysis

• Evaluate newly released models as they launch (e.g., Gemini, DeepSeek, etc.)
• Run large-scale assessment workflows using internal evaluation infrastructure
• Compare model performance across enterprise-grade task categories

Design New Benchmarks from Scratch

• Identify high-value model application domains through research exploration
• Construct datasets, including labeling strategy and workforce coordination
• Write short white-paper-style summaries explaining benchmark purpose, method, and findings

Advance Automated Evaluation Methodologies

• Improve systems for scoring generated text beyond standard metrics
• Explore research in reference-free, rubric-based, and human-aligned evaluation
• Develop new techniques for reliability, consistency, and repeatability

Cross-functional Collaboration

• Work closely with engineering to scale evaluation infra
• Partner with customers to refine evaluation relevance and task fit
• Influence product direction through research insights

Qualifications

• 03 years post-grad experience (Masters, PhD, or equivalent applied research)
• Publications, preprints, or demos showing cutting-edge work
• Experience with diffusion models, NLP, multimodal, or benchmarking work
• Ability to operate independently with ambiguity
• Clean, maintainable research codebases
• Candidates from:
• top engineering/research universities
• applied AI startups
• well-regarded research internships

Required Skills

• Built or contributed to a benchmark or evaluation methodology
• Experience in enterprise task model evaluation
• Stanford / top lab adjacency (per their historical hiring success)

Preferred Skills

• Wants to publish as primary motivation
• Purely academic with slow timelines
• Big-tech culture fit concerns (Meta / Google / Salesforce specifically notedbut case-by-case)

Pay range and compensation package

• Base Salary: up to $250K depending on background
• Equity: typically 0.3% 0.5%, flexibility for exceptional candidates
• Rapid equity refresh possible based on impact

Equal Opportunity Statement

Visa sponsorship available. Relocation support. Health & dental coverage. Lunch + dinner provided, snacks & coffee. Unlimited PTO. Weekly happy hours with community guests. Team events (bowling, hiking, rock climbing, etc.). Swag program (hats, etc.).

Work Environment & Culture

In-person, San Francisco HQ (required). Core hours: 95, some teammates extend voluntarily. Most team members work 1 weekend day per week (flexible). High-ownership, low-ego, collaborative. Live demos Mondays, team lunch Thursdays, community Fridays. Early-stage pace, applied focusnot academic publishing.

Tech Environment

(while research-focused, exposure beneficial) Backend: Python / Django. Frontend: React + TypeScript. Infra: AWS. Evaluation frameworks + internal tooling.

Why This Role Is Unique

The company already collaborates with foundation model labs, high-growth AI vertical product companies, and Fortune 500 enterprises (not publicly facing). ChatGPT Vals AI $5M seed raised, runway of 2+ years at current burn. Only one research scientist is being hiredtrue founding impact. Opportunity to define industry standards for model trust, reliability, and certification. Positioned to become the rating agency for generative AI.

Job Tags

Similar Jobs