Software Engineer (AI Technical Advisor)

Note

The job is a remote job and is open to candidates in USA. Bee Talent Solutions is helping its client identify exceptional software engineers to evaluate where frontier AI models fail on real engineering work. The role involves solving difficult problems in production-grade codebases, reviewing model-generated pull requests, building evaluation benchmarks, and collaborating with AI researchers to document technical failures and reasoning.

Responsibilities

• Evaluate Agent Sessions: Analyze how AI coding agents behave over entire coding sessions—tracking what the agent investigates, verifies, assumes, and leaves undone
• Review Model-Written PRs: Audit model-generated pull requests against real production repositories, documenting every identified issue alongside its severity and detailed technical rationale
• Build Evaluation Benchmarks: Design and construct hard, container-based problems that serve as rigorous test benchmarks for frontier models
• Collaborate with AI Researchers: Work directly alongside AI researchers on frontier problems, producing clear written analyses that explain root-cause failures and boundary-condition gaps
• Maintain High Written Rationale Standards: Author original, clear technical rationale for all evaluations (all written deliverables must be independently authored without AI text generation, though AI tools are welcomed for codebase exploration and running test suites)
• Solve difficult engineering problems across real, production-grade codebases
• Identify where model-generated code fails, and articulate precisely why
• Work directly with Anthropic researchers on problems they are actively investigating
• Hold a technical bar that others build on

Skills

• • Are comfortable dropping into unfamiliar codebases and languages you don't use every day - the repos change weekly
• • Are comfortable with containers and reproducing results locally
• • Can explain your reasoning as clearly as you can write the solution
• • Are more interested in why a solution is right than in shipping it quickly
• • Want to spend a focused period on this rather than committing to a permanent role
• • Have deep, demonstrated expertise as a software engineer - we care about the depth of your judgment, not your years
• • Write code that other strong engineers learn from
• • Have reviewed a lot of other people's code, and are known for catching what CI and the author both missed

Company Overview

• Bee Talent Solutions provides business consulting and provisioning talent solutions for engineering and life sciences companies. It was founded in 2020, and is headquartered in Kirkland, Washington, USA, with a workforce of 51-200 employees. Its website is https://www.beetalentsolutions.com.