[Remote] Prompt Engineer (Model Behavior & Evaluation)

Note: The job is a remote job and is open to candidates in USA. Yuna Health is dedicated to transforming access to mental health support through empathetic conversational AI. The Prompt Engineer will be responsible for defining and evaluating AI conversation behaviors, collaborating with clinicians, and continuously improving the quality and alignment of conversations at scale.

Responsibilities

• Own conversational behavior across a multi-agent, multi-model conversational system
• Collaborate with clinicians to design and operate evaluation frameworks for conversational quality, empathy, usefulness, and alignment
• Diagnose failures by analyzing real conversation traces, agent routing, memory usage, and system context
• Use product data and qualitative review to prioritize high-impact improvements
• Evaluate and select models across providers (e.g., OpenAI, Claude, Gemini, Vertex AI) based on task, cost, latency, and qualitative output Skills
• Demonstrated experience owning model or agent behavior in production
• Hands-on experience designing or operating evaluation systems (qualitative + quantitative)
• Ability to define what “good” means and defend it with evidence, not vibes
• Comfort working hands-on with tools like LangGraph, LangSmith, or comparable tracing and evaluation frameworks
• High data literacy: able to reason from logs, traces, metrics, and analytics to isolate root causes
• Clear communicator who can collaborate across product, clinical, and engineering teams in ambiguous problem spaces
• Working knowledge of Python; ability to read, debug, and collaborate within agent logic and evaluation code
• Familiarity with alignment concepts (e.g. human values, safety tradeoffs, refusal behavior)
• Proficient with SQL, amplitude, and excel
• Background in psychology, linguistics, neuroscience, education, or adjacent fields Benefits
• Competitive salary (based on experience) + equity options
• Remote-first culture with flexibility
• A fast-growing, talented, and empathetic team dedicated to transforming mental health care
• An opportunity to use AI for good
• Level of ownership to make measurable impact, building cutting-edge AI systems that improve lives every day Company Overview
• Our mission at Yuna is to radically transform the way mental health support is accessed and delivered. It was founded in undefined, and is headquartered in San Francisco, California, USA, with a workforce of 11-50 employees. Its website is

Apply tot his job Apply To this Job