Gen AI / ML Application Testing Engineer (BA + QA)

Company: ADPMN Inc
Location: San Jose
Posted on: May 3, 2025

Job Description:

Job Title :: Gen AI / ML Application Testing Engineer (BA + QA)Location :: San Jose, CA - Hybrid - Tue, Wed, Thu (ONLY LOCALS)Duration :: Long TermRole SummaryWe are seeking a detail-oriented engineer with experience in Gen AI / ML application testing, business analysis, and product validation. The successful candidate will help shape the quality of next-generation AI products through systematic testing, prompt validation, and tool-driven evaluation.Key Responsibilities

• Design and execute test cases for Gen AI / ML features and user workflows
• Utilize GenAI-specific tools to validate prompt consistency, hallucination detection, etc.
• Collaborate with product managers to convert requirements into test cases and test data
• Perform exploratory testing, regression testing, and prompt-based scenario testing
• Develop automation scripts to simulate user behavior and backend interactions
• Track and manage issues using QA platforms and agile tools
• Document test plans, test reports, and AI evaluation metricsRequired Skills

• Hands-on testing experience with Gen AI / ML products
• Experience with LLM testing tools such as:
  • Promptfoo (prompt testing & evaluation)
  • LangSmith (LangChain tracing & evals)
  • TruLens (feedback tracking for LLMs)
  • Rebuff (security and behavior testing)
  • Solid understanding of LLM behavior, hallucinations, prompt design
  • Scripting skills in Python, Shell, or JavaScript
  • Experience with REST APIs, JSON, YAML
  • Familiarity with testing tools like PyTest, Postman, Selenium, or similarNice-to-Have Skills
    • Experience testing RAG, chatbot, or LLM agent systems
    • Familiarity with LangChain, LlamaIndex, or Haystack
    • Business analysis experience in AI projects
    • Knowledge of AI/ML model evaluation metricsEducationBachelor's or Master's degree in CS, Data Science, AI/ML, or related field
      #J-18808-Ljbffr

Keywords: ADPMN Inc, Alameda , Gen AI / ML Application Testing Engineer (BA + QA), Engineering , San Jose, California