Role Summary
Own quality across the POD through automation. Both cover functional, API and integration testing.
Key Responsibilities
- Build and maintain automated test suites across the Python and JVM services and React UIs.
- Own API, contract and integration testing for the Orchestrator, Workflow Builder, Connectors and MCP / A2A surfaces.
- Integrate tests into CI/CD gates; report coverage and quality signals.
- AI-eval focus: design and run evaluation harnesses for the Signal / Action Engines and Librarian Agent — accuracy, hallucination, regression and guardrail checks for LLM/agent outputs.
- Performance & data focus: build load and resilience tests for orchestration and event driven flows, and data-quality tests for pipelines.
Must-have Skills & Technologies
- Test automation with Pytest / Playwright / Selenium and a JVM framework (REST Assured, JUnit) for API and UI testing.
- API and contract testing (Postman, Pact) and CI integration.
- One of: LLM/agent evaluation frameworks and techniques (for the AI-eval role) or performance testing (k6, JMeter, Locust) and data-quality validation (for the performance/data role).
Prior Experience
- 3–6 years as an SDET / automation-focused QA engineer.
- Shipped automation running in CI for microservices; comfortable reading application code.
Nice to Have
- Experience testing event-driven / workflow systems (Temporal, Kafka / Event Hub).
- Familiarity with AI/LLM quality practices or data observability.
