AI Automation Engineer _ Virtual

$160,000 - $190,000/yr

AI Automation AI Workflow Design (using GPTs / LLMs) Automation Architecture & Maintenance Data Quality Validation OpenAI API GitHub Actions

Alight Solutions

Apply Now

Role Overview

Join Alight's AI Center of Excellence as an AI Automation Engineer, where you'll play a pivotal role in establishing the foundational automation framework for AI Quality. This position offers an exciting opportunity to work at the forefront of AI testing and quality assurance, building and scaling automated test suites for cutting-edge AI models and agents.

You'll collaborate closely with AI Quality Leads and developers to implement comprehensive testing frameworks for LLM, RAG, agentic, and classic ML systems. This role combines technical expertise with innovation, allowing you to shape the future of AI quality assurance while working with the latest tools and technologies in a rapidly evolving field.

As part of a growing global organization, you'll contribute to responsible AI practices, bias mitigation, and regulatory compliance while building robust automation processes that ensure the reliability and safety of AI systems.

Responsibilities

Develop and maintain automated test frameworks for LLM, RAG, agentic, and classic ML systems
Implement golden/adversarial test cases covering functional, safety, bias, prompt injection, refusal correctness, and tool use scenarios
Integrate evaluation tools (promptfoo, Ragas) into CI/CD pipelines and support batch quality assessments
Author and validate OpenAPI/JSON Schema contracts for model inputs/outputs and enforce schema checks in pipelines
Instrument observability metrics (latency, success rate, cost, drift) using OpenTelemetry and export to dashboards
Support data QA, lineage tracking, and drift monitoring using tools like Evidently and OpenLineage
Collaborate with AI developers, SDETs, and SRE teams to troubleshoot, optimize, and document automation processes
Contribute to red-team playbooks and security testbooks mapped to OWASP/ATLAS risks

Requirements

Experience in test automation, software engineering, or quality assurance
Proficiency in Python, JavaScript/Node.js, or similar programming languages
Familiarity with CI/CD, version control, and cloud-based deployment environments
Experience with test automation tools (promptfoo, Ragas, Postman)
Understanding of AI model evaluation metrics (faithfulness, answer relevancy, hallucination, refusal correctness)
Basic knowledge of fairness, safety, and security testing for AI systems
Experience with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
Bachelor's degree in Computer Science, Engineering, or related field, or equivalent practical experience
US Citizenship required (role supports federal government services)

Nice to Have

AI/ML experience
Interest in Responsible AI and bias mitigation
Knowledge of regulatory compliance in AI systems

Benefits & Perks

Competitive total rewards package with salary range $160,000 - $190,000
Comprehensive health coverage options
Wellbeing and support programs
Retirement plans
Vacation and sick leave
Maternity, paternity & adoption leave
Continuing education and training opportunities
Flexible working arrangements
Various voluntary benefit options

About the Company

Alight Solutions is a global organization that helps clients gain a benefits advantage while building healthy and financially secure workforces by unifying the benefits ecosystem across health, wealth, wellbeing, absence management and navigation. The company has been recognized as a "Top 100 Company for Remote Jobs" for 5 consecutive years and champions people-focused values with a commitment to diversity and inclusion.