We are seeking a Senior AI Test Automation Engineer to lead the quality assurance efforts for AI/ML-powered applications and systems. This role will be critical in designing and implementing test strategies that ensure the accuracy, robustness, fairness, and reliability of AI models and ML pipelines. The ideal candidate will combine deep experience in test automation frameworks with a strong understanding of AI/ML workflows and data science practices.
Key Responsibilities
Design, develop, and maintain automated test frameworks and test suites for AI/ML systems, APIs, and data pipelines.
Validate the integrity, performance, and accuracy of machine learning models in both development and production environments.
Implement test coverage for edge cases, fairness, bias, explainability, and model drift in ML workflows.
Collaborate with data scientists and ML engineers to review model assumptions, inputs, and outputs.
Create synthetic and real-world test data to evaluate AI/ML model behavior under various conditions.
Monitor CI/CD pipelines to ensure models and services are thoroughly tested before deployment.
Apply data validation and statistical testing to verify training data quality and results consistency.
Drive root cause analysis of model failures and identify opportunities to improve test effectiveness.
Lead performance benchmarking and regression testing for ML services at scale.
Primary Skills (Must-Have)
Test Automation Expertise in tools like PyTest, Robot Framework, Selenium, Postman, or REST Assured.
Python Programming Strong experience with Python, particularly in writing test scripts for ML/AI services.
ML/AI Knowledge Understanding of machine learning lifecycle, supervised/unsupervised models, and common ML frameworks (e.g., scikit-learn, TensorFlow, PyTorch).
Data Validation Experience with tools/libraries like Great Expectations, Pandera, or custom Python-based validation.
CI/CD Integration Proficiency with Jenkins, GitLab CI, or similar tools for integrating automated tests in pipelines.
API Testing Proven ability to test RESTful APIs and ML endpoints.
Model Evaluation Metrics Familiarity with metrics like precision, recall, F1 score, confusion matrix, AUC-ROC.
Secondary Skills (Nice-to-Have)
Cloud Platforms Familiarity with AWS, Azure, or GCP for testing AI services deployed on cloud infrastructure.
ML Monitoring Tools Exposure to tools like Evidently AI, MLflow, or Weights & Biases.
Bias/Fairness Testing Knowledge of ethical AI practices and tools like Fairlearn or Aequitas.
Containerization Experience with Docker/Kubernetes to manage test environments.
Performance Testing Tools like JMeter, Locust, or k6.
Version Control Git/GitHub for codebase management.
SQL and Data Engineering Basics For testing data pipelines and verifying large datasets.
Qualifications
Bachelor or Master degree in Computer Science, Data Science, Engineering, or related field.
5+ years of experience in test automation with at least 2+ years working in AI/ML environments.
Demonstrated ability to work in cross-functional agile teams with ML engineers and data scientists.
Strong analytical, problem-solving, and communication skills
About Virtusa
Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth — one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.
Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.
Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.
