AI Evaluation Engineer - Planning & Operations

About Us

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.

Role overview

We are looking for an AI Evaluation Engineer specialized in planning and operations to design and build benchmark tasks that simulate real-world scenarios such as scheduling, logistics, and resource allocation.

This role focuses on planning, scheduling, and operational optimization problems, where multiple agents must collaborate to solve constraint-rich scenarios involving resources, timelines, and dependencies.

Commitments Required: 8 hours per day with an overlap of 4 hours with PST.

Employment type: Contractor assignment (no medical/paid leave)

Duration of contract: 4 weeks+

Location: Bangladesh, Brazil, Colombia, Egypt, Ghana, India, Indonesia, Kenya, Nigeria,Turkey, Vietnam

Interview: take home assessment (60min) + short interview

Responsibilities

• Design and build multi-agent benchmark tasks involving:
• Planning, scheduling, and resource allocation
• Operational decision-making (logistics, project planning, incident response, capacity planning)
• Create constraint-rich problem statements with multiple interacting variables
• Develop verification scripts to evaluate:
• Feasibility (all constraints satisfied)
• Completeness (all requirements met)
• Optimality (efficiency of solutions)
• Define task decomposition strategies across specialized sub-agents (e.g., resource allocation, constraint resolution, optimization)
• Model realistic operational systems with dependencies, timelines, and constraints
• Implement validation logic and evaluation pipelines using Python
• Work with Docker environments for reproducibility and execution
• Collaborate with internal teams to improve task quality, coverage, and evaluation rigor

• 5+ years of experience in operations, project management, logistics, or supply chain
• Strong ability to formalize constraints, dependencies, and scheduling logic
• Proficiency in Python for building validation and verification scripts
• Experience with optimization techniques (linear programming, constraint satisfaction, scheduling algorithms)
• Strong structured problem-solving and decomposition skills
• Experience with AI benchmarks or evaluation frameworks (e.g., SWE-bench or similar)
• Hands-on experience with Docker (Dockerfiles, image builds, debugging)

Nice to Have

• Background in operations research or optimization-heavy domains
• Experience with simulation or modeling tools
• Familiarity with AI planning systems or automated reasoning
• Project management experience or certifications (PMP, Agile, etc.)