AI Evaluator - POLISH

About Us

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.

This opening is on behalf of one of our clients, and we’ll work closely with you to make the process clear and straightforward.

Role Overview

We are looking for Polish-speaking AI Content Analysts to support evaluation of a new personalization capability within a leading AI assistant platform.

In this role, you will design realistic conversational prompts based on your own context and experiences, then rigorously evaluate how effectively the AI uses personal signals (such as prior conversations or activity context) to produce relevant, grounded, and helpful responses.

This position combines creative prompt design, analytical evaluation, and structured quality assessment, making it ideal for candidates with experience in AI evaluation, annotation, content review, or analytical research roles.

Responsibilities

• Design and run short multi-turn conversations (typically 1–5 turns) intended to test AI personalization behavior
• Create prompts grounded in realistic personal scenarios to evaluate contextual understanding
• Review AI responses to determine whether personalization is correctly applied
• Check grounding quality to ensure the model does not invent unsupported claims about the user
• Evaluate integration quality — confirming personal signals are used naturally (not forced or robotic)
• Compare two responses side-by-side and determine which is more helpful, natural, and relevant
• Write clear, structured rationales explaining rankings and referencing specific conversation turns
• Verify debug information showing whether correct data sources were used
• Maintain strict workflow hygiene (including deleting evaluation conversations when required)

Notes:

• The role is 100% remote, working hours within your local time zone (this is a MUST)
• 30-40 hours/week commitment. Paid by hours logged and approved.
• Contracting Model
• Duration: 1 month (possible extension)

• Strong Polish proficiency (reading & writing required) — Polish is the primary evaluation language
• BS/BA degree or equivalent experience in a relevant field (e.g., Policy, Law, Ethics, Linguistics, Journalism, Computer Science, or a related analytical field)
• Strong analytical thinking and ability to assess nuanced AI outputs
• Excellent written communication skills with ability to produce structured evaluation notes
• High attention to detail when comparing similar responses
• Ability to work independently in a fully remote environment
• Reliable desktop/laptop and stable internet connection
• Willingness to use your primary personal Google account and enable personal data sources for evaluation purposes