AI Conversation Research & Evaluation Specialist

About Gramian Consulting

Gramian Consulting is a boutique consultancy specializing in IT professional services and engineering talent solutions. The firm leverages a strong background in software engineering and leadership to connect companies with professionals who meet their specific needs, thereby supporting the development of high-performing teams.

The Opportunity: AI Conversation Research & Evaluation Specialist

Gramian Consulting is seeking an AI Conversation Research & Evaluation Specialist to support AI conversation evaluation research. This full-time role, based in Saudi Arabia, involves reviewing interactions between individuals and AI models to assess contextual understanding and identify opportunities for improvement. The ideal candidate will possess 2-5 years of relevant professional experience and strong analytical skills.

Key Responsibilities

• Review AI conversations that involve real-world interpersonal situations and everyday decision-making.
• Identify missing context and recommend relevant follow-up questions to enhance the quality of AI responses.
• Evaluate whether gathering additional information would significantly alter an AI-generated recommendation.
• Distinguish between critical information that impacts a decision and non-essential or merely descriptive details.
• Conduct qualitative reviews and document findings using structured evaluation guidelines.
• Assess if conversations adequately consider relevant personal, psychological, social, financial, practical, and organizational factors.
• Identify missed opportunities for clarification, deeper understanding, and information gathering.
• Review conversations related to relationships, work, education, family, wellbeing, planning, and resource constraints.
• Collaborate with research teams to ensure consistent, evidence-based assessments.
• Maintain high standards of accuracy, clarity, and consistency while adhering to project deadlines.

Required Qualifications and Experience

Candidates must have at least 3-5 years of relevant direct professional experience in one or more of the following domains:

• Counseling, Therapy, or Psychology:

  • Master's degree in Social Work (MSW) or equivalent graduate-level qualification, with LCSW, LICSW, LMSW, or an equivalent applicable social work license/credential. Experience in client assessment, case review, interviewing, or counseling.
  • Master's degree in a relevant discipline with an LMFT, LPC, LCPC, or equivalent counseling/therapy license/credential.
  • PhD or PsyD in Clinical, Counseling, Social, or Industrial-Organizational Psychology. Relevant experience in counseling, therapy, psychological assessment, interviewing, or client-facing professional work.

• Qualitative Research, HCI, or Communication:

  • MA, PhD, or equivalent advanced expertise in Human-Computer Interaction, Communication, Anthropology, Sociology, Qualitative Research, or a closely related discipline.
  • Experience conducting qualitative research, interviews, user research, conversation analysis, thematic analysis, or qualitative coding.
  • Ability to systematically analyze conversations and identify missing context, communication patterns, and opportunities for clarification.

• Mediation, Coaching, Student Services, Organizational Consulting, or Related Advisory Fields:

  • Relevant professional qualification, credential, or demonstrated subject-matter expertise.
  • At least 5 years of direct advisory or client-facing experience for candidates qualifying through this pathway.
  • Experience in coaching, mediation, student advising, organizational consulting, or comparable professional work.
  • Demonstrated ability to gather context, understand individual or organizational needs, and provide informed recommendations.

Skills and Attributes

• Strong analytical skills with the ability to distinguish information that could materially change a recommendation from less relevant details.
• Ability to provide structured feedback to improve conversational quality.

Work Type

This is a full-time position.