Exam Development Manager

The Examinations Development Manager is responsible for compliance with documented policies and procedures as it pertains to exam development and maintenance. This person will work with Subject Matter Experts (SMEs), stakeholders, vendors, and staff to develop and maintain examinations at standardized rigor for continued assurance of validity and reliability as well as fairness and impartiality and meeting relevant and required standards (e.g., ISO/ICE 17024, ASTM E2659).

The Examinations Development Manager is charged with meeting the needs of ISA’s current and future credential holders by managing the development and maintenance of all ISA’s credentialing examinations (including both written and performance-based examinations). This is accomplished by utilizing experience in exam development and maintenance, data analysis, psychometrics and testing, volunteer recruitment and engagement, and project management. 

ESSENTIAL DUTIES AND RESPONSIBILITIES include the following. Other duties may be assigned. 

Examination Development & Maintenance

• Leads the design, development, revision, and maintenance of certification examinations.
• Manages examination development and publication projects from end to end through publication and implementation.
• Ensures examination questions are designed to measure competence based on, and consistent with, the scheme for each given credential. 
• Ensures examination forms (including forms associated with performance-based examinations) have comparability of results for each single examination, both in content and difficulty, including validity of fail/pass decisions. 
• Anticipates item and test development needs for all ISA examinations and build multi-year plans to meet those needs. 
• Manages SME review and approval of examination items and exam forms in a timely and cyclical fashion according to documented policies and procedures, organization needs and industry best practices. 
• Responsible for item bank management system, ensuring statistical data is available to inform and report on item and exam development activities and needs. 
• Works with Publication and Documentation Lead and other key staff to ensure that relevant policies, procedures and communication is updated and accurate with regard to examinations (e.g., credentialing schemes, exam outlines, cut scores) 
• Coordinates standard setting studies and implementation of passing scores. 
• Verifies blueprint alignment, item accuracy, scoring integrity, and examination formatting. 

Psychometrics, Quality & Continuous Improvement

• Collaborates with psychometric vendor(s) to ensure examinations meet accepted psychometric standards.
• Reviews item statistics and examination performance data to support continuous improvement. Monitors examination form quality using statistical analyses and candidate performance trends. 
• Coordinates standard setting studies and implementation of passing score changes. 
• Recommends improvements based on psychometric findings and stakeholder feedback. 
• Maintains documentation supporting examination development decisions and audit readiness. 

Subject Matter Expert (SME) & Committee Management

• Recruits, trains, and manages diverse panels of subject matter experts.
• Trains Subject Matter Experts (SMEs) on current best management practices for item development and review. 
• Serves as main point of contact for translators, editors, and SMEs working on item development, review, and translation. 
• Facilitates SME meetings (virtual and in-person) related to item and exam development and maintenance well as to ensure validity and reliability of both written and performance-based examinations. 
• Facilitates item writing workshops and examination review meetings. 
• Ensures SME activities are documented and follow established governance procedures.

Quality Assurance & Examination Security 

• Establishes and implements safeguards for item banking, examination publication, and content handling.
• Complies with ISO/IEC 17024 to prevent fraudulent examination practices, monitor examination statistics to ensure the integrity of the examination process, and works with vendors (if applicable) to investigate security incidents, breaches, and other concerns related to exam security. 
• Protects the confidentiality and security of examination content throughout its lifecycle. 
• Participates in investigations involving examination security incidents or compromised content. 

Examination Program Leadership & Operations

• Develops professional relationships with appropriate industry stakeholders, fellow staff members, volunteers, vendors, and others in the credentialing industry. 
• Utilizes external services to support and ensure that quality credentialing examinations are developed and maintained in a psychometrically sound, legally defensible manner. 
• Fosters an environment of continuous improvement. Encourages staff to look for ways to improve department policies, procedures, processes and operations that result in improved efficiencies and positive experience for ISA candidates and      credential holders. 
• Oversees the daily transfer of appropriate Computer Based Testing and Paper and Pencil exam data and statistics into test banking software according to departmental policies and procedures. 
• Serves as point person for examination publication and project management. 
• Ensure exam development process documentation is kept current and accurate. 
• Manage examination development timelines, budgets, and project deliverables. 
• Monitor vendor performance related to examination development, publication and delivery. 

Documentation, Governance & Collaboration

• Work with Publication and Documentation Lead and other key staff to ensure that relevant      policies, procedures and communication is updated and accurate with regard      to examinations validity and reliability (e.g., credentialing schemes,   exam outlines, cut scores). 

Supervisory Responsibilities 

Yes; 1 direct report. 

Clearly defines expectations of staff to meet departmental goals. Coaches and counsels as needed and holds staff accountable to deliver results. Assists with removal of roadblocks and prioritization. 

QUALIFICATIONS To perform this job successfully, an individual must be able to perform each essential duty satisfactorily. The requirements listed below are representative of the knowledge, skill, and/or ability required. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions. 

Education and/or Experience 

· 4-year degree from an accredited college or university required. Education, Instructional Design, Organizational Psychology preferred; 

· Previous experience in examination development required. 

· Previous supervisory or volunteer management experience preferred.

· Prior project management experience exhibiting superior analytical and organizational skills with high levels of attention to detail. 

· Involvement and/or experience with professional, high-stakes, accredited credentialing program management is preferred. 

Language Skills 

· Fluency in English, both verbal and written. Fluency in additional languages is a plus (especially for one of the languages in which our examinations are delivered). 

· Excellent written and oral communication skills. 

· Ability to read, write, analyze, and interpret complex documents. 

· Ability to respond diplomatically to sensitive inquiries or complaints. 

· Ability to write articles for various publications and audiences. 

· Ability to make effective and persuasive speeches and presentations on controversial or complex topics to top management, public groups, volunteers, and/or boards of directors. 

Decision Making/Reasoning Ability 

· Ability to define problems, collect data, establish facts, and draw valid conclusions. 

· Ability to interpret an extensive variety of technical instructions in mathematical or diagram form and deal with several abstract and concrete variables. 

· Ability to demonstrate good judgment in knowing which issues require escalation to director, SMEs, vendors, and other relevant stakeholders. 

Additional Competencies 

· Ability to assist in the creation of budgets and interpret and analyze financial data. 

· Ability to understand and interpret psychometric data associated with managing test bank questions, which requires higher level mathematical operations such as frequency distribution, determination of test reliability and validity, analysis of variance, correlation techniques, sampling theory, and factor analysis. 

· Ability to communicate clearly and consistently with key staff members, vendors, and volunteer SMEs concerning the performance of essential duties and responsibilities for the position. 

· Ability to inspire and motivate staff and volunteers.

· Ability to coordinate efforts and workflow with key staff members from all levels and departments of the organization. 

· Ability to maintain positive attitude under stressful conditions.

Physical Demands The physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions. 

• Must be able to remain in a stationary position 80% of the time. 

• Moves about inside the office to access file cabinets, office machinery, etc. 

• Regularly operates office productivity machinery such as printer, copier, etc.

• Frequently communicates with internal and external customers. 

• This position requires occasional lifting, under 25 lbs. 

Work Environment The work environment characteristics described here are representative of those an employee encounters while performing the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions. 

• Working conditions are normal for an office environment. Work may require weekend and/or evening work. 

• The noise level in the work environment is usually quiet to moderate. 

• Domestic and occasional international travel may be required. 

Technology and Equipment 

· Latest Microsoft Windows/Office Applications; proficient in Word, PowerPoint, Outlook, and Teams. 

· Advanced Microsoft Excel skills is a plus.

· Experience with Pearson VUE applications is a plus.

· Experience with examination question banks and other content management systems is a plus. 

· Experience using AI for examination development and translation is a plus.

· Database applications, and the Internet.