Journal of Social Science and Human Research Studies (JSSHRS) · Research article · Open access

Measuring AI-Pedagogical Knowledge in Pre-Service Teachers: Development and Initial Validation of a Vignette-Based Assessment

Abdul Hafid, Rukayah, Muh. Faisal, Nurul Mukhlisah Abdal, Ahmad Syawaluddin

2 October 2026 · Vol. 2, Issue 10, pp. 1815-1826

DOI: 10.65150/EP-jsshrs/V2E10/2026-02

Download full text (PDF)Cite this article

Abstract

The growing use of generative artificial intelligence (AI) in education creates a need for assessments that measure teachers’ pedagogical judgement rather than relying solely on self-reported competence. This study developed and examined initial validity evidence for a vignette-based assessment of AI-pedagogical knowledge among pre-service teachers. Instrument development proceeded from construct definition and blueprinting to an initial pool of 36 researcher-developed vignettes, developer screening, a diagnostic pilot with 60 pre-service teachers, systematic item revision, and content review by three independent experts. The revised 24-item assessment was administered to an analytic sample of 484 pre-service teachers. Validity evidence was examined through expert ratings, internal structure, item and distractor functioning, score reliability and conditional precision, differential item functioning, and relations with external variables. Aiken’s V coefficients ranged from .83 to 1.00 (M = .94), although the small expert panel and restricted use of rating categories warrant cautious interpretation. A one-factor model adequately represented the response data, supporting interpretation of a total score rather than separate domain scores (CFI = .990, TLI = .989, RMSEA = .034, SRMR = .065). Internal consistency was high (KR-20 = .93; categorical omega = .95), but precision was concentrated at the lower end of the ability distribution. Of the participants, 24.2% obtained the maximum score and 47.5% scored 22 or higher. No item showed non-negligible differential functioning by gender or field of study. Knowledge scores were moderately associated with AI self-efficacy (r = .36) and teaching self-efficacy (r = .33) and more weakly associated with self-reported instructional practice (r = .22). The assessment may support formative and research uses when interpreted as a single total score, but it is not yet suitable for ranking or high-stakes decisions. Further development requires more difficult items, revision of weak items and distractors, cognitive interviews, and cross-institutional evaluation.

Keywords: AI-pedagogical knowledge, Pre-service teachers, Vignette-based assessment, Instrument development and validation, Generative artificial intelligence, Teacher education, Psychometric properties.

Abdul Hafid, Rukayah, Muh. Faisal, Nurul Mukhlisah Abdal, & Ahmad Syawaluddin (2026). Measuring AI-Pedagogical Knowledge in Pre-Service Teachers: Development and Initial Validation of a Vignette-Based Assessment. Journal of Social Science and Human Research Studies, 2(10), 1815-1826. https://doi.org/10.65150/EP-jsshrs/V2E10/2026-02

@article{Abdul2026,
  title   = {Measuring AI-Pedagogical Knowledge in Pre-Service Teachers: Development and Initial Validation of a Vignette-Based Assessment},
  author  = {Abdul Hafid and Rukayah and Muh. Faisal and Nurul Mukhlisah Abdal and Ahmad Syawaluddin},
  journal = {Journal of Social Science and Human Research Studies},
  year    = {2026},
  volume  = {2},
  number  = {10},
  pages   = {1815-1826},
  doi     = {10.65150/EP-jsshrs/V2E10/2026-02},
  url     = {https://doi.org/10.65150/EP-jsshrs/V2E10/2026-02}
}

← All articlesAbout JSSHRS and how to submit