Journal of Social Science and Human Research Studies (JSSHRS)

Measuring AI-Pedagogical Knowledge in Pre-Service Teachers: Development and Initial Validation of a Vignette-Based Assessment

Hafid, Abdul, Rukayah, Muh. Faisal, Abdal, Nurul Mukhlisah, Syawaluddin, Ahmad

2 October 2026 · Vol. 2, Issue 10, pp. 1815-1826

DOI: 10.65150/EP-jsshrs/V2E10/2026-02

Abstract

The growing use of generative artificial intelligence (AI) in education creates a need for assessments that measure teachers’ pedagogical judgement rather than relying solely on self-reported competence. This study developed and examined initial validity evidence for a vignette-based assessment of AI-pedagogical knowledge among pre-service teachers. Instrument development proceeded from construct definition and blueprinting to an initial pool of 36 researcher-developed vignettes, developer screening, a diagnostic pilot with 60 pre-service teachers, systematic item revision, and content review by three independent experts. The revised 24-item assessment was administered to an analytic sample of 484 pre-service teachers. Validity evidence was examined through expert ratings, internal structure, item and distractor functioning, score reliability and conditional precision, differential item functioning, and relations with external variables. Aiken’s V coefficients ranged from .83 to 1.00 (M = .94), although the small expert panel and restricted use of rating categories warrant cautious interpretation. A one-factor model adequately represented the response data, supporting interpretation of a total score rather than separate domain scores (CFI = .990, TLI = .989, RMSEA = .034, SRMR = .065). Internal consistency was high (KR-20 = .93; categorical omega = .95), but precision was concentrated at the lower end of the ability distribution. Of the participants, 24.2% obtained the maximum score and 47.5% scored 22 or higher. No item showed non-negligible differential functioning by gender or field of study. Knowledge scores were moderately associated with AI self-efficacy (r = .36) and teaching self-efficacy (r = .33) and more weakly associated with self-reported instructional practice (r = .22). The assessment may support formative and research uses when interpreted as a single total score, but it is not yet suitable for ranking or high-stakes decisions. Further development requires more difficult items, revision of weak items and distractors, cognitive interviews, and cross-institutional evaluation.

Keywords: AI-pedagogical knowledge, Pre-service teachers, Vignette-based assessment, Instrument development and validation, Generative artificial intelligence, Teacher education, Psychometric properties.

Read the full text on Journal of Social Science and Human Research Studies →The full peer-reviewed article and PDF are hosted on the journal's site (the version of record).

Hafid, Abdul, Rukayah, Muh. Faisal, Abdal, Nurul Mukhlisah, Syawaluddin, & Ahmad (2026). Measuring AI-Pedagogical Knowledge in Pre-Service Teachers: Development and Initial Validation of a Vignette-Based Assessment. Journal of Social Science and Human Research Studies, 2(10), 1815-1826. https://doi.org/10.65150/EP-jsshrs/V2E10/2026-02

@article{Hafid2026,
  title   = {Measuring AI-Pedagogical Knowledge in Pre-Service Teachers: Development and Initial Validation of a Vignette-Based Assessment},
  author  = {Hafid and Abdul and Rukayah and Muh. Faisal and Abdal and Nurul Mukhlisah and Syawaluddin and Ahmad},
  journal = {Journal of Social Science and Human Research Studies},
  year    = {2026},
  volume  = {2},
  number  = {10},
  pages   = {1815-1826},
  doi     = {10.65150/EP-jsshrs/V2E10/2026-02},
  url     = {https://doi.org/10.65150/EP-jsshrs/V2E10/2026-02}
}

← Back to all articles