LLM embeddings on test items predict post hoc loadings in personality tests.

In this article we examine the application of Large Language Models (LLMs) in predicting factor loadings in personality tests through the semantic analysis of test items. By leveraging text embeddings generated from LLMs, we assess the semantic similarity of test items and their alignment with hypothesized factorial structures, without relying on human response data. Our methodology uses embeddings from the Big Five personality test to explore correlations between item semantics and their grouping in factorial analyses. Preliminary results suggest that LLM-derived embeddings can effectively capture semantic similarities among test items, potentially serving as a valid measure for initial survey design and refinement. This approach offers insights into the robustness of embedding techniques in psychological evaluations, indicating a significant correlation with traditional test structures and providing a novel perspective on test item analysis.