In higher education, introductory Computer Science (CS) programs offer a range of foundational courses. These encompass not only the standard CS1 and CS2 courses but may also include more specialized options like CS0 and CS1.5. In order to appropriately assign students to the suitable introductory courses, many institutions utilize placement tests, which assess students’ pre-existing knowledge and skills. While most institutions rely on accuracy alone to make these determinations, there is often additional information concealed within the completed tests. This paper delves into the potential of Large Language Models (LLMs) to uncover this hidden information, particularly in gaining insights into how students perform in different concepts. Moreover, our framework has the flexibility to accommodate variations in curricula across different institutions, providing additional analytical perspectives. Initially, we built a concept inventory (CI) using the concepts covered in an institution’s CS0, CS1, and CS2 curricula. Next, an LLM, specifically GPT 3.5, was applied to associate each question in the placement test with one or more concepts in the CI. Finally, the results of the placement tests were scrutinized, allowing the calculation of mastery levels in each concept for individual students. These mastery levels enable institutions to gauge a student’s prior knowledge across various concepts simply by using a CS placement test. Additionally, we presented a case study demonstrating the application of this framework to 267 existing placement test results at Boston College.