The excitement around ChatGPT 3.5 underscores its potential to transform various fields in education, including STEM. However, we must approach these claims cautiously, especially concerning AI’s impact on STEM education. While AI can enhance STEM education, there are ethical concerns as well as potential inaccuracies linked to unsupervised automated responses.

To comprehensively evaluate ChatGPT’s influence on STEM, we conducted a controlled experiment which involved answering a question set in mathematics and computer science in a time-limited session. Students who used ChatGPT were tasked with explaining how and where they employed the tool. Conversely, students who did not use ChatGPT were asked to showcase their problem-solving process. We carefully analyzed the responses from these four groups, alongside the analysis of ChatGPT conversations.

Performance, confidence level, and completion time of each participant were recorded. Experts in mathematics and computer science were then consulted to review participant responses. These experts were subsequently interviewed to gain deeper insights and draw conclusive findings.

Our findings shows that students who didn’t use ChatGPT in Mathematics produced solutions that scored better than those who did. Similarly, in programming, ChatGPT led to less elegant code. Our findings provide valuable insights into the benefits and challenges of AI integration in these fields, helping educators to adapt their teaching style to the AI advancements.