Say hello to ChatGPT: The ultimate conversational AI assistant
February 8, 2023
“OpenAI has released ChatGPT, a state-of-the-art language model designed to make it easier for developers to create conversational AI systems that can understand and respond to human inputs in a more natural and intuitive way. ChatGPT is able to generate responses based on context and handle long-form input, making it well-suited for applications such as chatbots and virtual assistants.”
The paragraph you just read was not written by a human being. In fact, neither was the headline. While the above text, not nearly perfect, still contains plenty of errors that would make any practiced journalist cringe, ChatGPT is not wrong when it states that it has “revolutionized” the way our society interacts with software. Thanks to artificial intelligence, a news lede that would have taken a human reporter at least 10 minutes to research and write can now be completed in less than 10 seconds.
After its release on Nov. 30, ChatGPT soared in popularity, gaining over 1 million users in less than a week. According to OpenAI, the company that developed ChatGPT, the chatbot can do much more than simply respond to prompts. Its ability to “answer follow-up questions, admit its mistakes, challenge incorrect premises and reject inappropriate requests,” all while maintaining a human-like conversation with the user, sets this language model apart from its predecessors as arguably the most advanced artificial intelligence to date.
“I am not a real person and do not have a personal identity or background like a human,” ChatGPT said when prompted to describe itself. “I was designed to be able to communicate with users in a natural and helpful way and to provide accurate and useful information on a wide range of topics. I do not have personal experiences or feelings, but I am able to understand and respond to the experiences and feelings of others.”
Unlike past computer chatbots, ChatGPT uses reinforcement learning to craft elegant responses to almost any prompt based on a snapshot of the internet from 2021. Additionally, ChatGPT can also remember its past answers to prompts, meaning that the bot is constantly learning and improving.
“ChatGPT has this unique ability to look at a well-formed grammatical sentence, identify the keywords from those sentences and use natural language in its response in a very strongly formed grammatical manner,” upper school computer science teacher Anu Datar said. “It also uses a feedback loop that allows the A.I. to learn from its past interactions, so ChatGPT has a bizarre ability to remember things. No other A.I. has done that before.”
ChatGPT’s almost uncanny ability to use information from the internet and present it in human-like forms has quickly given rise to many discussions surrounding its possible use for homework problems and, in some cases, even exams.
How Smart is ChatGPT?
To test ChatGPT’s accuracy and ability to imitate a high school student, Harker Aquila entered three sample free-response questions from past Advanced Placement (AP) exams and presented the AI’s solutions for teachers to grade.
1. AP United States History
Scroll down to see answer and analysis.
Answer: Pass
Explanation (given by upper school history teacher James Tate): “There’s a lot of correct content there that does address the question and provide a relatively good answer. The order in which it is presented strikes me as a bit odd. The chatbot says, ‘One specific historical change in relations was the signing of the Treaty of Greenville in 1795.’ The signing of a treaty might encompass or represent or delineate changes in relations, but the signing of a treaty itself was not a change in relations. If ChatGPT had opened with something like ‘A historical change in relations between American Indians and the United States was the Native American tribes in the Ohio Valley turning to a more peaceful set of relations which is encompassed in this particular treaty,’ that would logically make more sense to me. The ordering of the presentation of information strikes me as a bit odd; the word ‘artificial’ immediately comes to mind. Maybe I wouldn’t be thinking of the word ‘artificial’ if I didn’t know an AI wrote it, but it still feels a bit odd the way that that’s presented. If I had read this, I wouldn’t immediately think AI, but I would think it’s weird because it addresses the question in sort of an oblique way. It comes at it from a logic that’s at least very different from the way I would think about it.”
2. AP Calculus BC
Scroll down to see answer and analysis.
Answer: Fail
Explanation (given by upper school mathematics teacher Bradley Stoll): “From an AP standpoint, there’s a couple of things that are wrong. For one, the name of the function was changed at various times. It went from capital G to lowercase g, and those are actually two different functions. Second of all, from an AP standpoint, they would not accept an answer that was rounded to the tenths, not the thousandths, so that could lose a point. Another thing was it didn’t use proper notation. It used the word ‘Integral,’ and it didn’t use the integral symbol from one to four, and the College Board does not accept calculator notation. Even without verifying that the answer was right, there were three mistakes in there. The math was all right, integrating from one to four, dividing by the length of the interval, and then setting that equal. So it’s kind of impressive that it did all that. It’s just a little bit interesting that it changed a few things in there. I also think if a student went into that depth of writing an explanation like ChatGPT did, I think it would be pretty obvious that they got it from somewhere else.”
3. AP Computer Science A
Scroll down to see answer and analysis.
Answer: Pass
Explanation (given by upper school computer science teacher Anu Datar): “The answer is beautifully composed, and the code is written quite well because [ChatGPT] followed all Java naming conventions and coding standards. It did use a construct that is not on the AP curriculum, and it used a random class instead of using ‘Math.random’. But that’s a minor thing because as an AP reader, I’ve also been told that even if the construct is outside the AP curriculum, we are supposed to award points if it works. Although, I would recognize that this is either a student who self-studied or a student who knows more than what they’re expected to for the AP because they’re using constructs that are beyond the AP curriculum. I’ve seen a response like this written by an actual student on the AP when I’ve been grading, but I know that those students either self-study or are taking the AP after working in Java for three or four years.”
While ChatGPT nails the general problem-solving process of the questions and excels at a surface-level examination, it falls short in the specifics of arriving at a definitive answer and properly imitating the structure of a student-produced response. Even so, both Tate and Stoll expressed concerns about students plagiarizing the chatbot and submitting its work as their own.
“I think that while there might be an argument to be made that using ChatGPT isn’t technically plagiarism, doing that goes against the spirit of the law, which is that work done in an academic setting is supposed to represent not only your ideas but also your effort,” Tate said. “Using an AI is neither your ideas nor your effort.”
ChatGPT’s performance on the problems demonstrates that the AI can now successfully impersonate a human student, for the most part. Although platforms like GPTZero are already uncovering ways of detecting ChatGPT’s writing, they have still yet to reach the same popularity as the program they were designed to target. For now, students will have to use their own judgment for now to decide if using a chatbot for their homework is really the best decision.