Week 12 - User Testing with physical screens

This week, user tested various students using ChatGPT when doing homework. For these tests, we came with printed screens to allow users to best understand the product. These played a decisive role in conducting fascinating and insightful user tests, which were carried out in a two-pronged manner.

The first axis was observation - how does the student prepare homework?

One student copied the homework question into ChatGPT, skimmed the answer, and copied it into a notebook. The other, copied the first question into ChatGPT, copied the second question into ChatGPT, read the answers and copied parts of them into his notebook. Another student, copied the question to ChatGPT, read the answer and copied part of it to the notebook.

The second axis was an imaginary scenario - in which we asked the students "Suppose you received a question from the history teacher about the French Revolution. What do you do first thing?'

All students copied the question from their homework into ChatGPT.

After that, we continued the conversation with them and asked them different questions such as "After you received the answer, what do you do?" and "Let's say X appears on your side, have you read the answer of the chat?" What does that make you think? What does it make you do? What do you think the purpose of this is?'

Quotes from user tests:

"If there was someone who stopped me and told me like that, I would think twice and check things better."

"I will still rely on the chat and his answer, but I will look in the notebook if they tell me."

"If I see that it doesn't match the material, I won't use it."

From the user tests, we realized that CRITI can help users develop critical thinking towards the answers of the ChatGPT homework preparation.