The Instrument Doesn't Decide What Question to Ask

Last week Carroll College convened a panel on Magnifica humanitas, Leo XIV's encyclical on artificial intelligence, and what it asks of a Catholic liberal arts college. The bishop of the diocese, a philosophy professor, and the college's director of academic technology each took up the encyclical directly, laying out the genuine complexity and genuine promise of AI in education. It was a substantive evening before the fourth speaker stood up.

Dr. Rebecca Coates teaches general chemistry. She opened by narrowing her claim as far as it would go.

I am not a theologian. I am a chemist, and I teach chemistry. What I can tell you about is only my classroom, not others, and what watching my students use AI has taught me about what I actually want for them.

We should say plainly what she said from the podium: the software she describes is LabNotes, which she co-founded and is piloting in her own course. She was not selling anything, and this post is not either. Her argument stands on its own, and the analogy at the center of it is one we would not have arrived at ourselves.

What she saw

She was cautiously hopeful when ChatGPT first reached her students, and for a specific reason. It does not just produce a final answer. It can show the steps. And in chemistry, she said, the path is the point. A student who has the number but cannot say how they got there has not learned the chemistry.

Then she watched what students did with it. At one point she watched a student paste a homework problem into ChatGPT and scroll straight to the bottom. The steps were on the screen. They did not read them.

Her reading of that moment was careful:

What bothered me was not just the possibility of cheating. The tool had given the student correct information. It just had not asked anything of them. The student either knowingly or inadvertently offloaded their learning. They walked away with an answer and no understanding, and I could not tell from the grade that anything had gone wrong.

She connected it to a line from the encyclical:

The speed and ease with which answers or summaries can be obtained risk extinguishing the desire to ask questions, which is a process that bears fruit only over time.

And then she was precise about what worried her. Not that students were lazy. Not that technology is bad. That we were handing students tools which made asking unnecessary, and then were surprised when they stopped asking.

The instrument

This is where the talk turned, and it turned on something only a working scientist would have reached for.

In science, I use powerful tools every day. I could not do much of my research without instruments that measure things faster, more precisely, and more sensitively than I ever could myself.

But the instrument does not decide what question I should ask. It does not decide whether the method is appropriate. It does not look at a result and wonder, "Does that actually make sense?"

That is still my job as the scientist.

An instrument does not diminish the scientist. It extends her reach and leaves the judgment where it belongs. Nobody thinks an instrument is cheating, and nobody thinks it does the science.

So the question she started asking was whether AI in education could work the same way: extend what a student is capable of doing without taking the thinking away from them.

That framing is more useful than most of what gets written about AI in the classroom, because it stops the argument from being about whether the technology is good or bad. An instrument is neither. What matters is which part of the work it takes over and which part it leaves with the person.

Asked once what she wanted instead, her answer was short:

I wanted something that could sit with a student at eleven at night and refuse to think for them.

Why the final answer is the wrong thing to check

What she described is a tutor built around a refusal. It will not hand over the answer. If a student jumps to the end, it asks how they got there. If the reasoning is incomplete, it asks again. The problem goes back into the student's hands.

Her example was significant figures, which students reliably struggle with. A student can arrive at exactly the right numerical answer with completely wrong reasoning about precision. They can get it right by luck. A homework system that checks only the final answer sees "correct" and moves on. She has watched students carry nearly perfect online homework grades into an exam and then struggle, because the understanding was never there.

She declined to blame the students for that:

I do not think that is a failure of the student. We built systems that could not distinguish understanding from a correct answer.

What happens when a student is wrong

The second line she took from the encyclical is the one that cuts toward her, and she said so:

For an algorithm, an error is a flaw to be corrected; for a person, however, an error can be a catalyst for profound change.

She has helped build an algorithm that evaluates student work. She named that rather than hoping nobody would.

Her answer was about where an error goes. Not corrected for the student, but returned to them as a question, so they stay in the work of figuring it out. And surfaced to her, so she can see it.

She wrote the reasoning steps that count as understanding before the software ever saw a student's answer. She reads the conversations. She can see where students are struggling, and she can overturn the system when she thinks it got something wrong.

Then this:

My old gradebook would never have shown me what was happening. I would have seen the scores. I would not have seen the asking. And the asking was the thing I actually cared about.

The test she applies

She tied this to Carroll's mission and its commitment to an education rooted in freedom of inquiry. Inquiry means asking questions. It means struggling with something long enough to understand it, and being willing to be wrong and then work out why.

So the test she now applies to any tool that enters her classroom is one question.

Does this make my students ask more questions, or fewer?

She read the encyclical's line about education as a long journey requiring patience and engagement with reality beyond appearances, and translated it into her own terms. In class, the appearance is the number typed into the box. The reality is whether the student understands what that number means and can stand behind the reasoning that produced it.

A student who works a problem, gets it wrong, tries again, and eventually figures it out has gained something past the correct answer. They have learned that they can do something difficult.

One of her students had told her, unprompted, that what they appreciated was being told they were close but not quite there, and then asked another question rather than simply handed the correction. She had thought about that a lot:

Being told you are close and being asked to try again is also being treated like someone who is capable of getting there.

The boundary

She was equally clear about what she does not want this to become. Not a replacement for office hours, the classroom, or the relationships at the center of teaching. It can be there at eleven at night when she is not, and that is useful. But:

It cannot see a student's face. It does not know that the student who is struggling with dimensional analysis today was starting to understand it yesterday. And it cannot recognize when the real problem has nothing to do with chemistry.

The encyclical, she noted, says schools should offer what the digital world by itself cannot: shared time for learning, and trustworthy relationships. She reads that as a boundary and intends to keep it.

What she did not claim

She was direct about the size of her evidence. About forty students. A few weeks in a real course. An IRB application still pending, so no research data showing improved learning. Observations, student conversations, and early experiences she finds encouraging, and she said as a scientist she needed to make that distinction out loud.

We built something based on an idea about how students learn. Now we have to test that idea, observe what actually happens, and be willing to revise it. We already have. And I expect we will again.

That, she said, is familiar territory for a scientist and should be familiar territory for a teacher. Test, observe, revise.

But she did not leave it there. Science can tell you whether something works. It cannot tell you what you were trying to accomplish in the first place.

What she wants is not students who are more efficient at producing correct answers. It is students who can use powerful tools without surrendering their own judgment, who are willing to sit with a hard problem, who ask for help when they need it, who can recognize being wrong as part of learning rather than evidence they are incapable of it. And who leave her classroom knowing the understanding they gained is actually theirs.

Which brought her to the close:

I do not think the question for Carroll is whether we embrace AI or reject it. I think the harder and more important question is what we want AI to help us do while remaining clear about what it should never do for us.

For me, in one chemistry classroom, that starts with something very simple: the student should still be the one doing the thinking.

The questions

The audience pressed on regulation, on safety when students are working with a model unsupervised, and on how to think about AI in scientific work specifically, where the gap between a plausible answer and a correct one matters enormously. None of those were settled in an evening, and none of them should have been.

What the chemistry seat on the panel offered was narrower and, we think, more portable: an instrument extends what you can do. It does not decide what is worth asking. Someone has to write down what understanding looks like in a given discipline before any of this works at all, and that someone is the teacher.