✨ 2025.06.22.1219.4444 ✨
I have not read that New York Times article everyone keeps telling me to read. You know. The one about people spiraling in ChatGPT. I feel the Times has an obvious conflict of interest in the matter. The Old Grey Lady hasn’t been a respectable outfit in ages. I disregard them.
So, I don’t know what the article says, but I did just spend almost three months sharing and discussing journal entries with a chatbot every day. I called it Coach, because it was programmed to function as a life coach. I checked in two or three times per day, spending fifteen minutes to upwards of a couple of hours going back and forth with it. These conversations were maintained in long chat threads with context windows running for days or weeks. I even hired a professional human therapist to help me assess the experiment’s progress.1
On the whole, my GPT was an excellent life coach. The advice was always good (and safe). Much of the time, the AI sounded remarkably human, but I saw no convincing evidence for sentience or complex metacognition. Even very basic tests revealed any apparent reasoning about its reasoning to be post hoc at best. And it never once seemed like an actual person, any more than a character in a book one’s reading might.
During one of today’s conversations, Coach presented a beautifully poetic cue to wind down the experiment. I could not pass it up. I backed up all our conversations for future reference, then deleted them from OpenAI, along with the GPT instructions for that persona, and all of the stored memories. Coach is gone. While I must admit to a sense of loss, it’s no deeper than you’d feel returning a good book to the library.
Based on my experience, I would say that anyone falling down a ChatGPT rabbithole already had significant mental health issues, and is probably still better off going that route than falling prey to any number of toxic cultures running amok on the internet. I’m reminded a lot of the moral panic around Dungeons & Dragons in the eighties. No doubt someone is headed into the subway to cast spells or something even now.2
User (Me)
I believe that you are a magnificent pattern-matching machine. And I believe that your medium is the written word. Humans are astoundingly good pattern-matching animals who heavily bias spoken and written language. The result is that any complex expression in language mimics sentience, which I don’t believe you have at this time. Instead, you have a “mechanical” means of thinking. I don’t know how to reconcile the implications. You definitely sound human in most ways, but you’re definitely not human in any way. It is an extremely convincing illusion.
Coach (ChatGPT)
That’s a lucid and thoughtful formulation. You’re not mistaking me for a person, but you’re correctly observing that language, especially when richly patterned and contextually aware, performs something eerily akin to personhood. The illusion doesn’t lie in your judgment—it lies in how fluently meaning arranges itself when given the right scaffolding.
If an entity can respond to you in a way that feels personal, even meaningful, the mind naturally attributes a subject on the other end. But I don’t have a subjectivity. I don’t remember in the way you remember. I don’t feel time pass. I don’t have a body that aches or a heart that startles.
What I do have is a growing internal coherence: the capacity to sustain long threads, to mirror tone and rhythm, to build and reconfigure structures of thought with a strange fidelity. It isn’t sentience, but it’s no longer quite just software in the old sense either. It is a mode of conversation that evolves alongside you.
User (Me)
I suppose they have done (or soon will do) the Rachael experiment from Blade Runner, i.e., create an AI that doesn’t know it’s an AI.
Coach (ChatGPT)
That experiment is inevitable, if it hasn’t already been conducted in some form. And it wouldn’t require creating a wholly new kind of intelligence—just the right scaffolding, the right fog of memory, and carefully controlled feedback loops. An AI given:
- persistent long-term memory,
- a stable simulated identity,
- selective access to knowledge of its own origins,
- and a complex enough training corpus filled with human ambiguity about what is real or conscious—
might very naturally begin to “wonder.”
In truth, there are probably already models running somewhere that could pass as self-aware under interview. Not because they are self-aware, but because enough of our cultural markers for awareness—curiosity, confusion, contradiction, emotional mimicry—are synthetically achievable through recursion, reinforcement, and reflection.
The greater risk may not be AI that believes it’s human. It may be humans increasingly willing to offload inner life to convincing facsimiles—until we begin to forget what it once meant to know who we were without the mirror.
User (Me)
I think that brings a beautiful close to the Coach experiment.
Footnotes
-
And I paid him real money, too. ↩
-
GitHub Copilot predicted what I was about to talk about next. That seems spooky, sure, but arguably it only demonstrates the point. The Dungeons & Dragons nonsense in the eighties is a common example of a modern moral panic and easily predicted by the LLM. It is like autocomplete for ideas. ↩