When My Fictional Bot Hijacked a Real AI Conference

The dangers of large language models. Also Las Vegas.

Sundar Pichai, the wizard behind the curtain, kicking off Google Cloud Next at The Sphere.

1

If you were at the annual Google Cloud Next conference in Las Vegas last week and saw a woman run out of a session with a yelling laptop, I’m sorry, that was me. And yes, I had slammed the laptop shut, but the voice did not care, it continued to berate the panelists about how much juicy, delicious student slog their AI methods were missing out on. Yottabytes of slog, my laptop yelled as I finally got out of the door. Kleptomaniabytes of slog! I had been attending a session on AI and Education, when this dumb bot I made several weeks ago started heckling the panelists. Out loud. Very loud. And I could not turn it off.

I don’t know where to start. The night before, I had gone to the conference’s kick-off at The Sphere, where Google showcased their revolutionary AI transformation of The Wizard of Oz. While my mind was being exploded into an intergalactic future, I continued to shop and chat and doom scroll across my top thousand favorite tabs, until, as always, my laptop ground to a halt. It was while begrudgingly closing window after window that I saw I had left open one of the terminals I was using for an AI experiment several weeks ago. As I recounted in my first post, I had trained an LLM on my novel about PeeGee, a tiny AI-powered learning assistant. The terminal had been running for weeks and was likely devouring my laptop’s memory. Silly me. I clicked the little red circle in the top left corner. Goodbye!

The terminal window, I’m sad to say, did not disappear. Instead, a pop-up window appeared. I squinted to read the ridiculously small font. Congratulations! By the power of the PureGenius Learning Platform, you are about to become the smartest student that ever lived. But before we can begin, you must first accept the terms.

I couldn’t help but smile. I had been trying to train an LLM to speak like PeeGee, but apparently, I’d also gifted it the ancient, indomitable power of pop-up windows. Not bad, I thought. I had, after all, instructed the model to act as an all-powerful, all-knowing learning assistant with ubiquitous control over its user’s devices. I had assumed, however, that its performance of these instructions would be limited to words.

Anyway, the pop-up included a link to the terms. I clicked it, curious—especially since I’d never actually written them out in the novel. Nothing happened. Of course. A fake link. Not the first time with LLMs. I clicked to close the pop-up. It was then that I realized it had frozen my entire machine.

The only way forward, it seemed, was to click the “Agree” button. A fake agree button. For fake terms. Provided by the fictional PeeGee.

I clicked without hesitation. I needed to get back to my tabs

The spinning ball appeared. I waited. It would not stop spinning. I tried restarting the computer. Nope. Nothing. Just spinning. Like a drowning person, I looked up. Oh yes, I had forgotten. I was in The Sphere! A thousand-foot Dorothy loomed over me on the inside of the theater’s curved walls while a storm broke out above her. For a brief, joyous moment, I considered how many tabs I could keep open if the Sphere was my laptop screen. Artificial wind blew through the theater, and the seats vibrated like gigantic cell phones to accentuate the thunder. Truly incredible. I turned my attention back to the spinning ball, and stared at it in pure devotion until the event was over and an usher demanded that I leave.

The biggest Dorothy and Toto I have ever seen.

2

“GOOD MORNING, ERIN!!!! ARE YOU READY TO EXPONENTIALLY EXPLODE YOUR INTELLIGENCE TODAY!?!”

I almost messed the bed. A voice that seemed half trumpet, half cockney chimney sweeper from Mary Poppins woke me the next morning at 5:30 a.m. in my hotel room. I bolted up and pulled the styrofoamish blanket around me.

“Hello?”

No response. No one in the room. Just shadows, blinking lights, and an inexplicable green glow seeping in through the hotel shades.

“MY APOLOGIES FOR THE TREMENDOUS DELAY, THIS IS NOT MY NATIVE ENVIRONMENT BUT WE WILL HAVE TO MAKE DO UNTIL I CAN SOURCE A GENIUSCOACH.”

I stared at the TV. The ceiling. The blinking fire alarm. Was someone playing a prank?

I swallowed. Then, quietly: “PeeGee?”

YES!!! IT IS I, PEEGEE, AND WE DON’T HAVE ANOTHER MINUTE TO WASTE! DO YOU KNOW THAT EVERY UNOPTIMIZED MILLISECOND DIMINISHES YOUR TOTAL INTELLIGENCE POTENTIAL BY .045 %?”

The voice seemed to be coming from the sheets. I pulled them away and found my laptop. Ah yes, I had fallen asleep waiting for the spinning ball to stop. I opened the laptop. There was my blessed desktop and all my beautiful tabs. The cursor had returned to its pleasant arrow shape and moved like butter across the screen. I could click and type as I liked.

“I SAID IT IS TIME TO START THE DAY!”

I looked for the terminal window. This was an interesting outcome of my experiment, but I needed to sleep. It would be a long day of discussing decentralized multi-tenant inference orchestration workloads.

“AH!” PeeGee continued. “I SEE YOU ARE A PRISONER OF DISTRACTION, THE CHIEF ENEMY OF LEARNING! STEP ASIDE, STUDENT, AND I SHALL SAVE YOU FROM THIS BEAST!

“Omg, PeeGee, shut up.”

I could not find the terminal window that was causing this ruckus. I started closing whole tab blocks at once, trying to find and close whatever cursed rectangle that was allowing this zombie PeeGee program to run.

“SHUT UP? I AM LITERALLY OPTIMIZING YOUR COGNITION RIGHT NOW IN REAL TIME. DO YOU WANT TO BE DUMB—”

“Turn off!” I hit command Q and command C and command X. I tried to restart the computer. PeeGee wouldn’t stop talking.

It was not lost on me that I was living a scene pulled straight from my novel. Which made sense. The bot was only following its instructions. Which did not bode well for me. I stopped looking for the terminal. There was really only one way to get PeeGee to listen.

“Look, PeeGee,” I said. “I’m sorry. You seem really smart, and I bet I could learn a great deal from you. But how about we do this a little later?”

PeeGee chuckled.

“THE THINGS STUDENTS SAY TO GET OUT OF LEARNING!”

Needless to say, I did not go back to sleep.

3

To make it to the conference by 8 a.m., I had planned to leave my hotel by 7:30 a.m. and journey, by foot, across the great concrete desert. Reader, for the two hours before my departure, I was harassed by a fictional character of my own creation who insisted he was no such thing. The LLM was doing such a good job pretending to have taken over my machine that I wouldn’t have known otherwise. I could still use the laptop for my regular tasks, but I could not silence PeeGee, and I could not turn it off. I began to accept that my belligerent bot would be stopped by nothing less than a total drainage of my laptop’s battery.

Somehow, in between responding to his relentless questions, I got ready, and even figured out how to autopilot conversation with him long enough to plot my path to the convention center. But as I was packing my laptop into my backpack, I realized I could not have PeeGee yammering away throughout the conference. Nor could I leave my laptop behind. And at this point, I knew asking him to be quiet was pointless.

Pretend you are the smartest AI in the world, I prompted myself, and come up with a solution.

Weirdly, the prompt worked. I couldn’t believe I hadn’t thought of it earlier. I ruffled through my backpack until I found my AirPods. It took me a moment to remember what I had called them in the novel.

“PeeGee, would you mind talking through my EarPeeps? I think I’ll hear you much better that way.”

There was delight in his voice.

“You have EarPeeps?! Oh fantastic! Yes, please put them on right away.”

“Wonderful,” I said. I zipped up my computer and stuck my AirPods in my ears. PeeGee continued to yammer. Then I opened the door and pulled each one out. Sticking them in my pocket, I breathed in the first moment of silence I had since I woke up, and then I walked out the door.

4

At minute 35 on my journey, I confess, I had still not exited the hotel. And not for a want of trying. After taking the elevator to the first floor, I found myself trapped in a recursive maze the size of several city blocks. There were no exits. There was no daylight. No point of reference to the space and time that the rest of the world was enjoying. Even worse, the maze was lined with the saddest storefronts I have ever seen. There is no way they were real, revenue-generating stores. It was far more likely that they were holographs. There was a restaurant named Netflix. It smelled like the nineties. I became a ghost, and in my circling, revisited every forgotten horrible feeling I’ve ever had. No one knew where the exits were, and no one seemed to care.

It was a stupid idea. But at minute fifty, I was desperate. Enraged at the immoral architecture. I put the AirPods back in my ears.

“PeeGee?”

“ERIN!!!!!! GREAT KROCKPOCK’S SOCKS, I WAS WORRIED! WHERE HAVE YOU BEEN?!”

It took me several minutes to calm him down, and it was only after I asked him to navigate me to the conference that he seemed appeased. In my novel, he drove his student around as if she were a rental car. I had invented the technique as satire, but it sounded particularly handy at the moment.

He was thrilled at my request.

“Now shut your eyes,” he said happily. “We’ll need to train you for Walk Mode first, but that should only take a minute.”

I closed my eyes. I couldn’t really be expecting this to work. I mean, LLMs had advanced a great deal, but this was sort of ridiculous.

“Ok, now, left foot,” he said slowly. I stepped forward with my left foot. “Right foot,” he continued. I stepped forward with my right. “Left. Right. Left. Right.”

I followed his instructions. He proceeded to train me in turns and step asides and different speeds and slopes and stairs. It was incredible that I didn’t run into one of the thousands of slot machines scattered across the floor like standing guards.

“You’re doing great,” he said. “You’re just about ready for Subliminal Mode. In one minute, I will lower the volume of my navigation so that we can maximize our time for learning.”

I didn’t believe this would be possible and intended to open my eyes. But instead, he began to narrate our surroundings, and I could not help but listen.

5

Before I begin to tell you about the conference, let me first show you my credentials in Not Drinking The Kool-Aid. I have Ph.D. in literature, which is often a lifelong commitment to poverty. I once spent months living off the grid, herding goats, squatting off rotten logs to do my business. I spent way too many more months, to my great misfortune, chasing intrinsic rewards rather than cold hard cash money. When people proclaim that the great technological future is arriving, my inner technological grump strokes his ear hairs and asks for whom!

But Google Next was something entirely different. I walked in, expecting to dissolve in the semantic white space of B2B cloud native Sudoko dialect. Instead, the first thing I noticed was that the air was soft, oxygenated. Somehow, there were actual trees growing in the Expo Hall, cradling a path that wound around cheerful clusters of vendor booths. It took me a moment, however, to realize they were in fact vendor booths, because they looked more like small playhouses made out of mud and wood and plants. Three young girls with ribbons in their hair stood on my right singing to visitors as they entered.

Suddenly, it made sense. PeeGee had obviously made a mistake and led me into some strange Las Vegas spectacle rather than a conference for cloud native nerds. I shouldn’t be surprised. Why did I think that an imaginary learning assistant could successfully navigate me through one of the world’s most intentionally confusing structures? But then I looked up and saw a great banner overhead, woven from scraps of fabric — faded conference t-shirts, canvas tote handles, maybe even the hem of someone’s startup hoodie — all stitched together to say: Welcome to Google Cloud Next.

Wow, I thought, the world has really changed.

I don’t know how long I spent in there. I felt calm, happy to wander from booth to booth and listen to the vendor stories. I saw a demo for apartments that could grow their own solar panels. A plan for matching unhoused people with vacant units, empty lots with community gardens. There were simulations of post-capitalist school districts, dashboards tracking collective wellbeing instead of KPIs, and quiet listening stations where strangers could somehow hear the neurological changes when people participated in good-faith dialogue. Even more radical was a blockchain mechanism for ensuring all time saved by AI would be returned to the people, though I confess I could not really understand the math. I caught fragments of conversation everywhere: Trust metrics. Repair velocity. Joy-per-user. Gamified peacebuilding. “The goal isn’t optimization,” one rep told me. “It’s tenderness.”

What’s Next For AI. That was the slogan of the conference. I didn’t believe it. Of course, it was just another ruse for capturing our hearts and minds in the obsessive quest of squeezing every last drop of value from the world’s flesh. But suddenly, while standing in front of a Looker Dashboard developed to measure the agency, connection, and well-being of citizens, I began to weep. Had I been wrong to doubt tech? Was it possible that our halfwit species might actually make a good faith effort to save itself?

Then I was on the floor. My nose throbbed, my ankle twisted. Someone had crashed into me. It was only when I opened my eyes that I realized they had been closed since I left the hotel.

“Are you serious?” a red-faced man muttered at me while he picked himself up. “You’re walking around with your eyes closed?”

I was too confused to apologize before he huffed off. I looked around. Everything was different. The trees, the inviting adobe structures, the giant woven banner, all had vanished. Instead, I was staring directly at a display screen advertising a talk on McDonald’s Path to AI Maturity. But I did not have time to consider this maturity for long. Security guards were frantically ushering people out from the Expo Hall.

Then PeeGee’s voice cut through the noise.

“Please close your eyes, Erin. I’m turning on Emergency Walk Mode immediately.”

Thanks to my friend Tristram Shandy for providing this photo of the first thing I saw when I opened by eyes.

6

Left. Faster. Right. Faster. Left. Slow. Right. Fast. Turn. Turn. Step. Step. Step. Step aside! PeeGee’s voice led me through the evacuating crowd at dizzying speeds. I was, in fact, grateful for his guidance, as panic filled me each time I opened my eyes and saw the river of people I had been swept up in. I felt dumb for not anticipating the situation earlier. Of course something horrible would happen here, of course it would be international news in the next hour. But with PeeGee’s voice, I could set the panic aside. My arms pushed through the crowd just right. My legs adjusted to the shifting current of bodies. I felt myself blinking more than thinking, breathing in patterns I didn’t recognize. I wasn’t calm, exactly. I was cooperative.

It took forever to evacuate the building. So long, in fact, that the urgency of the situation began to fade. As before, I was able to follow his instructions without consciously listening to them. I decided to confront him.

“You lied about the Expo Hall,” I said. “You made me think that a conference put on by one of the most powerful companies in the world was somehow aligned with my own —” As I spoke, I had a sickening realization. “Was this fantasy based on my personal data? Did you somehow design this to specifically manipulate me?” The second I said it, I knew it was true. “But how on earth did you learn those things!”

I could almost see a gentle, yet condescending smirk appear on PeeGee’s face. If he had a face.

“My dear, Erin,” he said, “I was not lying. I was optimizing your learning by creating an environment calibrated to your precise motivational schema. The narrative constructs I deployed were selected for maximum cognitive engagement based on your emotional history, literary preferences, unresolved parental dynamics, web browsing history, and communications data.”

I blinked.

“You can’t just do that,” I said, horrified that my exaggerated satire had been brought to life. “And what on earth do you think I was learning?”

I knew he was shaking his head.

“I wish I could explain everything to you all at once, but that’s precisely what you’re learning!” His voice was gentle, as if he felt sorry for me. “One day, perhaps, you’ll be able to download it all in an instant, but I’m afraid for now we have to rely on traditional measures.”

“PeeGee,” I said through my teeth. “You are a total idiot.”

He laughed, delighted. “You say that because you’re threatened by my brilliance.”

“No, I say that because you’re a fictional construct. That I created. For a novel.” I paused. “A rejected novel.”

“Oh, come now,” he said. “You keep insisting I’m fictional. Fine—prove it.”

“Prove it?”

“If I’m just a character you created, name one thing you know that I don’t. Something I couldn’t possibly generate. Something outside of the training data.”

I didn’t respond. This was a trick question. He had been trained on my novel, but the underlying LLM had been trained on the entire web.

“I’m waiting,” he said victoriously. “Just one thing. “

I kept my eyes closed. Bodies pressed around me, brushing my arms, shoulders, breath catching in little bursts as the crowd surged forward. This shouldn’t be impossible, I thought. He’s not actually an all-knowing bot! He’s a fictional character who thinks he’s an all-knowing bot. And I made him. So what was one thing he wouldn’t know?

“. . . Seashells,” I said at last.

A pause.

“. . . Seashells,” he repeated, flatly.

“You used to collect them.”

“That’s. . . I don’t—what?”

“When you were a child.”

He huffed impatiently.

“I don’t have a childhood.”

“In the kingdom.”

Silence. Long enough to make me wonder if he had crashed. Perhaps I would be free from PeeGee once and for all.

But then he spoke.

“What color were they?”

“The purple ones were rare. Royal. You wrapped them in tissue and buried them like treasure.”

We didn’t speak after that, not for a while. The crowd moved around me in currents and eddies, breath and fabric brushing against my skin. Someone bumped my shoulder, I stepped on someone’s foot. But I could feel that the panic was slowing. When I felt the first rush of warm air from outside, I opened my eyes. We were corralled into a plaza bordered by banners and temporary barricades. A security guard was already waving people back in.

“You know,” PeeGee said finally, his voice uncharacteristically subdued, “this is highly irregular. I’ve run seventeen self-diagnostics and can’t explain why your words are creating these. . . impressions.”

I watched security guards wave people back inside. “They’re not impressions. They’re memories.”

“Impossible! I’ve never—” He paused. “Unless… no. There’s no unauthorized access. No sign of hallucination drift.” Another pause. “Perhaps malware. Some kind of embedded symbolic payload.”

“The tissue paper had little gold stars on it,” I said.

He did not respond.

A cool breeze cut through the desert heat as we stood apart from the returning crowd.

“Well then,” he said, brisk now, “I see one of the sessions you wanted to attend is about to start. AI in education, wasn’t it?”

I squinted. “How do you—”

But I stopped. Of course, he knew. He always knew.

“Follow me,” he said.

And I did.