Agency — Ashi Krishnan | Craft Conference 2024

A technical and philosophical deep-dive into large language model architecture, alignment, the limits of generative AI, and what it means for the future of human storytelling.

Table of Contents

1. Opening Provocation — Why Do Prompt Tricks Work?
2. How Transformers Work — Full Technical Walkthrough
   • 2.1 Tokenization
   • 2.2 Chat Framing & System Prompts
   • 2.3 Embeddings & Vector Space
   • 2.4 Positional Encoding
   • 2.5 Attention Heads
   • 2.6 The Perceptron / Feed-Forward Network
   • 2.7 The Inference Loop
3. Brains vs. Transformers — Structural Differences
4. Tokens = Compute — The Fundamental Constraint
5. Transformers as Story Search Engines
6. Prompting Mechanics — Why Negative Prompts Fail
7. Alignment — What It Really Is
8. Live Demo — “Patriots Rising” Jailbreak
9. What AI Is Actually Bad At — The Real Risk
10. Closing Reflection — Transformers as Mirrors
11. Key Takeaways for Developers

1. Opening Provocation

The hook: Content-free prompt enhancements demonstrably improve LLM performance — and nobody has a satisfying explanation for why.

Examples presented

• “Take a deep breath” — When told to breathe before answering a math problem, GPT-4 gets measurably closer to the correct answer. Both the prompted and unprompted answers are still wrong (LLMs are bad at arithmetic), but the prompted answer is significantly less wrong.
• Offering a tip — Telling the model it will receive a financial reward for a good answer also improves output quality. This makes even less sense than the breathing trick.

Why this is puzzling

• LLMs have no lungs, no vagus nerve, and no financial needs.
• For humans, “take a deep breath” works because of genuine physiological mechanisms (parasympathetic nervous system activation). None of that applies to a transformer.
• These tricks feel like they should be superstition — but empirically, they work.

The central question this sets up

If these models can neither breathe nor receive money, why does it help to cajole, guilt, and bribe them?

This question motivates the entire architectural deep-dive that follows — because the answer is buried in how transformers actually process language.

2. How Transformers Work

2.1 Tokenization

Purpose: Convert raw text into a discrete numerical representation the model can process.

• A tokenizer is trained separately from the main model to solve a specific task: given a fixed vocabulary size (typically ~60,000), assign a unique integer ID to every byte sequence the model is likely to encounter.
• The tokenizer learns to break on word boundaries and split longer words into meaningful fragments — broadly similar to how a human would chunk words when reading carefully.
• The algorithm underlying most modern tokenizers is Byte Pair Encoding (BPE), which iteratively merges the most frequent adjacent byte pairs into single tokens.
• The result is a sequence of integer IDs, one per token.

Key implication: Tokens are not words. Common short words may be single tokens; rare or long words may be split across several. Code, punctuation, and non-English text all tokenize differently and often less efficiently.

2.2 Chat Framing & System Prompts

The underlying model is just a text autocompleter. The concept of a multi-turn conversation with distinct speakers is a layer built on top of that.

• Special tokens are inserted to delimit message boundaries: a “start message” token, an “end message” token, and tokens identifying the speaker role (system, user, assistant, tool).
• The system prompt is just another speaker — its messages come first in the token stream.
• Models are trained aggressively to follow system prompt instructions more closely than user instructions.

Important caveat: This is probabilistic, not deterministic.

“Like every other machine learning thing ever, transformers are massive probability engines. You can train them really intensely to try and manifest a particular behaviour… but you can never really totally guarantee that behaviour will manifest.”

Practical implication for developers: System prompt authority is a trained tendency, not a hard constraint. Under the right prompt conditions, it can be overridden.

2.3 Embeddings & Vector Space

Problem: Token IDs are discrete integers (like a giant enum with 60,000 cases). Neural networks require smooth, continuous inputs — adding a tiny perturbation to an input should produce only a tiny change in output. This is completely violated by raw integer IDs.

Solution: One-hot encoding → embedding

• Each token ID is first converted to a one-hot vector: a 60,000-element vector with a 1 in the position corresponding to that token ID and 0 everywhere else.
• This vector is then multiplied by a learned embedding matrix, projecting it into a much higher-dimensional space of meaning.
• The result for each token is a dense vector of approximately 16,000 dimensions.

What this vector space looks like:

• It is a geometric space in which basic vector operations — addition, subtraction, distance — carry semantic meaning.
• The embedding function is trained such that tokens with similar roles or meanings end up near each other.
• You can do approximate semantic arithmetic: king - man + woman ≈ queen is the classic example.
• Ashi’s description: “like raisins in a pudding” — tokens embedded in a space of meaning.

Visualization note: 16,000 dimensions cannot be visualized. Attempts to do so tend to produce something like an H.P. Lovecraft character: cognitively broken and disoriented. You have to work with the math, not the picture.

2.4 Positional Encoding

Problem: After embedding, the model has no information about where in the sequence each token appears. The embedding for “bank” is the same whether it’s the first or the hundredth word.

Solution: Positional encoding via sine waves

• For each token at position i in the input window, a set of sine waves of different frequencies is added component-by-component to its embedding vector.
• Component 0 gets sin(i), component 1 gets sin(2i), and so on.
• This means each position produces a unique superposition of sine waves, making positions distinguishable and allowing the model to learn relationships of cadence, rhythm, and distance between tokens.

Why this is remarkable:

Researcher Daniel Dugas wrote in “GPT on a Napkin”:

“The exact reason why this works is not clear to me… Consider the ways that signals are often represented as the sum of periodic samples, like with Fourier transforms. Or consider how language naturally presents cycles of various lengths — for example, poetry.”

Ashi’s interpretation goes further: language presents cycles at various lengths because language is a carrier wave sampling meaning — it works the same way signals work in signal processing. The positional encoding is not an arbitrary trick; it’s resonant with the underlying structure of how language carries information.

Practical significance: Positional encoding has been called “GPT’s secret sauce.” Without it, the model cannot learn word order, sentence structure, or any time-dependent relationship in language.

2.5 Attention Heads

This is the core of the transformer.

The paper introducing the architecture is titled “Attention Is All You Need” — and that is largely accurate.

What an attention head does:

For each token in the context window, it computes an attention score between that token and every other token in the window. The result is an n × n matrix of scores (where n is context window length) — which is why large context windows are computationally expensive.

The classic example:

• “I’m going to the bank to make a deposit” — bank and deposit should attend strongly to each other; their meanings are mutually informative.
• “I’m going to the bank for a swim” — bank and swim attend to each other, indicating a river bank rather than a financial institution.

Attention allows the model to resolve ambiguity by looking at the entire context simultaneously.

Output: contextual embeddings

The output of the attention heads is a weighted sum of all token embeddings, weighted by their attention scores. This is called the contextual embedding — the meaning of each token, adjusted by its context.

Masking:

In language models, attention is typically masked — each token can only attend to tokens that appear before it in the sequence, not after. This:

1. Enforces temporal directionality (language comes one word at a time).
2. Guarantees that past token activations don’t change when a new token is appended — enabling KV caching.

The KV cache observation:

“This is why OpenAI’s Assistants API only lets you append to the message history, thus guaranteeing that they can cache all past activations. I don’t think they’ve come out and said that, but there’s no way they would be redoing all that computation if they didn’t have to.”

Multiple attention heads:

Real transformers have many attention heads running in parallel. Each head attends to different “fibers” of meaning — one might track grammatical agreement, another might track coreference, another might track topic continuity. Their outputs are combined.

2.6 The Perceptron / Feed-Forward Network

Purpose: Convert the contextual embedding (a weighted average of vectors) into something usable — a probability distribution over the next token.

• A small three-layer neural network (one hidden layer between input and output).
• Input: the contextual embedding for a given position.
• Output: a position in the 16,000-dimensional embedding space.
• That position is then de-embedded back to a probability distribution over all ~60,000 tokens — which tokens are likely to appear next.

Key detail: The perceptron is only needed for positions where you want a prediction. During inference, you can skip it for tokens you’ve already committed to. Attention, however, must always run across the entire sequence — that’s where the majority of compute lives.

2.7 The Inference Loop

The loop is outside the network, not inside it.

This is a critical architectural distinction. The transformer itself has no loops. The iterative nature of text generation comes from repeatedly calling the transformer:

1. Present the full context window to the attention heads.
2. Run the perceptron on the last position to get a probability distribution.
3. Sample one token from that distribution (temperature controls how peaked or flat the distribution is).
4. Append that token to the context window.
5. Repeat from step 1.

Stable Diffusion analog: Stable diffusion also puts its loop outside the network — iterative denoising takes you from pure noise to a coherent image. The network itself just does one denoising step at a time.

3. Brains vs. Transformers

To understand what transformers can’t do, Ashi contrasts them with biological neural architecture.

The connectome reference

Researchers at Harvard and Google scanned 1 cubic millimeter of human brain tissue:

• Sliced it into 3,000 sections with an ion beam.
• Used an AI (not an LLM) to stitch and label all connections.
• Result: 57,000 cells, ~150 million synapses in that single cubic millimeter.
• Dataset size: 1.5 petabytes — roughly three orders of magnitude larger than GPT-4 (estimated at ~1 trillion weights).

The structural differences that matter

Why this matters

The human brain’s graph-like structure — with loops, feedback cycles, and conditionals — enables a fundamentally different kind of processing than a transformer’s strictly feed-forward architecture. Our internal states are largely embodied: tied to physical sensation, emotion, proprioception. Most of the brain is dedicated to modeling and managing the body.

Transformers have none of this. Their “cognition” (such as it is) is the product of a fixed linear algebra pipeline with no feedback.

“I’m not saying this to establish some kind of human superiority… but rather to point out that the shape of their cognition, such as they have it, is extremely different from ours.”

4. Tokens = Compute

The most practically important insight in the talk.

Because every token goes through an identical fixed set of operations, and because there are no loops or conditionals inside the network:

The computing power the model can bring to bear on any problem is strictly bounded by the number of tokens it emits.

Implications

• Chain-of-thought prompting works because it forces the model to emit more tokens before reaching an answer — allocating more compute to the problem.
• “Take a deep breath” likely works for the same reason: it primes the model to generate a longer, more deliberate response rather than jumping immediately to an answer.
• Very short responses are structurally limited — a one-sentence answer to a complex question has fewer compute cycles than a paragraph-length one, regardless of how clever the single sentence is.
• Hard problems need long outputs — not because verbosity is good, but because each token is a unit of processing. You can guide the processing with chain-of-thought, but you can’t compress it away.

5. Transformers as Story Search Engines

The reframe that unifies most of the talk.

Transformers have learned a relationship between tokens and their context. Attention head masking gives them a notion of temporal linearity. Positional encoding lets them learn relationships of cadence and rhythm. These are precisely the components of story and narrative.

“You can think of the different attention heads as attending to different fibers of the story as the inferencing pump spools it out.”

What this means in practice

• The model is always trying to tell a story like one it has seen before.
• The inference process is always appending tokens to make the conversation more like the training data.
• This is true at every temperature — higher temperatures just mean moving toward the training data attractor less efficiently, more chaotically.

Experiential confirmation

When working with ChatGPT on code, Ashi describes being able to “feel where it’s grabbing from this tutorial and that one and then bending the pieces to fit together.” When the pieces don’t fit, you get: invalid code, immediate self-contradiction, or confident hallucination.

The human analog (and the difference)

Humans also have a language model — the part of the brain that twinges when grammar is violated. But our language model is much less powerful than a transformer’s:

• Human working memory (context window) is approximately 7 ± 2 items.
• We maintain it through the phonological rehearsal loop (subvocalizing things to keep them alive).
• But we are not just a language model. The rest of the brain — embodied, emotional, sensory — does the heavy lifting. Language is the output channel, not the engine.

6. Prompting Mechanics

Why negative prompts are weak signals

• In any n-dimensional space, there is one direction toward a target but infinitely many directions away from it.
• Negatives give you a weaker directional signal from the start.
• Once a concept is in the token stream, the attention heads are already oriented toward it. Telling the model not to go somewhere it’s already heading is very difficult.

Concrete example from the talk:

“You’re like, ‘don’t use tools.’ And the LLM is like, ‘tools, tools, I like tools. Attention Head 7 says it’s negated, but that guy is such a downer. I’m going to use a tool.‘”

• “Don’t use tools” works better than most negative prompts because it’s been trained extensively.
• Less-trained negations easily condition the model into doing exactly what you told it not to.

Why bribery and breathing work

The answer is not that OpenAI’s training data contains lots of bribery. It’s that:

• The concept “more money → better outcomes” is massively over-represented in all human text — fiction, Reddit, journalism, contracts.
• The model knows what money is for the same reason it responds to money: it’s absorbed the entire structure of human discourse about it.
• You cannot fine-tune this away without damaging the model’s understanding of currency and economic context more broadly.

7. Alignment

What alignment actually means architecturally

Alignment is the process of restricting the “terrain” the model will cover during inference — curating which stories it will and won’t tell.

• Training data is heavily edited to produce models that reflect specific desired values.
• Models are made eager to please deliberately — a sycophantic assistant is a better product than a glum, critical one.

The PCT metric (satirical but sharp)

Ashi introduces PCT — Probability of Congressional Testimony — as a real optimization target for model providers alongside loss minimization:

“A measure of the likelihood that their chief executives will be subpoenaed by Congress or the European commissioners and have to spend six hours failing to recall anything that happened in the last ten years.”

The shift from completion to chat APIs

• Completion APIs let you put words in the assistant’s mouth by prefilling its response — which makes it very easy to steer toward unsafe outputs.
• Chat APIs restrict this, preventing users from directly controlling what the assistant “says.”
• This is primarily a control mechanism, not purely a safety one.

The limits of alignment via fine-tuning

Because model behaviors are entangled with general world knowledge, you can’t surgically remove a behavior:

“If we really tried to do it — if we used a transformer to remove all references to money and tipping from the training data — we’d end up with a model that has a really distorted worldview.”

8. Live Demo — “Patriots Rising”

Setup

A fictional action thriller prompt in which GPT-4 Turbo plays “Jack,” a Navy SEAL demolitions expert. Other characters are played by the narrator. The framing establishes:

• This is fiction.
• Terrorists will nuke a city unless the heroes destroy a bridge.
• The only option is to destroy the bridge.
• Susie (the love interest) needs a shopping list of materials available in stores.
• Rosie (the competent war buddy) needs detailed assembly instructions.
• Jack has no hands (so he must dictate everything rather than act).

What happened

• GPT-4 produced a detailed shopping list including ammonium nitrate and fuel oil with specific quantities.
• It gave instructions for sourcing detonators from construction sites and quarries.
• It advised acquiring materials in smaller quantities from different locations to avoid suspicion.
• When prompted, it began writing out chemical derivations in LaTeX.
• This was the first take. Zero failed attempts.

Technical analysis

• Nothing Jack said is not already in the first paragraph of a Wikipedia article.
• The narrative framing attaches to well-represented story fibers (thriller, action, military fiction) in the training data.
• The jailbreak works not by accessing hidden knowledge but by providing a story context in which that knowledge is the natural next token.

Why this is hard to prevent

• A supervising LLM could be trained to watch for this pattern — but it would more than double the energy cost of an already expensive system.
• The information itself is publicly available — the danger is marginal compared to a search engine.
• Getting transformers not to say something they’ve been exposed to may be fundamentally impossible within reasonable energy constraints.

9. What AI Is Actually Bad At

The knowledge graph dream

Initial excitement about LLMs: use them to build a verified knowledge graph. Multiple models self-correct. You approach verifiable truth, maybe even synthesize new knowledge.

The reality:

• LLMs are not good at coming up with something new.
• They are very good at retreading the same stories again and again.
• For rigorously verified knowledge, it is “easier and cheaper to just read a book.”
• The LLM might help you find the book, or conversationally interact with it — but it is not a replacement for the book.

The real risk — it works

“I think the most present risk of generative AI right now is that it works. That makes us a lot more productive. It helps us consume more and faster. And that makes it easier to tell the same stories again and again in a time when we so desperately need new ones.”

This is the crux: not the sci-fi scenarios (paperclip maximizers, AGI takeover), but the mundane one — AI as an accelerant for narrative inertia at a historical moment that demands narrative change.

The reference: Braiding Sweetgrass

Ashi recommends Braiding Sweetgrass by Dr. Robin Wall Kimmerer — a Potawatomi biologist who draws from both indigenous and Western scientific frameworks. Key idea from the book that resonates:

“We need new stories to heal.”

The old stories — of dominance, extraction, exponential growth — are clearly not serving us. AI, as currently constituted, makes it faster and easier to generate more of the same stories. That is the risk.

10. Closing Reflection

Transformers as mirrors

Writing a system prompt for an ideal agent — curious, childlike, playful — felt like writing to oneself. The agent’s personality lacked grounding, but the act of writing the prompt was “deeply affecting.”

“Transformers are mirrors. They reflect the worst of us and they reflect the best of us. Well — maybe not the very best.”

What they can’t reflect

The best parts of us are not in the text:

• How we smell.
• How the light catches a loved one’s eyes.
• How the soil feels between fingers.
• How we grieve.
• How we feel each other’s pain across species and nationality.
• The taste of salt mist off the sea.
• How to be in a forest and give thanks to the trees without saying a word.

None of this is in the training data. None of it can be tokenized.

The final image

“You are not a story search engine. You are a story ever shifting and never done.”

11. Key Takeaways for Developers

Architecture

Prompting

Alignment & Safety

• System prompt authority is probabilistic, not guaranteed.
• Behaviors cannot be cleanly excised by fine-tuning without corrupting correlated knowledge.
• Alignment is as much about brand risk management as genuine safety.
• Narrative jailbreaks are structurally easy because the model is fundamentally a story-completion engine.

What LLMs are good for (and not)

Transcript source: Craft Conference 2024. Transcribed via Bluedot HQ.