The Past as a Compass for Systems Engineers in the Age of AI – John Michael Willis (Talk Outline)

John Michael Willis — 40+ years in IT; considered one of the founders of DevOps; co-author of The DevOps Handbook (with Gene Kim, Patrick Debois, Jez Humble), Beyond the Phoenix Project, Investments Unlimited (DevOps automated governance); ex-Docker evangelist (sold SocketPlane to Docker, Statacuity/Stratacuis to Dell), ex-Chef (“the old guy at Chef”), Red Hat exec; a self-described “Deming freak.” He spent ~2 years writing a book on the history of AI (Rebels of Reason / The Operational History of Generative AI). Thesis: “not understanding the history of AI is literally our bottleneck.” (The title itself was generated by AI — “when you get stuck on a title, give it your presentation and say ‘give me a better title.’“)

1. Framing & the DevOps Roast

• Three arguments he’s trying to evolve: a historical argument (the book’s core, ~100 years — first neural network on paper in 1943), an epistemological argument (how he “got roped into” it), and an ethical argument — but he deliberately avoids AGI/ethics debates: ”every breath you take discussing AGI is wasted time” (he cut a “stop killer robots” chapter).
• Self-roast (he was roasted by DevOpsDays 3 weeks earlier): the early Docker photos, the “save the world” DVO pose the kids memed (“pro tip to young speakers: never make this pose”). Callback to Andrew Clay Shafer (“read everything he’s done” — the talk “There Is No Talent Shortage”): a 15-year bet Willis lost meant jumping out of an airplane together. Shafer’s line — amid all the terminology fights (agile / lean / SAFe / Scrum) — ”maybe the real DevOps are the friends we make along the way.”

2. History Matters — Non-Linear & Human

• AI history isn’t linear; there are multiple types: symbolic (expert systems ≈ deductive logic) vs. sub-symbolic (neural networks ≈ inductive). Writing the book, he broke it into “senses” to avoid timeline whiplash: “AI could count, AI can think, AI can learn.”
• Diverse, forgotten characters absent from “those 25 books”:
  • Grace Hopper — first to treat language as an abstraction for bits and bytes.
  • Dr. Fei-Fei Li — “you don’t have ChatGPT in 2022 without Fei-Fei.”
  • Walter Pitts (co-author of the 1943 paper describing mathematically how the brain might work — Boolean logic they called “calculus”) — a real Good Will Hunting: homeless and uneducated at 13, spoke three languages, hid in the Detroit public library from a gang, picked up Russell & Whitehead’s Principia Mathematica (10 volumes), mailed Bertrand Russell corrections; Russell invited him to Cambridge not realizing he was a homeless 13-year-old. He ended up with Norbert Weiner (founder of cybernetics — “AI should have been called cybernetics, but John McCarthy called it something really stupid like AI”), working all night with bohemian scientists (jazz, pot, likely psychedelics) while employed as a janitor.
• Knowing the route (Deep Blue → AlphaGo → today, symbolic vs. sub-symbolic) is what lets you make sense of the daily shifts. Even the “T” in GPT — transformers — may be superseded (its own creators expect “another horizon” beyond transformers).

3. Benchmarks & the End of Denial

3.1 Hallucination is a bastardized term

• Humans hallucinate too — it’s how we converse via abstractions (“if you know physics and I don’t, you give me hallucinations of abstractions, not gory details”).
• LabCorp example: a year building an AI remediation system to replace the 24-hour help desk (doctors/nurses calling for lab/blood-test results — critical). At go-time the board said ”people may die.” He’d already proven the humans were worse: he ran 5 years of call transcripts through evaluations (quantitative + qualitative correctness metrics that calibrate hallucination) and showed the new system was 10% better than 5 years of human answers.

3.2 The benchmarks (all ~3 weeks old at talk time)

• GPQA (reasoning): leader 60% in June 2024 → ~84% (probably higher now).
• HumanEval (coding): 70% in June → top five within a point of ~99%. “To argue humans code better at scale than AI is nonsense.”
• SWE-bench (AI resolving a well-defined GitHub issue): 2% (March 2024) → ~60% in under a year (after Devin launched in April; cf. Patrick Debois’ talk).
• The point isn’t replacing coders (agrees with Simon Wardley — Jevons paradox) — it’s consistency: Capital One once had 20,000 Java developers — “you think they all consistently produce at 99%? If so, leave the room.” Even if you dispute the benchmark, you can’t dispute AI is consistent, “and that’s something we don’t get from humans.” “Dear CIO: stop ignoring this.”

4. The Technical-Debt Tsunami & Shadow AI

• Shadow AI (he coined the term ~2 years ago) repeats Shadow IT / cloud: every CIO who said “there will never be cloud here” now runs 30–40%+ in cloud. It took 10 years to clean up that mess.
• BYO AI (LinkedIn study): in early cloud, a 100k-person org had maybe <5,000 real users (API-only Amazon). With AI, a 100k-org will have ~70,000 users — ”two and a half orders of magnitude more complexity.”
• Prototype ≠ production — he won’t say “the V-word” (vibe coding), but ”don’t conflate prototyping with production” (Eric Ries’s MVP, 15 years ago). AI is, “being generous, 80% network/compute/storage” — cloud-native (Kubernetes, Kafka, Redis) on top, and only the last ~10% is “cool Python and node stuff.” ”If you were terrible at that before GenAI, Bob’s your uncle” — now with far more code and new probabilistic adversarial attack vectors.

4.1 The Air Canada disaster

• They built a RAG and scraped everything (“first clue how things go wrong”) — including both a system-of-record refund policy and an FAQ. 99/100 times inference would hit the system of record; one time, a young man whose mother died asked about a bereavement refund, got the other source, was told yes, bought the ticket — then was denied (“that was the AI; that’s not what we do”).
• Result: weeks on the Wall Street Journal front page at the worst possible moment; by napkin math, ~half a billion in market cap lost on brand reputation — over an $800 payout. ”Don’t be Air Canada. People tell you to throw everything into an LLM. No.”

4.2 Jailbreaks & attack vectors

• Read the OWASP LLM Top 10 (prompt injection, sensitive data, supply-chain, data/model poisoning, misinformation, leakage, vector-embedding weaknesses).
• Wiz white-hat demo: built an embedding model poisoned with Python pickle (escapes to the OS), put it on Hugging Face, ran a query — broke out of inference to the OS on a plain, non-hardened vanilla Kubernetes install in nanoseconds (“the kids who built Hugging Face probably never heard the word DevOps”), landing on a shared customer AWS host.
• Zero-byte codes in PDFs (Struts-2-style URL escape) can execute scripts just by loading a PDF into a vector and running an inference question against it.
• Even managed infra is shaky: ”Bedrock is terrible — I’d go Vertex” — because Bedrock is network/compute/storage with lots of cloud-native (Kubernetes, Kafka, Redis) inside that “doesn’t go away.”
• For regulated orgs forced by leadership to adopt AI: read the paper that points you to MITRE and NIST — “it’ll scare the hell out of you,” but at least you can mitigate the risks.

4.3 The Chief AI Officer problem

• CEOs hire Chief AI Officers who hire brilliant Stanford/CMU kids — great, but they don’t know how to protect the brand, don’t understand infrastructure/operations, and are often told by the CEO to ”stay away from the CISO, they’ll slow you down” — exactly the cloud mistake. ”When you fail an audit, they don’t go to the Chief AI Officer — they go to the CISO and the CIO.” (This is the plot of his Phoenix-Project-style novella.)

5. The NORMAL Stack

• Like the LAMP stack aligned everyone on web services in the early web, CIOs need a shared mental model. A friend rejected his original acronym “LAURA”; hence NORMAL (N = “N ways” — multiple choices at each level):
• Observability = Evaluations. The ML world’s ”evaluations” got conflated with vendor “observability” (Honeycomb, Datadog). Evaluations are software using computational/quantitative/qualitative analysis to judge correctness, bias, and toxicity — ”it’s geometry, not vibes” (a matrix of vectors finding the closest distance to the next word/sentence/paragraph). Use it to calibrate acceptable risk: when the CEO mandates AI despite hallucination, ask the policy org what risk is acceptable (e.g. a 5% incorrect-answer tolerance), then prove after every pipeline change you still meet it.
• RAG — “not dead,” just another tool to reduce hallucination/add value: vectorize your own corpus, feed a frontier model (GPT-4.1 / o3), retrieve the closest paragraphs, then still calibrate. Variants: CAG (cache-augmented generation), GraphRAG. Example: an HR hire/fire system of record — don’t ask ChatGPT directly (“give me steps to fire Steve”); vectorize the corpus and query that. Govern it: don’t end up with ”10,000 unmanaged Jupyter notebooks” and thousands of unmanaged models — every model belongs in an artifactory, in a pipeline, SHA-proved so you know which inference produced an answer. (Also: AIBOMs / SBOMs — for open-source models the “holy grail” is the weights + training data, not the code, which is “useless.“)
• Agentic protocols — “if you give an AI talk and don’t mention MCP, you failed” — but MCP isn’t the only one: Google’s A2A (agent-to-agent mesh, “a little better than MCP”), plus IBM’s approach.
• Managed Language models — behind a gateway (one client wires the gateway to Okta): you can’t reach any frontier-model SaaS without authenticating, which routes you to corporate-licensed, approved models.

6. The Philosophy of AI — Alien Cognition

• His friend John Allspaw (a DevOps frontier figure, working with cognitive/safety scientist Dr. David Woods) challenged him not to be a ChatGPT rah-rah and to take an epistemological view. “To study science you must understand the philosophy of science; to study AI you must understand the philosophy of AI — and we’re not doing that,” instead stealing time by arguing whether AI replaces programmers/humans (the Terminator / 2001 sci-fi meme).
• Common thread of agile / lean / Toyota Production System / Deming: scientific method, epistemology, evidence-based, hypothesis-driven. We spend too much time asserting “we can’t do this” or “it will be this” without that rigor.
• The book’s subtheme: a 100-year human quest to build a thinking machine (Turing → McCulloch & Pitts → Rosenblatt → McCarthy → Herbert Simon) — ”but it never occurred to us we might build a machine that doesn’t think like us.” Between the social (human) and technical (non-human) there’s now a middle piece — “alien cognition.” “You can’t argue it doesn’t have a cognition; it just isn’t a human cognition.”
• Arrival analogy: the linguist doesn’t ask the heptapods “can you ride a bike / do math as well as us?” — those are the wrong questions (the Sanskrit-for-”war” scene: the answer wasn’t “to argue” but ”more cows”). Yet those are exactly the questions we ask AI — instead of treating it as a new form of cognition to enable us and guardrail.
• Reasoning types: expert/symbolic systems = deductive; neural networks = inductive (correlation — “the driveway’s wet, it must have rained,” at incredible scale → what people call hallucination). What none of them do is abductive reasoning — the human “magic” (Columbo walking into a murder scene: “it’s that guy… mud on his shoes, I’ve seen this pattern”). Abductive reasoning traces to pragmatism (see Erik Larson’s The Myth of Artificial Intelligence). “This is the missing part of the AI conversation” — he ”had chills” when Dave Snowden said the same on the panel that day.
• Close: ”Stop wasting time talking about AGI. Any time someone brings up AGI, go to the other side of the room and do something with AI — figure out how to interact with it, where it’s useful, and how to guardrail it.” (Deming’s System of Profound Knowledge applies.) Teaser story: “four people who could have been the Beatles of AI — we’d have had ChatGPT 20 years earlier — broke up over a petty argument between someone’s daughter and someone else.”

7. Q&A

• Regular CIOs can’t cope with this speed and also run the business — who can win the race? Key CIO competency? Stop being in denial — “you can’t put your head in the sand because you think the CEO doesn’t want you talking to the Chief AI Officer.” When you fail an audit they come to the CIO/CISO first. His whole 45-year career is ”helping organizations protect the brand” (DevOps, audit, security are all brand protection). Since AI is inevitable, the question is how to protect the brand — get the CIO on board by educating them (the DevOps trick: buy them a physical copy of the book, tell them to read it; by the third guilt-laden “have you read it?” they do — “I get it”). Be the lone wolf if you must — getting this wrong is existential to the corporation.

People, Books & References Cited

• John Michael Willis — speaker; DevOps founder; The DevOps Handbook, Beyond the Phoenix Project, Investments Unlimited; writing The Operational History of Generative AI / Rebels of Reason.
• Gene Kim, Patrick Debois, Jez Humble (DevOps Handbook co-authors), Andrew Clay Shafer (“There Is No Talent Shortage”), John Allspaw, Dr. David Woods, Dave Snowden (abductive reasoning), Simon Wardley (Jevons paradox), Patrick Debois.
• Historical figures: Walter Pitts / Warren McCulloch (1943 neural-net paper), Bertrand Russell / Whitehead (Principia Mathematica), Norbert Weiner (cybernetics), John McCarthy (coined “AI”), Grace Hopper, Fei-Fei Li, Turing, Rosenblatt, Herbert Simon.
• Erik Larson — The Myth of Artificial Intelligence (abductive reasoning / pragmatism); Eric Ries (MVP); W. Edwards Deming (System of Profound Knowledge).
• Concepts: symbolic vs. sub-symbolic; deductive/inductive/abductive reasoning; evaluations; GPQA / HumanEval / SWE-bench; Shadow AI / BYO AI; prototype ≠ production; RAG / CAG / GraphRAG; OWASP LLM Top 10; MCP / A2A; artifactory/SBOM/AIBOM; the NORMAL stack; alien cognition; Arrival, Columbo.
• Companies/tools: Docker, Dell, Chef, Red Hat, LabCorp, Air Canada, Capital One, Wiz, Hugging Face, AWS Bedrock, Google Vertex, Okta, Kubernetes/Kafka/Redis, Devin, MITRE, NIST.

Video: https://www.youtube.com/watch?v=ShDMvzenNJ8 — Transcript via yt-transcript.sh; outline generated from the transcript.