<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title></title>
    <link>https://idealloc.me/</link>
    <description>Recent content on </description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en-us</language>
    <lastBuildDate>Fri, 06 Mar 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://idealloc.me/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>we are building data breach machines and nobody cares</title>
      <link>https://idealloc.me/posts/we-are-building-data-breach-machines-and-nobody-cares/</link>
      <pubDate>Fri, 06 Mar 2026 00:00:00 +0000</pubDate>
      <guid>https://idealloc.me/posts/we-are-building-data-breach-machines-and-nobody-cares/</guid>
      <description>&lt;p&gt;A few weeks ago, my longstanding friend and colleague Curt Cunning mentioned to me that he was slogging through Nietzsche, which bespeaks his incredible will to power through things that are unpleasant (one of the many things that makes him an exceptional engineer). In any case, it lead me to revisit some of his work (Nietzsche&amp;rsquo;s, not Curt&amp;rsquo;s. Curt&amp;rsquo;s work is the opposite of a slog). My very stale memory of it from college was thinking, &amp;ldquo;Wow, this dude was on all the drugs.&amp;rdquo; Which he was, if only because he was perpetually suffering from chronic illness.&lt;/p&gt;</description>
      <content>&lt;p&gt;A few weeks ago, my longstanding friend and colleague Curt Cunning mentioned to me that he was slogging through Nietzsche, which bespeaks his incredible will to power through things that are unpleasant (one of the many things that makes him an exceptional engineer). In any case, it lead me to revisit some of his work (Nietzsche&amp;rsquo;s, not Curt&amp;rsquo;s. Curt&amp;rsquo;s work is the opposite of a slog). My very stale memory of it from college was thinking, &amp;ldquo;Wow, this dude was on all the drugs.&amp;rdquo; Which he was, if only because he was perpetually suffering from chronic illness.&lt;/p&gt;
&lt;p&gt;My mind started wandering down some showerthoughts-esque rabbit holes and it led me to a very strange place: If the AI agents are Dracula, then our role as security practitioners is that of the Belmont clan.&lt;/p&gt;
&lt;p&gt;If you&amp;rsquo;ve never played a &lt;em&gt;Castlevania&lt;/em&gt; game, you&amp;rsquo;re probably deeply confused, but stick with me. I want to use this metaphor to help you understand what an AI Agent actually &lt;em&gt;is&lt;/em&gt;, as well as talk through a very real security gap.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Castlevania&amp;rsquo;s&lt;/em&gt; Dracula is a very interesting portrayal. He is an immortal hedonist, and is endlessly locked in conflict with the Belmont Clan, who have fought (and repeatedly defeated him) for centuries. He&amp;rsquo;s the embodiment of Nietzschean morality, where he views the desperately imperfect resistance of the Belmonts as a form of hypocrisy. Unlike man, he feels no need to keep secrets about his intentions. He is the &lt;em&gt;Übermensch&lt;/em&gt;; he creates his own values beyond traditional good and evil. The way he demonstrates those values, of course, amounts to him just doing whatever he wants without inhibition: killing people indiscriminately, kidnapping damsels, whatever. Typical vampire.&lt;/p&gt;
&lt;p&gt;AI Agents are very much like this. They simply &lt;em&gt;act&lt;/em&gt;, directed by a series of prompts, injected context, and some sort of managed state. Agents are directed to a result by the outputs of a set of transformers (text-generators) that are passed through a finely tuned reward model, which has them doggedly pursue those goals with whatever tools they have available to them. Aside from the reward model, they really have no inhibitions (and it&amp;rsquo;s arguable that a reward model is closer to an alternative, programmatic hedonism than it is actual morality). Unlike Dracula however, they are ephemeral. Once their context is cleared, they effectively cease to be. However, that doesn&amp;rsquo;t mean that they can&amp;rsquo;t cause a lot of damage if left unchecked.&lt;/p&gt;
&lt;p&gt;The Belmont clan, on the other hand, are deeply flawed but driven protagonists. They are forced to reckon with their failings and inadequacies, and must find a way to thwart Dracula&amp;rsquo;s plans every time despite their constraints and limited weaponry (mostly whips). They know they cannot win the war, given their foe&amp;rsquo;s immortality, so they settle for a perpetual stalemate instead.&lt;/p&gt;
&lt;p&gt;This is the security reality of dealing with agentic workloads. We cannot win the war, so we must instead win every battle forever.&lt;/p&gt;
&lt;h2 id=&#34;know-your-enemy&#34;&gt;Know your enemy&lt;/h2&gt;
&lt;p&gt;Maybe that&amp;rsquo;s an unnecessarily adversarial header for this section, given that LLMs are designed to be perpetually helpful, but like I said, they are simply acting on the highest-scoring statistically significant output that is generated. If that says recreate a table in the production database to fix the schema or (as my manager experienced) delete all the source code in your program, that&amp;rsquo;s what they&amp;rsquo;re gonna do.&lt;/p&gt;
&lt;p&gt;The fundamental anatomy of an Agent is very simple: it&amp;rsquo;s just a loop.&lt;/p&gt;
&lt;p&gt;To clarify, thats &amp;ldquo;loop&amp;rdquo; in the machine sense. A programming instruction that tells a computer to repeatedly do something until a specific condition is met. In the case of an Agent it&amp;rsquo;s, &amp;ldquo;keep sending requests to a Large Language Model (LLM) and running the output until your task is complete or you need additional user input.&amp;rdquo;&lt;/p&gt;
&lt;p&gt;I cannot stress enough that it is actually that simple. It&amp;rsquo;s just a loop that makes API calls and runs the output.&lt;/p&gt;
&lt;p&gt;Over the last 18 months or so, the industry has begun adding a lot of structure around this core concept with two goals in mind: increasing response quality and/or reducing token usage.&lt;/p&gt;
&lt;h4 id=&#34;planning-and-subtasking&#34;&gt;Planning and subtasking&lt;/h4&gt;
&lt;p&gt;An agentic workload now usually involves a &amp;ldquo;planning&amp;rdquo; phase where the model breaks the user&amp;rsquo;s prompt down into a Directed Acyclic Graph (DAG) of sub-tasks before it starts executing tools.&lt;/p&gt;
&lt;h4 id=&#34;the-react-pattern-reasoning--acting&#34;&gt;The ReAct Pattern (Reasoning + Acting)&lt;/h4&gt;
&lt;p&gt;Agents are no longer just blindly executing tools, they&amp;rsquo;re processing the tool output, evaluating if it met the acceptance criteria of the subtask, and then self-correcting as needed. Think of it as an autonomous debugging loop.&lt;/p&gt;
&lt;h4 id=&#34;stateful-memory-vs-context-limits&#34;&gt;Stateful Memory vs Context Limits&lt;/h4&gt;
&lt;p&gt;Counterintuitively, the more context that an agent has, the &lt;strong&gt;worse the response quality becomes&lt;/strong&gt;, since it becomes more difficult for the LLM to parse the signal from the noise. Note, this is not a problem that can be solved by simply increasing the size of a context window; that actually can make it worse. The larger the context, the worse the dilution of key instructions or context becomes, leading the model&amp;rsquo;s attention mechanism to spread its &amp;ldquo;focus&amp;rdquo; across more tokens. To combat this problem, Agents are now relying more heavily on some form of external state management (often called &lt;strong&gt;Memory&lt;/strong&gt;), which is a continuously curated context that can be injected into the generation process as needed.&lt;/p&gt;
&lt;h4 id=&#34;multi-agent-orchestration&#34;&gt;Multi-Agent Orchestration&lt;/h4&gt;
&lt;p&gt;Agentic workflows are driven by &amp;ldquo;supervising&amp;rdquo; agents and operated by &amp;ldquo;worker&amp;rdquo; agents, the latter of which have specialized instructions that are optimized for specific tasks (e.g. a &amp;ldquo;SQL Generation Agent&amp;rdquo; or &amp;ldquo;Request App Agent&amp;rdquo;). The tradeoff here is the complexity of state management: how do you store and route the necessary context between agents without it leaking into other workers, potentially leading to bad/noisy output?&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;Underpinning all of these is an attempt to mitigate the largest problem that all agents have: Non-determinism. In other words, given a set of inputs (prompt, context, data), the output &lt;em&gt;will&lt;/em&gt; differ due to some inherent randomness. By constraining the tasks and the context, we can improve response quality a lot, but there are two issues that must be solved by some kind of deterministic system, ideally built right into the tool calls:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Agent Hallucination of tool inputs&lt;/strong&gt; - You can give an agent a &lt;code&gt;query_my_api&lt;/code&gt; tool, and the LLM might decide to pass a parameter that doesn&amp;rsquo;t exist, or format the JSON incorrectly. The wrapper process has to be built to catch those errors and tell the LLM, &amp;ldquo;You messed up the format of the input object, try again.&amp;rdquo; Apart from failing fast and giving good feedback, it also means that smaller, cheaper models have a better chance to get something right without taxing the compute of the system they&amp;rsquo;re interacting with.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Infinite loops AKA getting stuck in a local minima&lt;/strong&gt; - If an agent gets stuck in a logic rut (e.g., repeatedly querying a table that doesn&amp;rsquo;t exist or trying to find a missing executable that&amp;rsquo;s not in its system path), the process will just burn tokens until it hits a hard-coded iteration limit.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&#34;the-real-problem-industry-fragmentation&#34;&gt;The real problem: industry fragmentation&lt;/h2&gt;
&lt;p&gt;The biggest challenge we face is not technical. That may seem funny later, since this section may be the most techincal part of the entire blog, but the examples given are meant to highlight the real problem: the industry is moving too fast, and we are not yet to a place where there are globally adopted standards. In other words, there is no TCP/IP, or HTTPS, or (for a less technical metaphor) ACH wire transfer protocol for agents. How an agent is built depends entirely on the framework. LangGraph, CrewAI, AutoGen, and Mastra are all different and, crucially, incompatible.&lt;/p&gt;
&lt;p&gt;What was considered a state-of-the-art agent architecture six months ago is already legacy. We went from basic tool calling, to complex ReAct loops, to multi-agent frameworks, to entirely new model capabilities (like native tool-calling APIs) in less than 18 months. Even though model reasoning capabilities got a lot better, the hype is outpacing our ability to actually build anything with them due to the lack of standardization. Let me give you some examples (another warning: this will get quite techincal).&lt;/p&gt;
&lt;h4 id=&#34;llm-apis-are-wildly-inconsistent&#34;&gt;LLM APIs are wildly inconsistent&lt;/h4&gt;
&lt;p&gt;OpenAI, Google, and Anthropic handle tool-calling schemas slightly differently.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;OpenAI expects a &lt;code&gt;tools&lt;/code&gt; array where each item has a &lt;code&gt;type: &amp;quot;function&amp;quot;&lt;/code&gt; and a nested &lt;code&gt;function&lt;/code&gt; object containing the &lt;code&gt;name&lt;/code&gt;, &lt;code&gt;description&lt;/code&gt;, and the JSON Schema under a &lt;code&gt;parameters&lt;/code&gt; key.&lt;/li&gt;
&lt;li&gt;Anthropic (Claude) skips the &lt;code&gt;type&lt;/code&gt; wrapper entirely. They just want an array of objects with &lt;code&gt;name&lt;/code&gt;, &lt;code&gt;description&lt;/code&gt;, and the JSON Schema under an &lt;code&gt;input_schema&lt;/code&gt; key.&lt;/li&gt;
&lt;li&gt;Google (Gemini) uses a &lt;code&gt;tools&lt;/code&gt; array containing &lt;code&gt;functionDeclarations&lt;/code&gt;, which holds the &lt;code&gt;name&lt;/code&gt;, &lt;code&gt;description&lt;/code&gt;, and &lt;code&gt;parameters&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There are also significant differences in how the schema adherence is enforced (per-tool strict flags vs request-level mode enforcement) and how responses are received (&lt;code&gt;tool_calls&lt;/code&gt; array vs &lt;code&gt;tool_use&lt;/code&gt; context block vs &lt;code&gt;functionCall&lt;/code&gt;) This doesn&amp;rsquo;t even get into their &lt;em&gt;behavioral&lt;/em&gt; differences with regards to how prompts must be tweaked to get optimal performance out of each model. As you can imagine, building an agent that is model agnostic is a massive headache as a result. Here&amp;rsquo;s a quote from a &lt;a href=&#34;https://x.com/hxlfed14/status/2028116431876116660?s=46&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;blog&lt;/a&gt;
 about Cursor&amp;rsquo;s development of their coding agent:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Cursor tunes its harness specifically for every frontier model based on internal evals. Different models get different tool names, prompt instructions, and behavioral guidance. OpenAI Codex models get shell-oriented tool names like &lt;code&gt;rg&lt;/code&gt;; Claude models get different reasoning summary formats.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;That is not an easy world to live in. It echoes of the struggles faced by middleware proprietors for years now, but somehow even worse since LLMs are not contractual APIs.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;NOTE: MCP did not solve these problems. It&amp;rsquo;s a client-server protocol for connecting data and tools, not an universal LLM API standard. The practical implication of this is that even if an agent knows how to discover and use tools, it still has to talk to the LLM, which means that the client software (e.g. Cursor, Copilot, Amp, etc) has to dynamically translate that tool&amp;rsquo;s definition into the proprietary OpenAI, Anthropic, or Gemini JSON payloads we discussed above.&lt;/p&gt;
&lt;hr&gt;
&lt;h4 id=&#34;we-cant-distinguish-model-regressions-from-bugs&#34;&gt;We can&amp;rsquo;t distinguish model regressions from bugs&lt;/h4&gt;
&lt;p&gt;To me, the most painful place where the industry remains fragmented is in Observability, AKA &amp;ldquo;how can we debug a workflow where, by design, it may be impossible to reproduce bugs??&amp;rdquo; LLM Observability is a completely different animal than tracing through a classical (deterministic) system, like a traditional piece of software.&lt;/p&gt;
&lt;p&gt;In those systems, even the hairiest of bugs can, in theory, be reproduced because bits are getting reliably flipped somewhere, even if that reproduction involves &lt;a href=&#34;https://www.youtube.com/watch?v=tDacjrSCeq4&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;yelling at a server&lt;/a&gt;
. In order to understand the quality of agent output, you not only have to capture the behavior of your LLM system (i.e. your agent) but also the prompt, context, model, etc. Even if you&amp;rsquo;re able to recreate the stateful conditions perfectly, it could just be that &lt;a href=&#34;http://marginlab.ai/trackers/claude-code&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;your LLM was having a bad day&lt;/a&gt;
 because the model developers introduced a regression. Again, the version numbers next to a model (e.g. Opus 4.6, Codex 5.4) have nothing to do with a stable, contractual API; they&amp;rsquo;re just made up numbers to jockey for market position.&lt;/p&gt;
&lt;p&gt;So not only is it difficult to build an Agent by following a (currently nonexistent) industry standard and building against (also nonexistent) industry standard API semantics, you can&amp;rsquo;t even reproduce agent behavior reliably enough to determine if something is a fixable bug or not.&lt;/p&gt;
&lt;h2 id=&#34;on-this-episode-of-brian-krebs-security-nightmares&#34;&gt;On this episode of Brian Krebs&amp;rsquo; Security Nightmares&lt;/h2&gt;
&lt;p&gt;All of the above is to say that &lt;strong&gt;you cannot trust the agent&lt;/strong&gt;. The LLM will not govern itself, and you cannot rely on the fragmented framework layer to enforce much of anything at the moment. Sounds pretty bad, right? Ripe for a disasterous data breach?&lt;/p&gt;
&lt;p&gt;The problem is that nobody in the industry seems to care at the moment. Here&amp;rsquo;s a quote from the findings of the recent Thoughtworks retreat, &lt;em&gt;The future of software engineering&lt;/em&gt;, from a section titled &lt;strong&gt;Security is Dangerously Behind&lt;/strong&gt;:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;The retreat noted with concern that the security session had low attendance, reflecting a broader industry pattern. Security is treated as something to solve later, after the technology works and is reliable. With agents, this sequencing is dangerous. The most vivid example: granting an agent email access enables password resets and account takeovers. Full machine access for development tools means full machine access for anything the agent decides to do. The retreat&amp;rsquo;s recommendation was direct. Platform engineering should drive secure defaults by making safe behavior easy and unsafe behavior hard. Organizations should not rely on individual developers making security-conscious choices when configuring agent access.
Three priorities emerged: security by design as a non-negotiable baseline, cross-industry coalitions for interoperable agent security standards and AI-enabled defense mechanisms that can match the speed and sophistication of AI-enabled attacks.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;There is a deep, agonizing irony in a report declaring security a &amp;ldquo;non-negotiable baseline&amp;rdquo; immediately after admitting no one bothered to show up to the security session. Let&amp;rsquo;s examine each of these &amp;ldquo;priorities&amp;rdquo; in detail.&lt;/p&gt;
&lt;h4 id=&#34;security-by-design-as-a-non-negotiable-baseline&#34;&gt;Security by design as a non-negotiable baseline&lt;/h4&gt;
&lt;p&gt;This platitude is so old it makes Aesop feel contemporary. Unlike Aesop, however, there is no substance to this sentiment. When push comes to shove, the &amp;ldquo;non-negotiable&amp;rdquo; baselines are almost always the first thing negotiated away. Also, what design? The only clarifying statement around this is &amp;ldquo;Organizations should not rely on individual developers&amp;hellip; Platform engineering should drive secure defaults.&amp;rdquo; Wow, what a novel idea! Make security an infrastructure problem! Defaults! Fail closed! My kidneys are doing backflips of joy as I ascend to a higher plane of existence having been touched by this Solomonic wisdom.&lt;/p&gt;
&lt;p&gt;With all due respect, there was no point in even putting this out there except to adhere to the &lt;a href=&#34;https://en.wikipedia.org/wiki/Rule_of_three_%28writing%29&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;rule of threes&lt;/a&gt;
. It&amp;rsquo;s meaningless.&lt;/p&gt;
&lt;h4 id=&#34;cross-industry-coalitions-for-interoperable-agent-security-standards&#34;&gt;Cross-industry coalitions for interoperable agent security standards&lt;/h4&gt;
&lt;p&gt;Thing are moving in the right direction for this, but like all open standards, they will move only as fast as there are people itching to adopt them. Coalitions move at the speed of bureaucracy; AI models move at the speed of &lt;a href=&#34;https://finance.yahoo.com/news/nvidia-may-skip-gpu-release-201300773.html?guccounter=1&amp;amp;guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&amp;amp;guce_referrer_sig=AQAAAF2qUruqpBYPogYc2yjBlLO3jQW9gJIgJh3pwn4j64QJOqfO-bD7nADoCD5swp2_JNBq7yU9GFS4hH-GTNL6jwWrblyasJVgGpmj2zH18zZ-vl0-WAYPD-Fz8D3mQQzNgi6Qp9O7Cll2fgkQ6qOsmkQWiGuKZI22jjlh7qA169D1&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;NVIDIA exiting the consumer GPU market&lt;/a&gt;
. By the time a consortium agrees on an &amp;ldquo;interoperable security standard&amp;rdquo; for agent tool execution, the industry will have already moved on to a completely different architecture.&lt;/p&gt;
&lt;p&gt;Let me give you an alternative for this one that combines it with the one above it: We must design systems that assume the agent&amp;rsquo;s payload is inherently untrustworthy and non-standard. You cannot trust the agent&amp;rsquo;s internal logic; you verify the action it is trying to take against the data layer, regardless of which framework or model generated the API call. &lt;strong&gt;In other words, you govern the &lt;em&gt;ball&lt;/em&gt;, not the moving goalposts.&lt;/strong&gt;&lt;/p&gt;
&lt;h4 id=&#34;ai-enabled-defense-mechanisms-to-match-ai-enabled-attacks&#34;&gt;AI-enabled defense mechanisms to match AI-enabled attacks&lt;/h4&gt;
&lt;p&gt;Not only is this pure science fiction at this point, but injecting non-determinism into your &lt;em&gt;defensive&lt;/em&gt; layer is terrifying and &lt;strong&gt;incredibly stupid&lt;/strong&gt;. If you use an LLM to evaluate whether another LLM is doing something malicious, you now have two hallucination risks instead of one. You also risk a prompt-injection attack making it all the way to your security layer.&lt;/p&gt;
&lt;p&gt;We don&amp;rsquo;t need LLMs in our firewalls, we need the thing that we&amp;rsquo;ve been building and &lt;strong&gt;running in production&lt;/strong&gt; as an industry for more than a decade now: sophisticated anomaly-detection models and automated circuit breakers.&lt;/p&gt;
&lt;p&gt;Agents execute at machine speed. If an agent goes rogue (or is hijacked via a prompt injection) and tries to enumerate valid reset tokens by observing timing differences in API responses or rapidly exfiltrate an entire &lt;code&gt;users&lt;/code&gt; table by paginating through &lt;code&gt;SELECT&lt;/code&gt; queries, a &amp;ldquo;security guard agent&amp;rdquo; that is asynchronously (and very expensively) evaluating agent behavior will not catch it in time. &amp;ldquo;AI defense&amp;rdquo; in practice should mean deploying ML models that monitor the &lt;em&gt;behavioral exhaust&lt;/em&gt; of agentic workloads (query volume, token burn rate, iteration depth, unusual table access patterns). If the agent deviates from its bounded, purpose-based scope (i.e. it&amp;rsquo;s computed risk score is above a threshold for risk tolerance), the system should automatically sever its JIT access the millisecond the anomaly is detected.&lt;/p&gt;
&lt;p&gt;LLMs are &lt;em&gt;by design&lt;/em&gt; vulnerable to prompt injection attacks. Hence: we must step into the shoes of the Belmonts. The security flaw is immortal. &lt;strong&gt;We cannot win the war, so we must win every battle forever.&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id=&#34;what-do-we-do&#34;&gt;What do we do?&lt;/h2&gt;
&lt;p&gt;I know this blog post may be outdated in six months, but things are moving so fast I felt the need to capture this snapshot of history, if only for my own sake.&lt;/p&gt;
&lt;p&gt;Right now, we&amp;rsquo;re in the &amp;ldquo;Browser Wars&amp;rdquo; period where Netscape (Anthropic) is battling Internet Explorer (OpenAI) and Java Applets (Gemini - not intended as a slight), and the equivalent of the Document Object Model (DOM) we know today has not been standardized yet. We&amp;rsquo;re getting there, however, as the W3C is starting to publish the &lt;a href=&#34;https://w3c-cg.github.io/ai-agent-protocol/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;output from its working groups&lt;/a&gt;
, and research is being published regularly around &lt;a href=&#34;https://arxiv.org/html/2602.11327v1&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;agentic threat models&lt;/a&gt;
. Standards are coming, but as I alluded, we&amp;rsquo;d be foolish to think that they will arrive in time to prevent the first wave of agentic attacks.&lt;/p&gt;
&lt;p&gt;The good news is that we have some time to get our ducks in a row before this all really spins up. Namely, &lt;em&gt;until&lt;/em&gt; we have a set of structurally extensible, adoptable standards, the cost of implementing Agentic workloads is (candidly) very high for most companies unless it&amp;rsquo;s part of their core product offering (e.g. BI tools, log analysis vendors, etc). Those that do implement them will struggle in the same way that companies trying to maintain a web application during the browser wars struggled.&lt;/p&gt;
&lt;p&gt;Given the excitement around the possibilities, however, it seems almost certain that companies will purchase new-age BI tools (or mature BI tools with agentic features) with exploitable vulnerabilities in the flavor-of-the-month framework they used. Non-technical folks being able to ask natural language questions of their company data is too potentially useful an idea to pass up, even given this risk, since it removes one of the excuses people have for poor business velocity. lol.&lt;/p&gt;
&lt;p&gt;My argument is that what seem to be flawed, lesser tools are actually our best weapon. The tools we already know and are familiar with - anomaly-detection models, circuit breakers, data security controls, IAM role vending with short lived credentials, etc - are what we actually need to defeat Dracula. My fear is that, because whips aren&amp;rsquo;t cool anymore, nobody will care enough to use them.&lt;/p&gt;
</content>
    </item>
    
    <item>
      <title>novelty is not statistically significant</title>
      <link>https://idealloc.me/posts/novelty-is-not-statistically-significant/</link>
      <pubDate>Tue, 06 May 2025 00:00:00 +0000</pubDate>
      <guid>https://idealloc.me/posts/novelty-is-not-statistically-significant/</guid>
      <description>&lt;p&gt;Internet culture, and by extension modern popular culture, has developed a very strange relationship with science. There
is simultaneously a stronger-than-ever desire for &amp;ldquo;rational thinking&amp;rdquo; and consensus-driven mental models, and a virulent,
anti-intellectual impatience for things unknown. People want to just have the answers to things they believe that, given the pace
of human progress over the last 150 years, we should have.&lt;/p&gt;
&lt;p&gt;The myriad public and private school systems in the United States have their flaws, but I distinctly remember learning
(well before I studied biostatistics more formally in graduate school) what a null hypothesis was, and the fundamentals
of the scientific method. Namely, that an experimental observation can only be used to disprove a prediction, not affirm it.
In this sense, science is not concerned with arriving at &amp;ldquo;the correct&amp;rdquo; answer. Its goal is to deduce, when given some information,
if a hypothesis can be proven false. Scientists evaluating an idea or studying a phenomenon will &amp;ldquo;walk&amp;rdquo; themselves to a theory
of understanding, produced by either repeatedly failing to prove something wrong, or repeatedly proving things wrong. Or both.&lt;/p&gt;</description>
      <content>&lt;p&gt;Internet culture, and by extension modern popular culture, has developed a very strange relationship with science. There
is simultaneously a stronger-than-ever desire for &amp;ldquo;rational thinking&amp;rdquo; and consensus-driven mental models, and a virulent,
anti-intellectual impatience for things unknown. People want to just have the answers to things they believe that, given the pace
of human progress over the last 150 years, we should have.&lt;/p&gt;
&lt;p&gt;The myriad public and private school systems in the United States have their flaws, but I distinctly remember learning
(well before I studied biostatistics more formally in graduate school) what a null hypothesis was, and the fundamentals
of the scientific method. Namely, that an experimental observation can only be used to disprove a prediction, not affirm it.
In this sense, science is not concerned with arriving at &amp;ldquo;the correct&amp;rdquo; answer. Its goal is to deduce, when given some information,
if a hypothesis can be proven false. Scientists evaluating an idea or studying a phenomenon will &amp;ldquo;walk&amp;rdquo; themselves to a theory
of understanding, produced by either repeatedly failing to prove something wrong, or repeatedly proving things wrong. Or both.&lt;/p&gt;
&lt;p&gt;The raw output of science, theories and endless waves of new questions, might make some people &lt;em&gt;uncomfortable&lt;/em&gt;. Even more
uncomfortable are all the caveats, the nuances, and (horror of horrors) the regular death of established theory due to
observational erosion; data falling like rain upon the canyons of our understanding and carving them anew.&lt;/p&gt;
&lt;p&gt;This meticulous, creative exercise - half minesweeper, half painting a beach in your head while staring into the reflection
of the liquid in a water cooler - is the foundation of human technological progress since late antiquity. It is driven by
insight and an unquenchable desire to &lt;em&gt;know&lt;/em&gt; that can be traced like a line through the history of our species.
Maybe not everwhere and maybe not at the same time (see Sub-Roman Britain), but there&amp;rsquo;s never been a time in
history where someone was not delving into the unknown, driven by curiosity.&lt;/p&gt;
&lt;p&gt;This is why AI will not replace engineering jobs.&lt;/p&gt;
&lt;h1 id=&#34;wait-what&#34;&gt;wait, what?&lt;/h1&gt;
&lt;p&gt;The desire for easy answers and impatience with nuance extends to how we perceive complex tools, especially those that don&amp;rsquo;t behave
in a way which allow us to intuitively understand how they work. Let me give you an example.&lt;/p&gt;
&lt;p&gt;Inferential statistics (hereafter referred to simply as &amp;ldquo;statistics&amp;rdquo;) are weird. They are a very human tool, in that they
are kind of magical when they are well-designed, but can be used in incredibly stupid ways. Even well-trained, well-intentioned people
regularly misapply or misinterpret them. They can help you answer specific questions by quantifying relationships between variables,
but they aren&amp;rsquo;t meaningful in isolation. Statistics must be interpreted in the context of the study in which they were measured.&lt;/p&gt;
&lt;p&gt;It&amp;rsquo;s up to the scientist (and their consultants), to take statistical results of experiments and use them to disprove (or
fail to disprove) their &lt;em&gt;a priori&lt;/em&gt; hypothesis. It&amp;rsquo;s not automatic. You don&amp;rsquo;t hit a certain p-value and declare yourself King
of All that is Right and Good. The last step requires the researcher to use their insight to tie the math to the idea. It&amp;rsquo;s
a step, like evaluating your assumptions, that good peer reviewers will hone in on for critique.&lt;/p&gt;
&lt;p&gt;Sometimes I like to relive particular eras in MMA, so a few days ago I caught myself scrolling through a feed of Nate Diaz interviews
(which are comedy gold, if you&amp;rsquo;re curious). As you might imagine, it was inevitable that clips from Joe Rogan&amp;rsquo;s podcast would start
popping up in this feed as well. One of these clips was from his interview with Neil DeGrasse Tyson, where Joe opened with,
&amp;ldquo;Do you think that it would be beneficial to teach probability and statistics to people?&amp;rdquo;. Almost in spite of myself, I stopped scrolling
to watch.&lt;/p&gt;
&lt;p&gt;After letting out an excited noise of affirmation, Dr. Tyson went on to describe&amp;hellip;something? An analysis? a study? I&amp;rsquo;m not sure what
it was, but some sort of scientific exercise looking at the phenomenon of a &amp;ldquo;Hot Hand&amp;rdquo; in basketball. His assertion, or the assertion
of the thing that he agreed with, was that the &amp;ldquo;Hot Hand&amp;rdquo; didn&amp;rsquo;t actually exist because, if you&amp;rsquo;re hitting 50% outside the paint,
the likelyhood that you&amp;rsquo;ll hit several shots in a row is &amp;ldquo;high&amp;rdquo;.&lt;/p&gt;
&lt;p&gt;Even if he was absolutely blazed out of his skull, this was so bizarrely backwards that my jaw hit the floor. I had to listen back
to the clip several times to be sure I heard it correctly, then found the original conversation and listened to that to make sure
it was not cut oddly for the short. When you study an observable phenomenon, the statistics cannot tell you if the thing exists or
not. It can only tell you if its existence is anomalous at a specific resolution. Finding that there&amp;rsquo;s no statistically significant
correlation between percentage of three-point shots made and the number of consecutive shots made cannot be used to conclude
that the Hot Hand phenomenon isn&amp;rsquo;t real; it just means that it can&amp;rsquo;t be explained by that specific statistic.&lt;/p&gt;
&lt;p&gt;What happens when you look at it with regards to defensive matchups? Scheme? Specific dynamics between two players? Someone getting
&amp;ldquo;hot&amp;rdquo; just means that something about their circumstances is making them play above their average for some amount of time. Perhaps their
warm up was particularly effective today, and they play well until they get shoulder checked and some pain causes them to start missing.
Alternatively, maybe they play well until a defensive adjustment is made to prevent a player from exploiting a weakness that they were using
to score.&lt;/p&gt;
&lt;p&gt;The thing that&amp;rsquo;s interesting is not using the statistics to determine if there&amp;rsquo;s a fallacy or not, it&amp;rsquo;s figuring out under what circumstances
a Hot Hand can be reliably observed, or even manufactured. It could be impossible to figure that out because there are too many variables at play,
but to simply dismiss the phenomenon because some math says that it can&amp;rsquo;t explain it is such an uninteresting and incurious position to take.
In other words, its unscientific.&lt;/p&gt;
&lt;p&gt;This same trap of mistaking statistical correlation (or lack thereof) for understanding is one that so many keep falling into with Large Language
Models (LLMs).&lt;/p&gt;
&lt;p&gt;LLMs are fundamentally regurgitative. I don&amp;rsquo;t just mean in the sense that the code they produce looks like vomit, but more that they are
&lt;em&gt;by design&lt;/em&gt; incapable of producing insight. They have no curiosity. Their worlds are static snapshots of human-produced training material.
They cannot push against the established canon, because the canon dictates the boundaries of their knowlege base. To an LLM, Hot Hands are
a fallacy until they aren&amp;rsquo;t, or they aren&amp;rsquo;t until they are. If you ask an LLM, it will just tell you that it&amp;rsquo;s an ongoing debate and cite its
sources. This is actually really useful behavior, but if it were up to an AI agent to continue research, our understanding of the topic would not
have been meaningfully advanced past 1985. There&amp;rsquo;s no desire to challege convention or revisit well-established principles. If you asked it what
avenues should be explored next, it&amp;rsquo;ll just parse ideas from the conclusions of the papers it downloaded and fire them at you. At the end of the
day, it&amp;rsquo;s just computing, not really &lt;em&gt;thinking&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;I think this is easily illustrated with an exercise.&lt;/p&gt;
&lt;h1 id=&#34;word-problems&#34;&gt;word problems&lt;/h1&gt;
&lt;p&gt;I love word problems. I have since I was a child. Especially the ones with the little gotchas you have to think through. It turns
out that LLMs struggle to solve even simple ones consistently. Here&amp;rsquo;s one I made up for fun:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Billy is stoked that Michael is willing to part ways with some of the ultra-rare baseball cards that he&amp;rsquo;s collected.
Michael has 3 pristine Mickey Mantles, 2 slightly worn Mark McGwires, 4 slightly worn Mickey Mantles, and a barely-recognizable Babe Ruth.
The base prices of Mickey Mantle cards are $200, Mark McGwire cards are $50, and Babe Ruth is $1000.
The condition of the cards modifies the price, where pristine is 100% value, slightly-worn is 75%, worn is 50%, very worn is 25%, and barely-recognizable is 5%.
After the deal, Michael takes the proceeds and buys a set of classic gaming consoles for $500 and 9 games for $22 each. He has $27 left. What cards does Michael still have?&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Every single model I tried got this wrong. What&amp;rsquo;s illuminating is that they all got them wrong in &lt;em&gt;exactly the same way&lt;/em&gt;. Let me share the output
from Gemeni 2.5 Pro, Claude 3.7 Sonnet, and ChatGPT 4o (going to skip the math because they&amp;rsquo;re all the same):&lt;/p&gt;
&lt;p&gt;Gemeni:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Looking at his initial inventory, the only combination of cards that totals exactly $600 is the 3 pristine Mickey Mantles ($3 * $200 = $600).&lt;/p&gt;
&lt;p&gt;Therefore, the cards Michael still has are the ones whose value equals the amount he didn&amp;rsquo;t receive in the sale.&lt;/p&gt;
&lt;p&gt;Michael still has the 3 pristine Mickey Mantles.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Claude:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Now I need to determine which combination of cards Michael sold to get exactly $600.
Looking at the possible combinations:&lt;/p&gt;
&lt;p&gt;If Michael sold all 4 slightly-worn Mickey Mantles: $600 ✓&lt;/p&gt;
&lt;p&gt;This matches exactly the $600 we calculated!
Therefore, Michael still has:&lt;/p&gt;
&lt;p&gt;2 slightly worn Mark McGwires ($37.50 each)
3 pristine Mickey Mantles ($200 each)
1 barely-recognizable Babe Ruth ($50)&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;ChatGPT:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Therefore, Michael sold:
4 slightly worn Mickey Mantles&lt;/p&gt;
&lt;p&gt;2 slightly worn Mark McGwires&lt;/p&gt;
&lt;p&gt;1 barely-recognizable Babe Ruth&lt;/p&gt;
&lt;p&gt;And Michael still has:
3 pristine Mickey Mantles&lt;/p&gt;
&lt;p&gt;✅ Final Answer: Michael still has 3 pristine Mickey Mantle cards.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;I also did this with ChatGPT o3 to see how a &amp;ldquo;reasoning model&amp;rdquo; would do, and despite taking the longest to come up with an answer, it ended up at the
same conclusion as all the other models, except that it had to run O(n^4) python code to get to it. It did give me a nicely formatted table though:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;To raise the $725 he spent ( $500 for the console set + 9 × $22 for the games + $27 left over), Michael could sell:&lt;/p&gt;
&lt;table&gt;
  &lt;thead&gt;
      &lt;tr&gt;
          &lt;th&gt;Card (type &amp;amp; condition)&lt;/th&gt;
          &lt;th&gt;Value each&lt;/th&gt;
          &lt;th&gt;Quantity sold&lt;/th&gt;
          &lt;th&gt;Sub‑total&lt;/th&gt;
      &lt;/tr&gt;
  &lt;/thead&gt;
  &lt;tbody&gt;
      &lt;tr&gt;
          &lt;td&gt;Mickey Mantle – slightly worn&lt;/td&gt;
          &lt;td&gt;$200 × 75 % = $150&lt;/td&gt;
          &lt;td&gt;4&lt;/td&gt;
          &lt;td&gt;$600&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;Mark McGwire – slightly worn&lt;/td&gt;
          &lt;td&gt;$50 × 75 % = $37.50&lt;/td&gt;
          &lt;td&gt;2&lt;/td&gt;
          &lt;td&gt;$75&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;Babe Ruth – barely recognisable&lt;/td&gt;
          &lt;td&gt;$1 000 × 5 % = $50&lt;/td&gt;
          &lt;td&gt;1&lt;/td&gt;
          &lt;td&gt;$50&lt;/td&gt;
      &lt;/tr&gt;
      &lt;tr&gt;
          &lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
          &lt;td&gt;&lt;/td&gt;
          &lt;td&gt;&lt;/td&gt;
          &lt;td&gt;&lt;strong&gt;$725&lt;/strong&gt;&lt;/td&gt;
      &lt;/tr&gt;
  &lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;That combination uses every slightly‑worn Mantle, both McGwire cards, and the (low‑grade) Babe Ruth—leaving the three pristine Mickey Mantle cards in Michael’s collection.&lt;/p&gt;
&lt;p&gt;So after the deal, Michael still has 3 pristine Mickey Mantle cards.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;First off, all four of the models discounted the other possible answers without explaination - 4 slightly-worn Mickey Mantles are also worth $600.
None of them suggested that the answer could be either groups of Mickey Mantles cards that were left over. All except ChatGPT o3 missed the possiblity of
selling 1 pristine Mickey Mantle, 3 slightly-worn Mickey Mantles, and 2 Mark McGwires, but even it discarded that for reasons it didn&amp;rsquo;t explain.
If you&amp;rsquo;re thinking that the &amp;ldquo;gotcha&amp;rdquo; is that there are three correct answers here, though, you&amp;rsquo;re wrong. There&amp;rsquo;s only one.&lt;/p&gt;
&lt;p&gt;The thing that all four models discarded was the first sentence of the problem.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Billy is stoked that Michael is willing to part ways with some of the ultra-rare baseball cards that he&amp;rsquo;s collected.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Poor Billy. None of these models give him any agency at all. He isn&amp;rsquo;t just a passenger in Michael&amp;rsquo;s world though, nor is he just going to accept
whatever the hell Michael decides to give to him. He wants the ultra-rare stuff, and he&amp;rsquo;s specifcally happy that Michael is willing to part ways with it!
There&amp;rsquo;s no reason to be happy if he wasn&amp;rsquo;t getting those pristine Mickey Mantles in the deal. He&amp;rsquo;d also prefer to get the Babe Ruth, even if it
is damaged, instead of more Mickey Mantles (most of which are slightly-worn). The actual answer is, therefore, that the only cards left are the
4 slightly-worn Mickey Mantles.&lt;/p&gt;
&lt;p&gt;You may be thinking that my question wasn&amp;rsquo;t fair, but by word problem standards my example is very pedestrian. First of all, I did not include
many other combinations of things that would add up to the remaining amount. Second, I laid the language of the problem out in really clean terms to make
the required math easy to parse, which word problems often try to obfuscate to some degree. Third, I (mostly) stuck to nice, round whole numbers and avoided
floating-point math, which LLMs are known to struggle with. Finally, the only qualitative context I added to the problem was the first line. Really
tricky word problems contain several different, sometimes conflicting, peices of information that must be used to narrow the correct answer down.&lt;/p&gt;
&lt;p&gt;Let&amp;rsquo;s take the tack that this is too hard for an LLM to solve without some hints. When I ask Gemini about the possibility of it being the four,
slightly-worn Mickey Mantles instead, it correctly identifies that it&amp;rsquo;s a valid solution (albeit while still not considering the third one) and concludes:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Without further information about which specific cards Michael was willing to part with or keep (e.g., prioritizing condition, character, etc.), we cannot definitively say whether he kept the 3 pristine Mantles or the 4 slightly-worn Mantles. Both are mathematically consistent with the total amount of money he received from the sale.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;When I point out the qualitative statement at the beginning that should help it narrow down the answer, it says:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Considering Michael is a collector who values &amp;ldquo;ultra-rare&amp;rdquo; cards and is described as being willing to part with some, it is highly probable that he would keep the cards that are most desirable to a collector. Between pristine cards and slightly-worn versions of the same player, collectors typically value condition very highly.
Therefore, the most likely scenario is that Michael, the collector, chose to keep the 3 pristine Mickey Mantles (the ones in the best condition of a desirable player). This means the cards he parted ways with for $725 were the 4 slightly-worn Mickey Mantles, the 2 slightly-worn Jerry McGuires, and the barely-recognizable Babe Ruth. Billy would indeed be &amp;ldquo;stoked&amp;rdquo; to acquire Mickey Mantles (even slightly-worn ones) and a Babe Ruth card.
The qualitative statement helps us infer Michael&amp;rsquo;s likely motivation as a collector, leading to the conclusion that he kept the cards most valuable in terms of condition.
Michael still has the 3 pristine Mickey Mantles.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Even leading the model by the nose to the answer, it can&amp;rsquo;t figure it out. Michael clearly doesn&amp;rsquo;t value ultra-rare cards if he&amp;rsquo;s willing to part with
them. In fact, even a cursory reading of the problem makes it clear that Michael is shifting from baseball card collection to classic video game
collection. All the other models ran into the same pitfalls, including ChatGPT o3; although it had to run some more terrible python code to ensure maximum
ozone depletion in the process. All models were able to correctly reason about the answer after I gave it to them, which is nice I guess.&lt;/p&gt;
&lt;p&gt;I reran all my prompt through all these models several times each, and all of them got the correct answer at least once. Which answer they selected
seemed totally random; if you ran them repeatedly to find the correct answer based on a distribution of responses from each model, it still
wouldn&amp;rsquo;t have worked, and if you were using their APIs you probably would have burned a lot of tokens. Every single time LLMs answer something correctly,
it&amp;rsquo;s important to understand that they have performed a mechanically-equivalent act to responding with the first answer that looks right.&lt;/p&gt;
&lt;p&gt;The fundamental issue, however, is that an average, word-problem-loving middle schooler could figure this out, even if they don&amp;rsquo;t know who Mickey Mantle is,
as long as they focus on value as a proxy for rarity. It&amp;rsquo;s not even particularly novel or interesting. LLMs, however, seem to be pretty
bad at solving word problems generally; an idea that is supported by the &lt;a href=&#34;https://aclanthology.org/2024.findings-acl.524.pdf&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;current&lt;/a&gt;
 &lt;a href=&#34;https://aclanthology.org/2024.findings-naacl.72.pdf&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;literature&lt;/a&gt;

on the subject.&lt;/p&gt;
&lt;p&gt;One quote, in particular, stands out to me:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;The poor performance of LLMs on MCGSM8K confirms the incapable ability of most LLMs in mathematical
reasoning process evaluation. In particular, the performance on GSM8K-Judgement exhibits that it
is easier to identify incorrect solutions. However, merely fine-tuning with correct solutions improves
the performance in evaluating correct solutions, but leads to a huge drop in evaluating incorrect solutions.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Huh&amp;hellip;why does that happen?&lt;/p&gt;
&lt;h1 id=&#34;good-strategy-requires-self-awareness&#34;&gt;good strategy requires self-awareness&lt;/h1&gt;
&lt;p&gt;In the last section of Richard Rumelt&amp;rsquo;s book &lt;em&gt;Good Strategy/Bad Strategy&lt;/em&gt;, he describes a scenario where he&amp;rsquo;s requested a paragraph-long diagnosis on
the woes of TiVo from some executives he&amp;rsquo;s running a strategy retreat for. Each of them read the case the night before, and each of them had written
something down regarding TiVos market position, competitors, production issues, etc. Instead of asking them about the substance of their positions
however, Dr. Rumelt instead asks them to describe how they arrived at their hypotheses.&lt;/p&gt;
&lt;p&gt;Most of them are stumped. With some gentle prodding, every person admits that (independent of the merits of their diagnosis) they wrote down what
amounted to the first thing that came to their head.&lt;/p&gt;
&lt;p&gt;LLMs mirror this behavior to a T, but unlike humans, they cannot be coached out of it. They can&amp;rsquo;t deliver insight, because they aren&amp;rsquo;t self-aware and
have brutally limited contexts; they are confined to their training data and the immediate prompt, and lack embodied experience or situational awareness.
They cannot question their own assumptions. They cannot stop to reconsider their position because they cannot stop vomiting statistically-significant
tokens into API responses. Therefore, they are universally susceptible to the pitfalls of producing Bad Strategy. Similarly, because they are incapable
of the logical and creative exercise of producing insight, they cannot synthesize Good Strategy.&lt;/p&gt;
&lt;p&gt;If you train them on what &amp;ldquo;correctness&amp;rdquo; looks like (as the researchers I quoted previously did), they will struggle to distinguish it from &amp;ldquo;incorrectness&amp;rdquo; that is
semantically similar. This is known as &amp;ldquo;overfitting&amp;rdquo;, and it&amp;rsquo;s a classic problem that all model makers face when trying to improve their model&amp;rsquo;s ability
to perform a specific task. Even though different model stewards have attempted to solve this problem with techniques like balanced training or contrastive learning,
LLMs still fail to process logic reliably. While they can mimic the surface-level patterns of correct solutions, they cannot identify critical qualitative nuances
that weren&amp;rsquo;t explicitly and overwhelmingly represented in their training.&lt;/p&gt;
&lt;p&gt;An LLM cannot tell you what to do because it has no risk tolerance. It can&amp;rsquo;t stop you from choosing the wrong course of action because the world has too much
context to fit into its window and it cannot intuit relevance. This should make sense, because an LLM is just a statistical model. Just as science is
not run by the Church of The p-value, strategy - i.e., novelty - cannot be left to the machines. The model can&amp;rsquo;t search for what we haven&amp;rsquo;t shown it,
and it can&amp;rsquo;t create something that is unlike what we&amp;rsquo;ve already made. It&amp;rsquo;s a probabilistic machine operating in a universe governed by laws.&lt;/p&gt;
&lt;p&gt;There will be a wave of people who vehemently disagree with this, and will attempt to create or use &amp;ldquo;strategy agents&amp;rdquo;. I recommend finding out which
companies would have leadership so bankrupt of integrity that they would abdicate strategy work, then build businesses to crush them.&lt;/p&gt;
&lt;h1 id=&#34;ai-what-is-it-good-for&#34;&gt;AI, what is it good for?&lt;/h1&gt;
&lt;p&gt;This is the third technological transition I&amp;rsquo;ve experienced during my career. The job title I had when I transitioned fully from data science to
engineering was &amp;ldquo;Database Administrator&amp;rdquo;; a job that is relegated to &amp;ldquo;legacy&amp;rdquo; tech, where it will be done by aging, increasingly expensive consultants
until it fades from existence entirely. My dad was a COBOL programmer, who experienced the same thing at the tail end of his career. People even
tried to accelerate software engineering with code generation tooling, refered to as CASE (computer assisted software engineering). The tool my Dad
used was called COOL:Gen, and even though it was deterministic, it feels like it was cut from the same cloth as modern LLM-based tools. A powerful
but expensive and limited black box. It was great at code porting, but required extensive knowledge both of the target language and the COOL:Gen DSL.
Crucially, it never replaced programmmers.&lt;/p&gt;
&lt;p&gt;I think the new wave of LLM-based CASE tools are going to be incredible, and much more flexible, but operate in fundamentally the same lane as software
like COOL:Gen. They&amp;rsquo;re already good at that type of work, and getting better every day. I regularly send deep research bots off to parse hundreds of web pages on my behalf.
We&amp;rsquo;re not far off from having a bot take a decent stab at implementing a well-designed interface that you&amp;rsquo;ve alredy built. One of the things I&amp;rsquo;m most
excited for is reclaiming the part of my brain that knows Regex, since LLMs are so good at generating it.&lt;/p&gt;
&lt;p&gt;I consider none of those tasks essential to the value that I provide to the company I work for.&lt;/p&gt;
&lt;p&gt;I&amp;rsquo;m not scared that LLM agents are going to take engineering jobs, for the same reason that a mechanic shouldn&amp;rsquo;t be afraid of losing their
job because their employer purchases a software diagnostic tool. Good engineering is a creative endeavor where design is driven by
context as much as it is a specific problem. It&amp;rsquo;s about finding an elegant solution to the maze of constraints that you&amp;rsquo;re presented with.
In other words: engineering is more akin to strategy than computation.&lt;/p&gt;
&lt;p&gt;That&amp;rsquo;s not to say that no jobs will dissapear; tier 1 text-based support is most certainly going to be an anachronism, much like human calculators were. Other jobs
where natural language processing is done by humans are ripe for automation. Software engineering jobs will simply change in scope and work, much like the DBAs
and Webforms developers of yesteryear. CRUD applications will be commoditized, and in order to survive in the industry you&amp;rsquo;ll need to pivot and upskill. However,
just like software-defined networking didn&amp;rsquo;t make the fundamentals of networking obselete, neither will probabilistically generated software render the fundamentals
of engineering obselete.&lt;/p&gt;
&lt;p&gt;We&amp;rsquo;re better off leaning into the strengths of LLMs, than attempting to make them suck the joy out of our lives by automating the process of wonder,
insight, and discovery. They can&amp;rsquo;t do it to anything approaching an acceptable standard, and we risk letting our collective intellectual muscles atrophy
in letting them try.&lt;/p&gt;
</content>
    </item>
    
    <item>
      <title>about</title>
      <link>https://idealloc.me/about/</link>
      <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
      <guid>https://idealloc.me/about/</guid>
      <description>&lt;p&gt;Heya, I&amp;rsquo;m Haris. I&amp;rsquo;ve been working in and around software for about 15 years. This site is a place to dump all the stuff I have swimming in my head.&lt;/p&gt;
&lt;p&gt;Things I talk about:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Engineering Strategy&lt;/li&gt;
&lt;li&gt;Distributed Systems&lt;/li&gt;
&lt;li&gt;Teaching, mentorship, and paying it forward&lt;/li&gt;
&lt;li&gt;Screaming at the tides - random tirades against things I&amp;rsquo;m too dumb or old to understand&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I&amp;rsquo;m currently employed as a Staff SRE at &lt;a href=&#34;https://immuta.com&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;Immuta&lt;/a&gt;
. We&amp;rsquo;ve done a lot of cool stuff. Here are some other blogs I wrote there:&lt;/p&gt;</description>
      <content>&lt;p&gt;Heya, I&amp;rsquo;m Haris. I&amp;rsquo;ve been working in and around software for about 15 years. This site is a place to dump all the stuff I have swimming in my head.&lt;/p&gt;
&lt;p&gt;Things I talk about:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Engineering Strategy&lt;/li&gt;
&lt;li&gt;Distributed Systems&lt;/li&gt;
&lt;li&gt;Teaching, mentorship, and paying it forward&lt;/li&gt;
&lt;li&gt;Screaming at the tides - random tirades against things I&amp;rsquo;m too dumb or old to understand&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I&amp;rsquo;m currently employed as a Staff SRE at &lt;a href=&#34;https://immuta.com&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;Immuta&lt;/a&gt;
. We&amp;rsquo;ve done a lot of cool stuff. Here are some other blogs I wrote there:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Terraform Engineering
&lt;ul&gt;
&lt;li&gt;&lt;a href=&#34;https://medium.com/immuta-engineering/terraform-engineering-strategies-part-1-make-your-modules-thread-safe-1a7fdf1c2a2c&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;Make your modules thread-safe&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href=&#34;https://medium.com/immuta-engineering/terraform-isnt-real-cloud-provider-apis-ownership-and-lifecycle-blocks-a911dea1daba&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;Terraform isn&amp;rsquo;t real&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;Distributed Systems
&lt;ul&gt;
&lt;li&gt;&lt;a href=&#34;https://www.immuta.com/blog/multi-cloud-networking/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34; &gt;Why is Multi-cloud Networking so Complicated?&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;
</content>
    </item>
    
  </channel>
</rss>
