Knowledge check: How AI actually works

You've Been Using AI. Now Understand What It's Actually Doing.

Picture this: It's Tuesday afternoon. Your team has just spent 45 minutes arguing about whether the AI-generated market analysis is trustworthy. One colleague insists ChatGPT "looked it up online." Another thinks it's "basically just autocomplete." Your VP wants to know why Claude gave a different answer to the same question. You're nodding along, but privately you're not sure who's right. This is the moment where understanding how AI actually works stops being academic and becomes a professional advantage. The people who grasp the mechanics — even at a conceptual level — make faster decisions, write better prompts, catch errors that others miss, and stop being surprised when AI behaves strangely. That's exactly what this lesson delivers.

What These Models Actually Are

ChatGPT, Claude, Gemini, and their cousins are all large language models, or LLMs. The "large" part is literal: GPT-4 is estimated to have around 1.8 trillion parameters — the numerical weights that determine how it responds to any input. These parameters were set during training, a process where the model processed hundreds of billions of words from books, websites, code repositories, and more. During that training, the model learned statistical patterns: which words follow which other words, how concepts relate, what a "good" answer to a question tends to look like. It did not memorize a database. It built a compressed, probabilistic map of language. That distinction matters enormously for how you use these tools at work.

When you type a prompt into ChatGPT, the model doesn't search the internet (unless it has a browsing tool explicitly enabled), consult a database, or "think" in any human sense. It processes your input as tokens — chunks of text roughly equivalent to three-quarters of a word on average — and predicts the most statistically likely continuation. GPT-4 processes up to 128,000 tokens in a single context window, which is roughly 96,000 words. Claude 3.5 Sonnet handles up to 200,000 tokens. That's the entire text of a long novel. Every word you've written in the conversation, plus the model's responses, sits in that window simultaneously. The model reads all of it every time it generates the next token. This is why long conversations can shift in quality — and why your early instructions still influence answers given 50 exchanges later.

Training vs. Real-Time Knowledge: The Distinction That Saves You From Bad Decisions

Every LLM has a training cutoff — a date after which it has no knowledge of world events. GPT-4's knowledge cuts off in early 2024. Claude 3.5 Sonnet's cutoff is April 2024. Gemini 1.5 Pro extends slightly further. This isn't a flaw that will be patched; it's structural. The model was trained on a static dataset. When you ask ChatGPT about a competitor's Q3 earnings, a regulation passed last month, or a market shift from six weeks ago, it either tells you it doesn't know — or, more dangerously, it confabulates. Confabulation (often called hallucination) is when the model generates a plausible-sounding but factually wrong answer. It's not lying. It's pattern-matching in the absence of real data, producing text that fits the shape of a correct answer without the substance. Recognizing this behavior is the single most important safety skill for professionals using AI at work.

Perplexity AI takes a different architectural approach: it combines an LLM with real-time web search, citing sources inline. This makes it far more reliable for current events, recent research, or live market data. ChatGPT's browsing mode (available in GPT-4o) does something similar — but it's a tool layer on top of the base model, not a fundamental change to how the model works. When browsing is off and no documents are attached, ChatGPT is operating purely from training data. Most professionals don't know when browsing is active versus inactive, which leads to misplaced confidence in outputs that are actually frozen in the past. Check the interface before trusting any time-sensitive answer.

How the Model Generates an Answer — Token by Token

The generation process is sequential and probabilistic. After processing your prompt, the model assigns a probability to every token in its vocabulary (GPT-4's vocabulary contains around 100,000 tokens) and selects one — not always the highest-probability option, because a setting called temperature introduces controlled randomness. At temperature 0, the model always picks the most likely next token, producing consistent, conservative outputs. At higher temperatures (ChatGPT's default sits around 0.7-1.0), it occasionally picks less likely tokens, which produces more creative, varied text. Then it repeats the process. The entire response — every word — is generated one token at a time, with each token influencing the probability distribution for the next. This is why AI outputs can start confidently and drift, why interrupting mid-sentence sometimes produces incoherence, and why regenerating the same prompt can produce a noticeably different answer.

Putting It to Work: Writing Prompts That Match the Mechanics

Understanding token-by-token generation changes how you write prompts. The model reads your entire prompt before generating a single word of response, which means the order and structure of your instructions genuinely matter. Instructions buried at the end of a long prompt are processed — but they compete with everything that came before. Research on prompt sensitivity shows that moving a key constraint from the end to the beginning of a prompt can change output quality significantly. Front-load your most important requirements. State the format you want before you describe the task. Specify the audience, the length, the tone, and any hard constraints in the first two or three sentences. Then provide the context and content. This mirrors how the model actually processes information and produces noticeably better results.

The same mechanical understanding applies to multi-step tasks. Because the model generates sequentially, asking it to "think step by step" before reaching a conclusion genuinely improves accuracy on analytical and mathematical tasks. This technique — called chain-of-thought prompting — works because it forces the model to generate intermediate reasoning tokens, which then influence subsequent tokens. The model that has written "revenue declined because unit volume dropped 12% while price held steady" is better positioned to generate a correct strategic recommendation than one that jumped straight to conclusions. You're not tricking the AI. You're using the architecture to your advantage.

Notice what that prompt did structurally. It front-loaded the role and output purpose. It gave the model a numbered reasoning sequence before asking for synthesis. It explicitly requested uncertainty flagging — which is how you get a model to self-regulate rather than confabulate confidently. The response is usable in a real meeting, but more importantly, it's auditable. You can see exactly where the model's reasoning came from, which lets you fact-check efficiently rather than treating the output as a black box. That's the professional standard for AI-assisted analysis.

RLHF: Why Claude Sounds Different From GPT-4

After initial training on raw text, every major commercial LLM goes through a second process called Reinforcement Learning from Human Feedback, or RLHF. Human raters evaluate model outputs and indicate which responses are better — more helpful, more accurate, less harmful. The model is then fine-tuned to produce outputs that score higher with human raters. This is why ChatGPT, Claude, and Gemini all feel different despite being built on similar transformer architectures. Anthropic trained Claude with a specific emphasis on being "helpful, harmless, and honest," and their Constitutional AI approach adds an additional layer of principle-based self-critique. OpenAI's RLHF process for ChatGPT optimized differently — with different rater pools, different reward signals, and different safety constraints. The result is that the same analytical question can get a notably different answer depending on which model you ask, not because one is objectively correct, but because their behavioral training diverged.

What This Means for Different Roles on Monday

• Managers: When your team's AI output looks authoritative, ask one question before sharing it up: 'Is this time-sensitive information?' If yes, verify independently or use Perplexity with citations. Training cutoffs make AI unreliable for anything that changed in the last 6-12 months.
• Analysts: Use chain-of-thought prompting for any multi-step reasoning task. Explicitly instruct the model to show its working before stating conclusions. You'll catch errors faster and produce outputs that are easier for stakeholders to scrutinize.
• Marketers: High-temperature outputs (creative, varied) are features for brainstorming. Low-temperature behavior (consistent, structured) is what you want for templated content at scale. When using the API or tools that expose settings, match temperature to task type.
• Consultants: Claude's tendency to flag uncertainty and Anthropic's Constitutional AI training make it well-suited for sensitive client-facing drafts where you need the model to self-regulate. For raw analytical horsepower on large documents, Claude's 200K context window is a practical advantage.
• Product and Operations Teams: GitHub Copilot and Notion AI are purpose-built for specific workflows — don't evaluate them against general-purpose models like GPT-4o. Each occupies a different architectural niche. Choosing the right tool for the workflow is more important than choosing the 'best' model.

The Practical Upshot: A Framework for Trusting AI Output

Armed with what you now know about how these models work — probabilistic token generation, training cutoffs, RLHF-shaped behavior, context window dynamics — you can build a simple mental framework for every AI output you encounter professionally. Ask three questions. First: is this information time-sensitive? If yes, treat the output as a draft hypothesis and verify with a sourced tool like Perplexity or direct research. Second: did the model show its reasoning, or did it jump to a conclusion? If it jumped, regenerate with a chain-of-thought instruction. Third: does this output match what I know about this model's training priorities? A Claude output that declines to speculate is behaving correctly. A ChatGPT output that gives you a confident number without a source is a prompt for verification, not acceptance.

This framework changes your posture from passive consumer to active director of AI output. You're not hoping the AI gets it right — you're designing prompts that make it harder for the AI to get it wrong. You're choosing tools that match the knowledge requirements of the task. You're reading outputs with an understanding of where errors are most likely to occur. That's not a minor upgrade to your workflow. Professionals who operate this way consistently produce better AI-assisted work, catch costly errors before they reach stakeholders, and build the kind of institutional credibility that comes from being reliably right rather than occasionally impressive.

Why the Same Prompt Gets Different Answers

Picture this: your colleague runs a prompt through ChatGPT on Tuesday and gets a sharp, structured analysis. You run the exact same prompt on Thursday and get something noticeably different — softer, more hedged, missing a key angle. You assume one of you changed the wording. You didn't. This isn't a bug or a server glitch. It's a direct consequence of how language models generate output — through probabilistic sampling, not deterministic lookup. Understanding this one mechanism changes how you interpret AI outputs and how seriously you take any single response.

Temperature and Sampling: The Randomness Engine

Every time a model generates a token, it doesn't just pick the single highest-probability word. Instead, it samples from a probability distribution — a ranked list of candidate tokens, each with a likelihood score. A setting called temperature controls how that sampling behaves. At temperature 0, the model always picks the top-ranked token, producing consistent, predictable output. At higher temperatures (say, 0.8 or 1.0), the model introduces controlled randomness, occasionally selecting lower-ranked tokens. This is why creative tasks benefit from higher temperature — you get more varied, surprising language — while analytical tasks often perform better at lower settings. ChatGPT's default temperature sits around 0.7, which explains why professional outputs can drift between runs even when your prompt stays identical.

Most consumer tools like ChatGPT and Claude don't expose temperature controls directly to users. But understanding that this dial exists — and that it's set differently across tools and use cases — helps you interpret inconsistency rationally. When you need reproducibility (legal summaries, financial analysis, compliance documentation), you want a model running at low temperature, or you want to compensate by using more specific, constraining prompts. When you want brainstorming, narrative drafts, or creative options, higher-temperature behavior actually works in your favor. The practical move: if a response is surprisingly good, save it immediately. The model won't reproduce it exactly on demand.

Context Windows: The Model's Working Memory

The context window is the total amount of text a model can 'see' at once — your prompt, the conversation history, any documents you've pasted in, and the model's own previous responses. Everything outside the context window is invisible to the model. It doesn't exist. This is the single most misunderstood limitation in professional AI use. When a long conversation with Claude suddenly produces an answer that ignores an important constraint you mentioned twelve messages ago, the constraint has likely scrolled out of the active context. The model isn't being careless — it literally cannot see that text anymore.

Context window sizes vary significantly across current models and matter enormously for professional tasks involving long documents. GPT-4o supports up to 128,000 tokens — roughly 96,000 words, or a full-length business book. Claude 3.5 Sonnet handles 200,000 tokens, making it particularly strong for contract review and research synthesis. Gemini 1.5 Pro pushes to 1 million tokens in some configurations. For comparison, GPT-3.5 was capped at 4,096 tokens — about three pages of text. This is why early ChatGPT users hit frustrating walls mid-conversation that modern users rarely encounter. Knowing your tool's context limit lets you structure long-form work sessions intelligently rather than debugging phantom 'memory' problems.

How Models Handle Documents You Paste In

When you paste a 20-page report into Claude and ask for a summary, the entire document gets tokenized and placed into the context window alongside your instructions. The model processes all of it in a single forward pass through its neural network — no separate 'reading' phase, no paragraph-by-paragraph scan. This is fundamentally different from how a search engine works. Google indexes documents and retrieves relevant chunks. A language model ingests your document as raw context and generates output by attending to all of it simultaneously through its transformer architecture. This is why models can answer nuanced questions about specific passages rather than just returning keyword matches — they're reasoning across the full text, not pattern-matching against an index.

Notice what made that prompt work: it specified the output format before asking the question, defined what 'forward-looking' means operationally, and added a quality filter (flag missing owners and deadlines). The model didn't need to be told what accountability gaps are — it inferred that from the framing. This is prompt design working with the model's training rather than against it. The model has seen thousands of governance documents, board reports, and accountability frameworks during pretraining. Your job is to activate the right reasoning pattern with precise instructions, not to explain domain concepts from scratch.

Retrieval-Augmented Generation: When the Model Needs Fresh Facts

Language models have a training cutoff — a date beyond which they have no knowledge. GPT-4o's cutoff is April 2024. Claude 3.5 Sonnet's is April 2024. This matters less than people assume for most professional tasks (strategic frameworks, writing quality, analytical reasoning don't expire), but it matters enormously for anything involving current events, recent pricing, live market data, or regulatory updates. The solution isn't to abandon these models — it's to use Retrieval-Augmented Generation, or RAG. In a RAG setup, the system retrieves relevant documents from an external source (a database, the web, your company's internal knowledge base) and injects them into the context window before the model generates its response.

You're already using RAG without knowing it. Perplexity AI runs a search query, retrieves current web pages, and feeds the content to a language model — that's RAG. ChatGPT's browsing mode does the same thing. Microsoft Copilot for Microsoft 365 retrieves your emails, documents, and calendar entries before answering questions. When you ask Copilot 'What did we agree in last week's client call?' it searches your Teams transcripts, pulls the relevant segments, and grounds the model's response in actual retrieved text. The model isn't 'remembering' your meeting — it's reading a retrieved document in real time. This distinction matters because it explains both why these tools are powerful and where they can fail (bad retrieval = bad answers, regardless of model quality).

Fine-Tuning vs. Prompting: Choosing the Right Tool for the Job

There are two ways to customize a language model for professional use: prompting and fine-tuning. Prompting — what you've been doing in ChatGPT — shapes the model's behavior through instructions in the context window. It's fast, free, and reversible. Fine-tuning involves taking a pretrained model and running additional training on a curated dataset of examples specific to your domain, style, or task. The result is a model that behaves differently at the weights level, not just the prompt level. A fine-tuned customer service model, for instance, doesn't need a lengthy system prompt explaining your brand voice — that behavior is baked into the parameters. OpenAI offers fine-tuning for GPT-4o mini starting at $0.003 per 1,000 training tokens, making it accessible for teams with sufficient data and a clear use case.

For most professionals, prompting handles 90% of use cases — and handles them faster, cheaper, and with more flexibility than fine-tuning. Fine-tuning makes sense when you have a high-volume, narrow, repeatable task where consistency is critical and you have hundreds of high-quality input-output examples to train on. It also makes sense when your prompts are getting unwieldy — if you're pasting a 2,000-word system prompt before every API call, fine-tuning that behavior into the model is more efficient. If you're a manager experimenting with ChatGPT to improve your team's output, prompting is your tool. If you're an analyst building a production pipeline that runs 10,000 document classifications per day, fine-tuning becomes worth the investment.

What This Means for Your Work on Monday

The mechanisms covered here — temperature sampling, context windows, RAG, and fine-tuning — translate directly into specific professional behaviors. When you're using ChatGPT for anything that requires consistency across team members (client-facing templates, compliance checklists, standard operating procedures), you should build a shared prompt library rather than expecting individuals to get consistent results from ad-hoc prompting. Variability isn't a user error — it's the model behaving as designed. A well-crafted, tested prompt stored in a shared Notion page or team wiki becomes an asset that reduces that variability systematically.

Context window awareness changes how you structure long AI work sessions. If you're doing a 90-minute research synthesis in Claude, the constraints you set in your first message may not survive to the end of the conversation. Experienced practitioners re-state critical constraints at key intervals — not because they're being redundant, but because they're managing context deliberately. A simple technique: at the midpoint of any long session, ask the model to summarize the constraints and objectives it's working within. If it misses something, you've caught context drift before it corrupts your output rather than after.

Understanding RAG fundamentally changes how you evaluate AI tool claims. When a vendor says their AI 'knows your company's data,' they mean their system retrieves from your documents and injects that content into a model's context. The quality of that retrieval pipeline — how documents are chunked, indexed, and ranked — determines whether the product actually works for your use case. Asking a vendor 'how does your retrieval work?' and 'what happens when retrieved documents conflict?' separates informed buyers from ones who discover the limitations six months into a contract. These questions don't require deep technical knowledge — just an understanding of the mechanism you now have.

How These Concepts Apply Differently by Role

• Managers: Use context window awareness to structure team AI workflows — break long projects into context-sized sessions with explicit handoffs, rather than expecting one marathon conversation to stay coherent.
• Analysts: Temperature variability means you should run high-stakes analytical prompts 2-3 times and compare outputs before committing to findings. Treat AI analysis like a survey sample — one response isn't the population.
• Marketers: RAG-powered tools like Perplexity give you current data that base ChatGPT can't access. Use Perplexity for market sizing, trend data, and competitor moves; use ChatGPT for drafting and ideation where freshness matters less.
• Consultants: The fine-tuning vs. prompting distinction matters when advising clients on AI investments. A client with a narrow, high-volume use case (invoice classification, support ticket routing) has a genuine fine-tuning case. A client wanting general productivity gains should start with prompt engineering and a good system prompt, not a custom model.
• Product managers: Context windows determine what's architecturally possible in AI-powered features. A feature that requires synthesizing a user's 18-month email history won't work in a 32K token model — understanding this prevents promising capabilities that the chosen model physically can't deliver.
• Operations leads: Consistency requirements favor lower-temperature settings and tightly constrained prompts. If you're rolling out AI for process documentation or SOP generation, test outputs at different temperature settings and standardize on the one that minimizes drift.

It's Monday morning. Your CEO forwards you an article about a competitor using AI to cut proposal turnaround time by 60%. She asks you to assess whether your team could do the same — by Thursday. You open ChatGPT, type a vague prompt, get a generic response, and feel like the tool isn't living up to the hype. The problem isn't the AI. The problem is that without understanding what's happening under the hood, you're essentially driving a Formula 1 car in first gear. Everything you've covered about tokens, training, context windows, and model behavior now becomes the diagnostic toolkit you use to answer your CEO's question with confidence.

Turning Technical Understanding Into Workplace Advantage

When you know that a model like GPT-4 predicts the next token based on patterns baked in during training, you stop asking it to 'know' things it can't know — like last quarter's sales figures or a client's internal strategy. Instead, you paste that data directly into the prompt, treating the context window as working memory. GPT-4 Turbo handles up to 128,000 tokens of context, which is roughly 90,000 words — enough to load an entire business report and interrogate it in natural language. Claude 3 Opus pushes to 200,000 tokens. Knowing these numbers tells you which tool to reach for when a task is document-heavy versus when you need fast, iterative brainstorming.

Training data cutoffs matter enormously in professional contexts. ChatGPT on GPT-4 has a knowledge cutoff of April 2023, meaning it has no awareness of regulatory changes, market shifts, or competitor moves that happened after that date. Perplexity AI solves this by combining a language model with live web search, making it the right tool for current-events research, recent case law, or up-to-date pricing data. Gemini 1.5 Pro, Google's flagship model, integrates with Search natively and supports a 1 million token context window in its experimental version — the largest commercially available as of mid-2024. Matching the tool to the task based on these constraints is what separates a professional AI user from someone who just 'tries ChatGPT.'

Choosing the Right Model for the Right Job

What Good Prompting Looks Like in Practice

The difference between those two prompts isn't creativity — it's structure. The informed prompt applies everything covered earlier: it gives the model a role, supplies context data directly into the window, specifies the output format, and sets a concrete objective. The model's job is pattern completion; your job is to make the pattern you want unmistakably clear. When you treat prompt construction as a professional discipline rather than a casual search query, your output quality shifts immediately and measurably. This is the operational insight that most AI training programs bury in chapter nine. Apply it in chapter one of your actual work.

What Changes at Work Based on Your Role

• Managers: Brief your team on context window limits so they stop expecting AI to 'remember' decisions from last week's session — set a shared prompt template library instead.
• Analysts: Use Perplexity Pro for live data research, then pipe structured findings into GPT-4 Turbo for synthesis — don't use a single tool for both jobs.
• Marketers: Understand that Midjourney and DALL-E generate images through diffusion models, not language models — prompting strategy differs; describe visual composition explicitly, not abstract concepts.
• Consultants: Match model choice to document length. Claude 3 Opus for reviewing 100-page RFPs; GPT-4 Turbo for iterative strategy drafting; Gemini for anything requiring current market data.
• All roles: Build a personal prompt library — save your best-performing prompts as templates the way you'd save Excel formulas. This compounds your productivity over time.

Key Takeaways

1. AI models predict the next token — they don't 'think' or 'know.' They complete patterns based on training data.
2. Context windows are finite working memory. Front-load every session with role, data, constraints, and format.
3. Training cutoffs make models blind to recent events. Use Perplexity or Gemini when currency matters.
4. Model choice is a professional decision, not a personal preference. Match capabilities to task requirements using real specs.
5. Prompt quality is the primary lever you control. Structured, specific prompts outperform vague ones by a measurable margin.
6. Hallucination is a structural feature of how these models work — always verify high-stakes outputs against primary sources.
7. Enterprise use requires data agreements. Never paste confidential client data into a consumer-tier AI product without checking your firm's policy.