The AI crash course.
Everything behind a production AI product, in the order it builds on itself, from a single token all the way to a team of agents. Loose plain English first, the exact terms second, and every idea shown running inside Bloort. This is the path I actually walked building it.
1 · The model, and its one big flaw
Everything starts here. A model is just a text predictor: show it some words and it guesses the next one. It is shockingly fluent, and it will also lie to your face. This first stretch covers what the raw thing actually is, and the flaw that every clever trick after it exists to fix.
Token
A token is one bite of text the model reads, usually a whole small word or a word-piece rather than a single letter. When you drop a URL into Bloort, the entire scraped page becomes thousands of these tokens before the model sees a thing.
A tokenizer chops text into tokens averaging four characters, so 'bookshop' splits into 'book' plus 'shop'. Both what goes in and what comes out are tokens, and every API bill is counted in them, input tokens plus output tokens.
Tokenizer & transformer
These two are easy to blur together, but they do different jobs. The tokenizer is the translator that turns your text into tokens and back. The transformer, by contrast, is the actual model: the big brain that does the thinking. One converts, the other reasons.
A tokenizer is a lookup step (text to token IDs and back) with no intelligence of its own. The transformer is the neural-network architecture the model is built from, the attention machine described below. So the flow runs from your text, to the tokens a tokenizer makes, to the transformer that processes them. In short, the transformer is the model, and the tokenizer just feeds it.
Model
The model is a giant pattern-matcher. Feed it the tokens so far, it guesses the most likely next token, sticks it on, and does it again. That repeat-guess loop is how a whole paragraph gets written.
A model is billions of tuned numbers called weights. Given the tokens so far it outputs a probability for every possible next token, samples one, appends it, and repeats. What it produces is whatever is plausible, never necessarily what is true. Hold that thought, because it is the whole reason RAG exists.
Parameters (weights)
Parameters are the model's knobs, the numbers it learned during training. People say 'a 70-billion-parameter model' the way you'd talk about engine size: it is how much brain there is, not how much it read.
Parameters are the model's weights, meaning its capacity or size, not how much data it trained on; that is a separate axis (training tokens). And more parameters is not automatically smarter: a well-trained small model can beat a bloated big one. Size is not quality.
Attention mechanism
Attention is the trick that made modern AI suddenly good. When the model picks the next word it looks back over everything said so far and decides which earlier words matter most right now. That is how it stays on topic instead of rambling.
For each token the model takes a weighted look at every other token and leans on the relevant ones; those weights are the attention. This mechanism is the core of the transformer, the 2017 'Attention is all you need' idea. Take 'the vein clinic on 5th, does it take walk-ins?': attention is what links 'it' back to the clinic.
Inference
Inference is just the model running live to answer you. Training happened once, long ago, in a lab; inference is every request after that, and it is what you pay for per call.
Training is the slow, one-time tuning of the weights. Inference is tokens in and next-token guesses out, one at a time, on every request. Every Bloort chat reply is one inference pass, plus the retrieval wrapped around it.
Context window
The context window is how much the model can hold in its head at once. Everything for one answer (your question, the instructions, the looked-up facts, the chat history) has to fit inside it. Run out of room and the oldest content falls off.
Measured in tokens (tens of thousands to millions depending on the model). This one limit is why Bloort chunks documents instead of pasting whole sites, and why long chats eventually get summarised: you are always budgeting a fixed window.
LLM
An LLM is a Large Language Model, the big, do-anything kind like GPT-4. Broad reasoner, handles weird questions well, but slower and pricier.
Tens to hundreds of billions of parameters. Strong general reasoning and edge-case handling; you pay for it in latency and cost. Great when the task is open-ended.
SLM
An SLM is a Small Language Model, with fewer parameters, so it is faster, cheaper, and can even run on a laptop. Narrower in range, but on a focused job it is often all you need. The word 'small' refers to size, not to being dumbed-down for one topic.
A few billion parameters (Phi, Llama-8B). Shines at classification, routing, and retrieval-answering. Feed it facts via RAG and it often beats a raw giant LLM on your own data, because the answer is handed to it rather than recalled from memory.
Hallucination
Here is the flaw everything after this fixes. The model's whole job is to sound coherent rather than to be correct, so when it does not know an answer it will invent one instead of stopping. Ask a raw model your shop's hours and it will confidently make them up. That confident fiction is a hallucination, and it is exactly why Bloort exists.
It predicts plausible tokens, not true ones, so fluency and truth come apart, and it stays just as confident when it is wrong. You do not prompt it away; you ground it in real facts (RAG) and check it (evals). The next two sections are those two fixes.
2 · The harness: making a wild model behave
A raw model is fluent and unreliable. The harness is all the machinery you build around it to make it do what you actually want. Bloort's whole chat route is one big harness, with the model as one line in the middle of it.
Harness
The harness is everything wrapped around the model: the loop that runs it, the tools it can call, the facts you feed it, the rules it must follow, the retries when it fails. The model is a rented commodity; the harness is where your actual engineering lives.
Bloort's chat route is a textbook harness: retrieval, a semantic cache (reuse an answer when a new question means the same thing), input guardrails, retry with timeout and fail-open, and rate limits, all wrapped around one gpt-4o-mini call. Same model everyone rents; the harness is the product.
Temperature
Temperature is the model's creativity dial, from 0 to about 1. Low means steady and repeatable, high means loose and varied. It is your main knob for how much you want it to improvise.
It is how randomly the next token is sampled. Bloort turns it down (about 0.35) when generating a business's system prompt, where stable and structured output matters, and up (0.7) for chat, where a little warmth reads better. For an agent deciding whether to fire a real phone call, you would pin it near zero.
3 · RAG: teaching it your facts
The fix for hallucination is to stop asking the model to remember your business and start handing it the relevant facts at answer time. RAG turns your text into geometry so the right passage can be found in an instant.
Embedding
An embedding turns a piece of text into a list of numbers that captures its meaning. Two passages about the same thing land on similar numbers even if they share no words, so 'when do you close' and 'opening hours' end up as neighbours.
A fixed-length vector (Bloort uses text-embedding-3-small, so 1536 numbers). Meaning becomes a coordinate, so similar meanings sit near each other in that space.
Vector / vector database
That list of numbers is really a point in space. A vector database stores millions of these points and finds the nearest ones instantly. Near in this space means near in meaning.
Bloort uses Pinecone. Your question is embedded into the same space, then a nearest-neighbour search returns the closest chunks by cosine similarity. Each client's data lives in its own namespace, so one bot can never read another's. That isolation is what makes it multi-tenant.
Chunk
You do not store whole pages, you store bite-sized passages called chunks, so retrieval hands the model just the relevant bit instead of a whole website. How you cut them matters a lot: chop too bluntly and a fact gets split in half and loses its meaning.
Bloort splits about 2000 characters with roughly 150 characters of overlap between neighbours (a sliding window). The overlap is the point: a fact that straddles a boundary still survives whole in at least one chunk, instead of being sliced across two and lost to both.
RAG
RAG, short for Retrieval-Augmented Generation, is the whole move: look up the relevant passages first, then let the model answer using them. That is how a Bloort bot talks about your business instead of guessing. Retrieval does not answer the question; it fetches the facts, and the model writes the answer from those facts.
Embed the question, retrieve the top chunks from Pinecone, paste them into the prompt as context, then generate. Grounding the answer in your real documents rather than the model's fuzzy memory is what kills most hallucinations. Note the two distinct jobs: the retriever finds the facts, the model writes the answer.
Prompt
The prompt is the entire bundle handed to the model for one answer, not just your question. Getting this bundle right usually matters more than reaching for a bigger, pricier model.
Bloort assembles, fresh every turn: system rules, the retrieved chunks, a strict language rule, the last ten or so messages, and your question. Assembling that well is called context engineering, and it is most of the quality.
Hybrid RAG
Hybrid search runs two searches at once: one for meaning, one for exact words. The meaning search is great until someone types a product code, a price, or a rare name, the exact stuff embeddings blur. The keyword search catches exactly those.
Dense retrieval (embeddings, for meaning) plus sparse retrieval (BM25, for exact-word or lexical matching), fused together. BM25 is lexical, the opposite of semantic, and that is exactly why you add it: to catch the literal tokens the meaning-search blurs over, like codes and prices. Bloort measured hybrid against plain dense and it lost on our clean corpus, so we did not ship it. Measure, do not assume.
≈ “open until”
“9pm”, “$40”
Reranking
Reranking is a second pass that re-sorts the first batch of results, putting the truly best passage on top. The cheap first search casts a wide net; the reranker is the picky judge that reorders the catch.
A cross-encoder reads the question and each candidate chunk together (slower, sharper than the initial vector search) and reorders them. It is the standard next upgrade when your right answer is being retrieved but ranked too low.
Graph RAG
Retrieval that follows relationships between facts, not just nearness in meaning. Useful when the answer is spread across several linked facts you have to hop between.
Builds a knowledge graph (entities plus typed edges) and walks it, which suits multi-hop questions where 'who manages the doctor that treated me' needs two connected facts, not one lookup.
Fine-tuning vs RAG vs prompting
Three ways to make a model 'know' your stuff, easiest first. Prompting just tells it in the instructions. RAG looks the facts up live and hands them over. Fine-tuning actually retrains the model's weights on your data. Reach for the cheapest one that works, and usually you do not need fine-tuning.
With prompting the knowledge lives in-context and nothing permanent changes. RAG injects fresh or private knowledge at answer time, which is best when facts change or must be cited. Fine-tuning updates the weights to bake in a behaviour or style, best for a consistent format or tone rather than for facts. Bloort is pure RAG, because the facts change per client and per site edit, so retraining would be insane.
Palette extraction (k-means)
A bonus of how Bloort reuses the same 'group similar things' idea elsewhere: it auto-picks your brand colours from a shot of your site, so the widget matches your look with zero setup.
Take a screenshot, then cluster the pixels by colour with k-means: it gathers similar colours into a few groups, and each group's average becomes a brand colour. That group-the-similar-points move is everywhere in ML, a cousin of what the vector search is doing.
4 · Evals: how you actually know it works
Anyone can demo a RAG bot once. Evals are how you prove it retrieves and answers correctly, on every change, instead of hoping. This is the skill that separates shipped from demo.
Eval
An eval is an automated test that checks whether the AI actually got it right, not whether it merely sounded confident. It is how you catch a hallucination before a customer does. Unlike a normal unit test, one run is not pass or fail; you run a whole set and read a score, because the model answers a little differently each time.
A fixed set of questions, each with a known-correct source or answer, scored automatically over the entire set (an 88% pass, not a green tick). Two things a unit test cannot do: it grades fuzzy output (through keyword checks or an LLM judge), and it catches drift, meaning the model or your users changing under you while your code stays still. The reusable rig that runs the set and blocks a bad merge in CI is the eval harness.
Golden dataset / fixtures
The fixtures are the question set itself, the answer key. An eval is only as honest as these, so they have to come from your real, indexed data, not made-up questions.
Versioned question-to-expected-source pairs written from content you have actually indexed. Make them up and you measure your imagination, since a question whose answer is not in the corpus scores zero even when retrieval is perfect. 'Versioned' means a score change is provably your pipeline's doing, not the test quietly shifting underneath.
Hit rate
The most forgiving score: did the right info show up anywhere in the chunks we pulled? Yes or no.
Keyword coverage over the top-k results pooled together. Easy to read and recall-flavoured, but blind to ordering, which is the exact gap MRR fills.
Recall@k
Did a correct passage make it into the top k results? A smaller k is a stricter test.
Recall@1 is the hard one (was the very first chunk right?), recall@3 is gentler (anywhere in the top three). Watching both tells you how good the ordering is, not just whether the fact was found at all.
Precision@k
The flip side of recall: of the chunks we pulled, how many were actually useful versus junk? High precision means we are not padding the prompt with noise.
Relevant results divided by retrieved results in the top k. It matters because junk chunks cost tokens and can distract the model. Where recall asks whether we got the good stuff, precision asks how much bad stuff came with it, and you want both high.
MRR
MRR rewards putting the first correct answer right at the top, not just somewhere in the pile. A 1.0 means it was ranked first, 0.5 means it was usually second.
Mean Reciprocal Rank: 1 divided by the rank of the first relevant chunk, averaged over all questions. Rank 1 scores 1.0, rank 4 scores 0.25. These three (hit rate, recall@k, MRR) are classic information-retrieval metrics, separate from Ragas, which is a framework that adds generation-quality scores like faithfulness. Bloort built its own harness rather than pulling in Ragas.
LLM as judge
Sometimes there is no exact right answer to check against, as with 'is this welcome message any good?'. So you use another model to grade the output. An LLM judging an LLM.
Great for fuzzy quality and for verifier steps (one model produces, a different one checks). Watch its known biases: it favours the answer shown first (position), longer answers (verbosity), its own family's style (self-preference), and whatever the prompt implies is right (sycophancy). Guard against this by shuffling order and using a different model to judge than the one being judged.
Trajectory scoring
For agents you grade the whole path of decisions, not just the final sentence. A right answer reached the wrong way still fails, because next time the wrong way gives a wrong answer.
Per scenario you check: did it call the tools it should, skip the ones it should not, finish cleanly, stay under budget, and pass sane arguments? Bloort's front-desk agent went from 83% to about 100% on its scenario suite once eval caught one hallucinated tool argument. That is the loop: the agent cannot check itself, so the eval checks the agent.
5 · Workflows vs agents: the real fork
Same tools, opposite control. In a workflow you wire the steps; in an agent the model picks them. Knowing which to reach for is most of the job, and reaching for an agent too early is the classic mistake.
Workflow
A workflow is a fixed, pre-wired path that runs the same steps every time, nothing decided on the fly. Bloort's n8n front desk is one: a hot lead is detected, so a WhatsApp alert fires. Same trigger, same action, always. Predictable and easy to debug.
Deterministic automation (an n8n graph, a cron job). You own the branches in code; the model, if used at all, only fills a fuzzy slot. Reach for this whenever the steps are known in advance, which is most of the time, and it beats an agent on cost and reliability.
Tool
A tool is something an agent can do besides talk: search the docs, check a calendar, send a WhatsApp, place a call. Tools are what turn answers into actions.
A typed function the model is allowed to call. Anything that touches the real world should default to dry-run: Bloort's WhatsApp and VAPI tools build the exact message or call but do not fire it until you explicitly flip them live. That dry-run gate is how you stay safe when the action is irreversible.
Native function calling
This is how tools actually get called today. The old way, back in 2022, was making the model write out 'Thought / Action' text that you parsed with fragile string-matching. Now the model returns a clean, structured tool call through the API. Same idea, far more reliable.
The model emits a JSON tool call (name and arguments); your harness runs it and feeds the result back as the next message. The reason-act loop is identical to old ReAct, and only the mechanism changed, from parsed strings to structured calls. If a course still teaches 'Thought:/Action:' parsing, it is showing you a museum piece.
Agent
An agent is a model handed tools and the freedom to decide when to use them. Less a chatbot, more a junior assistant that takes actions. The one thing that makes it an agent is that the model, not your code, chooses the path.
You wrap the model in a loop with a set of allowed tools; it calls one, you feed the result back, and repeat until it has enough to answer. Whether it is safe in production comes down to one question: can the environment tell it when it is wrong? Code can, because tests fail. 'Is this lead hot?' cannot, which is why Bloort's chat stays a plain RAG call and the agent version sits unused.
ReAct
ReAct means Reason then Act. Order-status example: the agent reasons 'I need their order id', asks for it, reasons again ('now I check the orders table'), calls it, sees the order, decides it has enough, and answers. Those thinking steps are the trace; the whole path is the trajectory. It stops when it is done, or when it hits a limit.
The loop behind most agents. Bloort's version has real exit conditions: it answers when done, but also bails on a step ceiling (max 6 loops), a wall-clock timeout (25s), or a cost ceiling ($0.05), with a safe fallback reply. Those guardrails are what keep a reasoning loop from spinning or burning money forever. It is the while-loop, for AI.
6 · Skills, MCP & multi-agent
Once one agent works, you compose: give it packaged abilities, a standard way to plug in tools, and teammates it can delegate to. This is the top of the ladder, and mostly overkill until you genuinely need it.
Skills
A skill is a packaged ability you can hand an agent: a bundle of instructions (and sometimes little scripts and files) that it loads only when the job calls for it. Not just a prompt, but a reusable module. Coding assistants like Claude Code expose commands such as /review and /qa exactly this way, as named abilities the agent pulls in on demand.
More than a system prompt: a folder of instructions plus optional tools and resources, pulled in on-demand when the task matches. This is called progressive disclosure, where the agent does not carry every instruction at all times but loads the relevant skill just in time. It is how you extend what an agent can do without bloating its context window.
MCP
MCP, the Model Context Protocol, is a standard plug so any model can talk to any tool server. Think USB-C, but for AI tools: build to the standard once and everything fits.
An open client/server spec: the server advertises its tools (and resources and prompts), and the client calls them the same way every time. Bloort's calendar check_availability runs over MCP, so we could swap the calendar provider for another without touching the agent at all. That decoupling is the whole value: write once, plug in anything.
Multi-agent
Multiple agents working as a delegated team, each with its own tools and job. The insurance example: an underwriter agent, a fact-checker, an auditor, each narrow and good at one thing, passing work along.
The point is not 'more brains'. It is decomposing a fuzzy task into narrow, checkable steps, and letting a separate agent verify the one before it (the fact-checker exists to give the generator a truth signal it lacks). Reality check: in production most 'multi-agent' systems are a fixed workflow of LLM nodes wired by code, not agents freely chatting, because that is what stays auditable.
Orchestrator
The orchestrator is the manager of the team: it breaks the big job into pieces, hands each to the right agent, and stitches the results back together.
In the orchestrator-workers pattern a lead decomposes the task, delegates to workers, and synthesises their output. Key production note: the orchestrator is very often plain code (a state machine) rather than another LLM, because in high-stakes work you want the routing predictable and the humans signing the decisions the agents only drafted.
7 · Speed, latency & cost
Two systems can return the same answer and feel completely different, because one takes 200 milliseconds and the other takes eight seconds. Here is where the time actually goes, and how you get it back.
Where the time goes
Unlike a sorting algorithm, a RAG answer's latency is not one number, it is a sum of stages stacked end to end. The surprise for most engineers: the vector search is the cheap part, and the model writing the answer is the expensive part, by a wide margin.
Optimise the biggest stage first (Amdahl's law). Shaving a 10ms search while generation takes two seconds is wasted effort. Generation time scales with the number of output tokens, so a shorter answer is a faster answer.
| Stage | Typical time | Notes |
|---|---|---|
| Embed the question | 20 to 100 ms | one small model call plus network |
| Vector search (ANN) | 5 to 50 ms | fast and sublinear, rarely the problem |
| Rerank (optional) | 50 to 300 ms | runs a model on each candidate |
| LLM generation | 300 ms to several sec | the elephant, 10 to 100x the rest |
In RAG, retrieval is cheap and generation dominates.
Retrieval: O(N) vs O(log N)
This is the part that behaves like an algorithms question. Exact search compares your query against every stored vector, which is linear in the size of the corpus. Approximate search walks a navigable graph instead, closer to logarithmic, so it stays fast even over tens of millions of vectors.
Production vector databases use approximate nearest neighbour (HNSW or IVF) and accept about one percent recall loss for a hundred-times speedup. Exact search is your linear scan; HNSW is your balanced-tree lookup. Keyword search (BM25) rides a decades-old inverted index and is often faster than dense, which is why hybrid adds latency: you run both.
| Strategy | Query cost | Speed | Accuracy |
|---|---|---|---|
| Flat / brute-force | O(N·d) | slow at scale | exact |
| HNSW (ANN graph) | ~O(log N) | 5 to 20 ms | 95 to 99% recall |
| IVF (ANN clusters) | sublinear | fast | tunable |
| BM25 (keyword) | O(terms · postings) | often fastest | exact on words |
| Hybrid (dense + sparse) | max of the two | slower | highest |
| + Cross-encoder rerank | O(k) model passes | +50 to 300 ms | precision boost |
N = corpus size, d = vector dimensions, k = candidates reranked.
Making it faster
Most wins come from three moves: cache answers by meaning so repeats skip the whole pipeline, hand the model fewer and better chunks so it reads less, and stream the reply so it feels instant even while the rest is still coming.
Streaming does not cut total time, it cuts perceived time: a first token at 300ms reads as fast even if the full answer takes five seconds. Pre-filtering by namespace or metadata shrinks N before the search even runs, which is exactly what Bloort's per-bot namespace does.
| Lever | What it cuts | Typical win |
|---|---|---|
| Semantic cache | embed + search + generate | 2 s down to ~50 ms on a hit |
| Stream the answer | perceived wait | feels instant, same total |
| Cap output tokens | generation time | roughly linear in length |
| Smaller / faster model | per-call time and cost | route easy queries to it |
| Rerank to fewer chunks | generation input | 5 great chunks beat 20 okay ones |
| Metadata pre-filter | search space (N) | search one tenant, not the index |
Agent latency: one second to infinity
Agents break the tidy pipeline model. Their latency is steps multiplied by per-step cost, and the step count is not fixed. A simple task finishes in one step, a hard one takes six, a confused one loops until something stops it. Each step also gets slower, because every tool result is added to the context the model must re-read.
Agent steps are serial, each depends on the last observation, so you usually cannot parallelise the loop the way you can a fixed pipeline. You do not optimise the infinity away, you bound it. Bloort's react-agent caps a step ceiling (6 loops), a wall-clock timeout (25s), and a cost ceiling ($0.05), which turns an open loop into a bounded worst case.
| Fixed RAG pipeline | Agent loop | |
|---|---|---|
| Shape | sum of known stages | steps × per-step cost |
| Bounded? | yes, predictable | no, until you add ceilings |
| Dominated by | generation | serial LLM round-trips |
| You fix it by | optimising the big stage | bounding steps, time, cost |