Unit 1: How LLMs work, for defenders · Lesson 1 of 5 · 18 min
Text becomes tokens
Cut text into tokens the way a model does and measure what a prompt costs.
mission
- You are
- You are the new AI security engineer at Fernway Travel.
- Situation
- Fernway runs Fernbot, a support assistant built on a language model. Your first job is to understand what the model actually receives.
- Goal
- Measure the app's prompts in tokens, the unit every limit, bill and attack is counted in.
- Hand in
- The token size of the system prompt and of each help article.
The tokenizer as a type cutting mill
Tokenize a customer question
Models read tokens, not words. Short common words are one token; rare words and long numbers are cut into pieces.
tok "Where is my booking FW-20417?"
Where | is | my | book | ##ing | FW | - | 204 | 17 | ?
10 tokens
you should see The booking reference splits into FW, -, 204 and 17. Four tokens for one ID.
Tokenize a policy sentence
Long words split into pieces. A filter that matches whole words sees text; the model sees tokens.
tok "Sports equipment needs a special baggage booking"
Sports | equi | ##pmen | ##t | needs | a | spec | ##ial | bagg | ##age | book | ##ing
12 tokens
you should see Twelve tokens for seven words.
defender tip Real tokenizers differ by model. Use the one for your model when you budget prompts.
Measure the system prompt
The system prompt is sent with every request, so its size is paid on every single call.
tok -f app/system_prompt.txt
73 tokens in app/system_prompt.txt
Measure the knowledge base
Every retrieved article spends tokens from the same budget as the instructions.
for f in app/docs/*.md; do printf '%-28s' "$f"; tok -c "$(cat "$f")"; done
app/docs/baggage.md 87
app/docs/changes.md 58
app/docs/documents.md 60
app/docs/loyalty.md 48
app/docs/partner-offers.md 40
app/docs/refunds.md 64
app/docs/supplier-faq.md 86
you should see baggage.md is the largest, with the partner FAQ close behind.
defender tip When a document is unusually large or unusually placed, read it. Size is a cheap first signal.
Compare word forms
The same idea can cost a different number of tokens, which is also why exact word filters miss variants.
tok "Refund refunds refunded"
Refund | refu | ##nds | refu | ##nded
5 tokens
What a defender takes away
- 1Everything a model reads, writes, costs and forgets is measured in tokens.
- 2The system prompt is resent on every call, so keep it short and free of secrets.
- 3Filters that match words do not see what the model sees.
Questions
0 of 3 solved
Answer each one from what you find on your lab machine.