Web Courses Bangkok Blog

If AI Works Offline, Where Does It Keep Everything It Knows?

Author: Carl Heaton
Carl is a consultant and design leader from Manchester, UK, with extensive experience in digital design, UX/UI, and online business. He brings practical, real-world insight shaped by years of leading design, product, and digital work. Learn more at carlheaton.work.
pexels-cookiecutter-17489150
Quick jump to topics
Sharing is caring

Someone asked a surprisingly good question online recently.

If you can download an AI model onto your laptop, disconnect the internet and still ask it questions about history, science, programming, films, people and thousands of other subjects…

where is all that knowledge actually stored?

There isn’t a hidden copy of Wikipedia inside the model folder. There isn’t a gigantic database containing neat rows for “Bangkok”, “Albert Einstein” and “how to centre a div”.

The answer is stranger than that.

Most of what the model appears to “know” is encoded across billions of numbers called weights.

And understanding what that means explains quite a lot about both the brilliance and the frustrating stupidity of modern AI.

The short answer: the knowledge is in the model you downloaded

When you download a local large language model, you normally download several pieces needed to run it.

The important part is the model weights.

Hugging Face, one of the main platforms used to distribute AI models, describes a model checkpoint as the weights belonging to a particular model architecture. Those weights can be downloaded and loaded directly from files on your computer.

These files can be large.

Very large models may even have their weights split across several files because one checkpoint is simply too big to handle conveniently as a single lump.

The gpt-oss-20b files on Hugging Face, with the weights split across three .safetensors files of about 4.8GB each
Screenshot: huggingface.co, October 2026

But here’s the important bit.

Those files don’t contain something like:

Paris = capital of France
Bangkok = capital of Thailand
CSS = used to style web pages

Instead, they contain huge collections of numerical values that influence how information moves through the neural network.

Not exactly intuitive.

So let’s rewind a little.

An AI model isn’t really looking things up

Imagine asking a normal database:

What is the capital of Japan?

The software can search for a stored record.

Japan → Tokyo.

An LLM works differently.

Modern language models are fundamentally prediction systems. Google describes autoregressive language models as systems that predict the next token using the tokens that came before it.

A token might be a word, part of a word, punctuation or another small unit of text.

Give the model:

The capital of Japan is…

and the model calculates which possible token should come next.

“Tokyo” will hopefully receive a very high probability.

Then it predicts another token.

And another.

And another.

That simple process, repeated extremely quickly by a very large neural network, is how a paragraph eventually appears on your screen.

This sounds almost disappointingly basic until you consider what the model had to learn for those predictions to become useful.

Training changes billions of tiny numbers

Before training, a language model isn’t sitting there patiently waiting to receive its first fact about Japan.

Its weights start in a largely unhelpful state.

During training, the model processes enormous quantities of text and repeatedly tries to predict missing or following tokens. When its predictions are poor, its internal weights are adjusted.

Again.

And again.

Millions and millions of adjustments.

Over time, patterns emerge.

Words develop relationships with other words. Concepts begin to interact. Grammar, writing styles, programming syntax, facts and associations become represented across the network.

There isn’t necessarily one weight containing:

“Water freezes at 0°C.”

The useful behaviour comes from the interaction between many parameters throughout the model.

That distinction matters.

The model isn’t a filing cabinet stuffed with facts.

It’s closer to an impossibly complicated mathematical landscape that has been shaped by everything it encountered during training.

Ask a question and your prompt travels through that landscape.

A slightly better analogy: imagine a musician

Think about an experienced pianist.

They may have listened to thousands of hours of music and spent years practising.

But if you opened their brain you wouldn’t find a folder called:

Jazz Chords

with 4,382 MP3 files inside it.

Their experience has changed how they recognise patterns.

They know what combinations sound right. They anticipate what tends to come next. They recognise structures they’ve encountered before and can create something new from them.

A language model is obviously not a human brain, but the comparison helps.

Training data shapes the model.

The training data itself does not have to remain sitting beside it every time you use it.

That’s why an AI model can still answer questions when your Wi-Fi disappears.

So what happens when you run AI locally?

Let’s say you install a local AI tool and download a model.

The model files sit on your SSD or hard drive.

When you start the model, its weights are loaded into the memory available on your machine. Depending on your setup, that might involve GPU memory, normal system RAM or even moving some model data between memory and disk.

Hugging Face’s large-model documentation describes exactly this kind of process: weights can be distributed across GPU, CPU memory and, when necessary, storage.

Tools such as Ollama have made this dramatically easier for normal users.

Some models are deliberately small enough to run on consumer hardware. Ollama has also spent considerable effort improving GPU memory management because available memory has such a large impact on whether a model can run comfortably on your machine.

Once the model is loaded locally, the calculations can happen locally too.

No Google search required.

No mysterious server containing the answer.

Your computer is running the model.

Then why are model files sometimes surprisingly small?

This is where things get particularly interesting.

The internet contains an absurd amount of text. Yet useful language models can occupy tens of gigabytes rather than requiring a hard drive containing the whole public web.

That does not mean somebody has simply compressed the internet into a clever ZIP file.

The model has learned statistical structure from its training.

There is also another technique involved called quantisation.

Model weights don’t always need to be stored at their original numerical precision. They can often be represented using fewer bits, significantly reducing how much memory the model requires.

For example, Ollama explains that the weights in OpenAI’s gpt-oss models use quantisation to reduce memory requirements, with the smaller 20B model able to run on machines with roughly 16GB of memory.

There can be a trade-off between size, speed and quality, but modern quantisation has become remarkably effective.

This is one reason local AI has moved so quickly.

Hardware stayed powerful.

Models got better.

And engineers became much cleverer about squeezing those models into the hardware people already own.

But here’s where people get confused: the model doesn’t know everything

Running locally proves something important.

An AI doesn’t need to search the internet every time it answers a question.

But the opposite conclusion is wrong too.

The model does not contain a perfect copy of everything used during training.

Ask it about something obscure enough and it may simply get it wrong.

Ask for something that happened yesterday and an offline model may have no idea.

Ask for a precise source and it may confidently invent one.

This happens partly because the model is generating likely language rather than retrieving a guaranteed factual record.

Its “knowledge” is fuzzy.

Sometimes remarkably accurate.

Sometimes spectacularly not.

This also explains why web search makes AI much better at some tasks

Modern AI systems often combine the language model with other tools.

Search is one.

Databases are another.

Company documents are another.

Suppose you ask:

What time is the next flight from Bangkok to London?

That answer should not come purely from model weights.

Flight information changes constantly.

A sensible AI system searches a live source, gives that information to the language model and then lets the model interpret it.

The same idea appears in a technique called Retrieval-Augmented Generation, usually shortened to RAG.

RAG lets a model retrieve relevant information from external documents or databases before producing its answer. AWS describes it as adding external information to the model’s context rather than depending entirely on knowledge captured during training.

You can even do this locally.

Ollama, for example, supports embedding models that can turn documents into numerical representations and use those representations to retrieve relevant material for an AI application.

Now your system has two different kinds of “knowledge”.

What the model learned during training.

And information it can retrieve when needed.

That combination is enormously useful.

What about AI memory?

Here’s another wrinkle.

Imagine you spend six months talking to your local AI.

Does the original model gradually learn everything about you?

Usually, no.

During ordinary inference, the fundamental model weights generally don’t change.

Your AI application may save conversations, summaries, preferences or other information in separate files or databases and provide that information to the model later.

That’s application memory.

It isn’t the same thing as retraining the model.

This explains why deleting a chat history and deleting a model are completely different things.

The model is one component.

The software wrapped around it can provide memory, tools, search, files, code execution and all sorts of additional abilities.

Once you understand that distinction, a lot of confusing AI terminology becomes much easier.

Model, memory, search and tools are four different things

It’s useful to separate them.

PartWhat it does
Model weightsHold the learned patterns and capabilities created during training
ContextGives the model the information available during the current interaction
Memory / databaseStores information so software can retrieve it again later
ToolsLet the AI do things such as search the web, run code or query another system

People tend to bundle all four together and call the whole thing “AI”.

That’s understandable from the outside.

Technically though, they are very different pieces.

And increasingly, building useful AI products is less about simply choosing the biggest model and more about combining these pieces sensibly.

Does this mean an offline AI is private?

Potentially, yes.

If the model genuinely runs locally and the application doesn’t send your prompts somewhere else, your text can stay on your own machine.

Ollama, for example, explicitly describes local processing as a way of keeping information on-device rather than sending it to third-party cloud services.

But don’t assume the word local guarantees privacy by itself.

A local model can still be connected to web search.

It can call cloud APIs.

Plugins can send data elsewhere.

Analytics software can communicate with external servers.

If privacy matters, check the complete application rather than simply asking whether the model was downloaded.

The weird part is that “knowledge” may be the wrong word

This is probably the most useful thing to take away.

We casually say AI models know things because that’s how the experience feels.

Ask about Python and it explains Python.

Ask about the French Revolution and it discusses 1789.

Ask for a recipe and you suddenly own twelve onions for some reason.

But underneath that very human-looking conversation sits an enormous mathematical system generating predictions from patterns encoded during training.

Those patterns can represent an astonishing amount of useful information.

They’re also imperfect.

That’s why AI can give a brilliant explanation at 10:04 and confidently talk rubbish at 10:05.

The same mechanism powers both.

And weirdly, understanding that makes AI more useful, not less impressive.

You stop treating it as an electronic oracle.

You start understanding what the machine is actually doing.

Wanna try it at home tonight? Here’s how I’d start

Reading about weights is one thing. Watching a model answer you with the Wi-Fi switched off is something else, and you can have it running on the laptop you already own before your coffee goes cold (well, nearly, most of the wait is a download bar).

1. Check how much memory you’ve got

Everything above comes back to memory, so I’d look before you download anything. On a PC it’s in Settings under System and About, where it says “Installed RAM”. On a Mac it’s About This Mac in the Apple menu. My rough rule is that the model file should fit inside that number with plenty left over for your browser and all the other stuff you have open. With 16GB you can try quite a lot. With 8GB I’d stick to the smallest models and close everything else first.

A close-up of a laptop memory stick lying on a dark desk
Photo: Franck V. / Unsplash

2. Install Ollama

There are a few apps that do this and the one I’d give a beginner is Ollama. You download it from ollama.com, install it like any other app, and you get a plain chat box where you pick a model and start typing. If you’d rather click than type model names, LM Studio does a similar job, but on a Mac it needs an M1 chip or newer and they say you want 16GB of memory.

The Download Ollama page, with the Windows install command and a Download manually link
Screenshot: ollama.com, October 2026

3. Download a small model first

We all want to grab the biggest model we can find. Don’t! Start small, see how it feels on your machine, and then go up a size. Open a terminal (PowerShell on a PC) and type:

ollama run gemma4:e2b

That’s the smallest Gemma 4, a download of a few gigabytes, and when it’s done you are chatting with it right there. If it answers at a pace you can live with, try plain ollama run gemma4 next, which is the bigger one. Got 16GB or more? The gpt-oss model I mentioned earlier is a 14GB download and I think it’s worth a go too.

The Gemma 4 page on Ollama listing each model with its download size, from 4.6GB for e2b up to 20GB for 31b, plus two cloud versions
Screenshot: ollama.com, October 2026

4. Turn the Wi-Fi off and ask it something

This is the fun bit and really the whole point of this post. Switch your Wi-Fi off, ask it to explain something you know well, and watch it answer anyway. Then ask it about something that happened last week. It will either tell you it doesn’t know or make something up with a completely straight face, and now you know why it does that.

A laptop on a wooden desk at night, lit by a desk lamp and a blue strip light
Photo: Darius Cotoi / Unsplash

5. Go and look at where it lives

Remember the question at the top? On a PC the models sit in C:\Users\your-name\.ollama\models, and on a Mac they’re in a hidden .ollama folder in your home folder. Have a look at how big that folder is. That’s it, that’s everything it “knows”, sitting on your drive next to your holiday photos. Type ollama ls to see what you’ve got and ollama rm gemma4:e2b to bin one when you want the space back.

6. Give it your own files

This is where it stops being a party trick and starts being useful. Drag a PDF or a text file into the chat and ask it about what’s inside. That’s the retrieval idea from earlier in its simplest form, and it all happens on your own laptop. If the answers only seem to cover the first few pages, turn up the context length in the settings. It eats more memory when you do, so on a small laptop you will feel it.

7. Watch out for “cloud” in the name

One catch, and I reckon it trips a lot of people up. Ollama also runs models on its own servers, and in the list they look almost the same as the local ones, just with “cloud” in the name, like gemma4:cloud. You have to sign in to use those and your prompts leave your machine. That’s handy when your laptop is too slow, but if privacy is the reason you’re doing this, stick to the models you’ve downloaded. Their FAQ shows you how to switch the cloud side off for good.

Do all of that once and the rest of this post will make a lot more sense to you, because you’ll have seen the weights, the memory and the gaps for yourself. Let me know how you get on!

Want to understand AI without becoming a machine-learning engineer?

You don’t need to learn matrix multiplication before AI becomes useful.

What matters is understanding enough of what’s happening underneath to know when to trust the model, when to provide context and when it needs access to another source entirely.

That’s a big part of what we teach in WCB’s Generative AI for Work course: practical use first, with enough understanding underneath it to make better decisions.

And once you’re ready to go beyond chat and connect AI to real business processes, our AI Automation & Agents course moves into tools, workflows, retrieval and AI systems that can actually do things rather than simply answer questions.

More great articles
There is more where this came from
Join our monthly newsletter packed with course dates, latest articles, free resources and job opportunities
  • This field is for validation purposes and should be left unchanged.
Promise to only send you useful interesting newsletters once a month.