How to run a free offline AI model on your phone, laptop, or Raspberry Pi 4 (253 MB GGUF guide)

Why most local AI models crash on a 4 GB phone or Raspberry Pi

When people search for a free local AI for phone or try to run an offline LLM on a 4 GB Raspberry Pi 4, they usually download a 7B or 3B parameter model first. Even quantized to 4-bit, a 3B model takes ~2.0 GB of disk space and over 2.6 GB of active RAM once the KV cache expands. On Android, iOS, or a 4 GB Raspberry Pi 4 running background services, the operating system kills the process as soon as the context window fills up.

That is why we built and open-sourced [Tot-360M](/tot-360m/) (345,359,296 parameters, community-funded, Apache-2.0 license). At 253 MB in `Q4_K_M` (or 369 MB in `Q8_0`), the entire model plus a 4,096-token context window fits in under 430 MB of RAM.

Here is how to run it 100% offline on your phone, laptop, or Raspberry Pi 4.

1. Which GGUF file size should you download?

All checkpoints from Run 1 through SFT v2 (1.87B tokens, 4,096 context) are published in three GGUF precisions on Hugging Face (`verycoololiver/Tot-360M-Family`):

• `tot_360m_sft_v2_q4_k_m.gguf` (241 MiB): Best for Android phones, iPhones, and 2 GB to 4 GB single-board computers. Fastest token generation with lowest battery and RAM usage.
• `tot_360m_sft_v2_q8_0.gguf` (351 MiB): Best balance for 6 GB+ phones, Raspberry Pi 4/5, and everyday laptops. Near-lossless 8-bit weights.
• `tot_360m_f16.gguf` (660 MiB): Unquantized 16-bit weights for desktop CPUs or HTML5/Python code generation where exact precision matters.

2. Running free offline AI on Android and iPhone (PocketPal AI & ChatterUI)

Because Tot-360M exports to the standard Qwen3 GGUF tensor layout (Grouped-Query Attention 15Q/5KV with QK-Norm), mobile llama.cpp frontends load it out of the box with no custom plugins:

1. Install PocketPal AI (available free on Google Play and the iOS App Store) or ChatterUI (Android APK). 2. Open Models -> Add from Hugging Face and search for verycoololiver/Tot-360M-Family (or download `tot_360m_sft_v2_q4_k_m.gguf` directly to your phone and choose Add Local File). 3. In the model settings, set Context Length (`n_ctx`) to 4096, Temperature to 0.3, and Repeat Penalty to 1.08. 4. Turn on Airplane Mode and test a prompt in English or Korean. Everything runs locally on your phone's ARM CPU with zero cloud API calls.

3. Running on a 4 GB Raspberry Pi 4 or Pi 5

We host our own multi-model testbed directly on a Raspberry Pi 4 Model B (4 GB RAM) alongside a Node.js multiplayer game server. To get maximum speed on ARM64 Cortex-A72 (Pi 4) or Cortex-A76 (Pi 5):

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF
cmake --build build --config Release --target llama-cli llama-server -j4

Then run llama-cli or llama-server using 4 CPU threads:

./build/bin/llama-cli -m tot_360m_q4_k_m.gguf -c 4096 -t 4 --temp 0.3 --repeat-penalty 1.08 -p "<|im_start|>user
Write a complete playable HTML5 canvas Snake game in a single file.<|im_end|>
<|im_start|>assistant
"

If you are hosting multiple checkpoints on a 4 GB Raspberry Pi 4 (for example, comparing Tot-360M 1.66B, Irish-360M 1.57B, and Voy-360M 1.43B), keep all the .gguf files on disk and configure your wrapper server to keep only 1 model loaded in RAM at a time (MAX_HOT_MODELS=1). On our Pi 4, idle system memory stays around 432 MiB / 3.7 GiB, leaving over 3 GB of RAM free.

4. Running on Windows, macOS, or Linux laptops (LM Studio & Tot)

On any Windows, Mac, or Linux laptop without a dedicated GPU:

• LM Studio / Ollama / llama.cpp: Drop tot_360m_q8_0.gguf (369 MB) or tot_360m_f16.gguf (692 MB) into your models folder. On a standard laptop CPU, Tot-360M streams at 35+ tokens/second.
• Tot Desktop Companion (Windows): Download Tot for Windows if you want a local-first desktop coding workspace with automatic .tot-trash file backups and approval-gated PowerShell commands.

All weights, training logs, and loss curves are open and free on the [Tot-360M Specification Page](/tot-360m/).