Our verdict
A usable project with some maintenance caveats. This assessment is derived from GitHub's own repository metrics on 2026-10-12, not from hands-on testing.
Yes, AirLLM can run a 70B model on a 4GB GPU, but slowly. It keeps only one layer of the model on the GPU at a time, so the VRAM you need depends on the size of a layer, not the whole model. The README lists Llama 3.x 70B at full precision at about 4 GB, Llama 3.1 405B at about 8 GB and DeepSeek-V3 (671B) at about 12 GB. The price is speed, because the weights stream from disk for every token. The README gives no tokens-per-second figure, and users in open GitHub issues report very slow output: 28.6 seconds per token on a 3B model (issue 364), about 30 minutes for a 120-token reply from a 20B model on an RTX 3090 (issue 374), and 13.3 seconds per token on a 32B model (issue 330, which also measured the README’s compression speedup as slower on that machine). Use it to try a large model on cheap hardware, not to serve chat. It is free: pip install airllm. The project is written primarily in Jupyter Notebook, released under Apache-2.0, and has 42,081 stars and 4,335 forks on GitHub. 18 contributors have committed to it, and the most recent push was 2026-11-10. The latest tagged release is v4.0.0 (2026-09-05). Figures come from the GitHub API on 2026-11-10 and are refreshed weekly; the score below weighs adoption, maintenance, release discipline, contributor breadth, licence clarity and issue hygiene.
Reviewed by AI Review Rating from GitHub API data How we score
Repository
- Licence
- Apache-2.0
- Language
- Jupyter Notebook
- Last push
- 2026-11-10
Releases
-
v4.0.0AirLLM v4.0.0 - train Qwen3.8-Flash-Next under 6GB -
v3.3.0AirLLM v3.3.0 - Qwen3.8-Flash-Next on 5.95GB -
v3.2.0AirLLM v3.2.0 - Qwen3.8-27B on 3.33GB -
v3.1.0AirLLM v3.1.0 - Kimi K3 (2.8T) on a single card -
v3.0.1AirLLM v3.0.1 -
v3.0.0v3.0.0
Pros and cons
Pros
Cons
Pricing
- Free tierYes
- LicenceApache-2.0
- Self-hostYes, source is public
- Public price listNot published
AirLLM does not publish a web price list. For consumer apps that usually means pricing appears inside the product; for enterprise tools it means talking to sales. Why we do this
Visit github.comTrends
Price points are recorded when a price moves, not on a schedule, so a flat run means the price held. Release cadence comes from GitHub releases. A series appears only once we have watched it for long enough to have more than one observation.
Changelog
User reviews
No one has reviewed AirLLM here yet. Reviews are moderated before they appear, and our own score is editorial and separate from user ratings, though once reviews land they do lift it.
Used AirLLM? Review it.
Sign in to rate it. One account, one review per tool, so the score means something.
Requiring an account is how we keep this page free of drive-by spam. We never sell your address.
AirLLM alternatives
Transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning…
🥈 
LocalAI
LocalAI is the open-source AI engine. Run any model…
🥉 
vLLM
A high-throughput and memory-efficient inference and serving engine for…
Unsloth
Local UI to run and train LLMs and diffusion…
RTK
CLI proxy that reduces LLM token consumption by 60-90%…
Modal
Bring your own code, and run CPU, GPU, and…
Side by side
Articles on AirLLM
Covers AirLLM
Best Open Source AI Tools You Can Self-Host in 2026 (And What They Replace)
Short answer: the best open source AI tools to self-host in 2026 are Firecrawl for web scraping, Crawl4AI and ScrapeGraphAI…
Best Open Source AI Agent Frameworks in 2026: 19 Ranked and Checked on GitHub, with Licences and Team Costs
For a startup or small dev team: 19 agent framework repositories checked on GitHub on 8 October 2026. The frameworks…
Questions
Can AirLLM run a 70B model on a 4GB GPU?
Yes. AirLLM loads one layer at a time onto the GPU, so a Llama 3.x 70B at full precision needs about 4 GB of VRAM according to the project README, with no quantization, distillation or pruning. The cost is speed, covered below.
How slow is AirLLM?
Slow. The README does not publish a tokens-per-second number. Users in open GitHub issues report 28.6 seconds per token on a 3B model with Windows and an RTX 5060 Ti (issue 364), about 30 minutes for a 120-token answer from a 20B model on an RTX 3090 (issue 374), and 13.3 seconds per token on a 32B model (issue 330). Expect batch use, not chat.
Is AirLLM free?
Yes. It is a public repository under Apache-2.0 and installs with pip install airllm. There is no paid tier or web price list; the real costs are disk space, system RAM and your time.
Does the 3x compression speedup work?
The README says 4-bit or 8-bit block-wise compression (needs bitsandbytes) can speed up inference by up to 3x with almost ignorable accuracy loss. One user measured the opposite on a 32B model (issue 330: 13.3 s/token uncompressed, 50.5 s at 8-bit, 106.7 s at 4-bit), and that issue is still open, so benchmark on your own hardware.
How much disk space does AirLLM need?
A lot. On first run it splits the original model into per-layer files, and the README warns to leave enough room in the Hugging Face cache. Its FAQ says the MetadataIncompleteBuffer error most likely means the disk is full. The delete_original option removes the downloaded original to save about half. The README quotes about 360 GB of checkpoint disk for Qwen3.8-Flash-Next.
Which models does AirLLM support?
The README lists Llama 2, 3, 3.1, 3.3 and 4, Qwen 1 to 3.8, DeepSeek V2, V3 and R1, Mistral and Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, Yi and Kimi K3, loaded through a single AutoModel call. Version 4.0.0 (2026-09-05) added streamed LoRA training for Qwen3.8-Flash-Next under 6 GB.
How is AirLLM scored here?
AirLLM scores 6.8 out of 10. That comes from a desk review of public evidence, which is capped at 8.5 out of 10: public pricing, documentation, maintenance signals and how much we could actually verify. Affiliate relationships are excluded from scoring entirely. The full rubric is published on our methodology page.