Janas: Efficient Inference for Large Language Models Exceeding RAM Capacity

Janas enables efficient inference of large language models (LLMs) exceeding system RAM by streaming model experts from SSD, achieving up to 37 tokens per second on a 32GB laptop. It outperforms llama.cpp in speed for multiple models while supporting pure C implementation with CPU/GPU acceleration.

Comments

Read original article