MLC Models¶
Available Models¶
| Model ID | Quantization | Link |
|---|---|---|
| Llama-3-8B-Instruct | q0f16 | HuggingFace |
| Llama-3-8B-Instruct | q3f16_1 | HuggingFace |
| Llama-3-8B-Instruct | q4f16_1 | HuggingFace |
| Llama-3-8B-Instruct | q4f32_1 | HuggingFace |
| Llama-3.1-70B-Instruct | q0f16 | HuggingFace |
| Llama-3.1-70B-Instruct | q3f16_1 | HuggingFace |
| Llama-3.1-70B-Instruct | q4f16_1 | HuggingFace |
| Llama-3.1-70B-Instruct | q4f32_1 | HuggingFace |
| Llama-3.1-8B | q0f16 | HuggingFace |
| Llama-3.1-8B | q4f16_1 | HuggingFace |
| Llama-3.1-8B | q4f32_1 | HuggingFace |
| Llama-3.1-8B-Instruct | q0f16 | HuggingFace |
| Llama-3.1-8B-Instruct | q3f16_0 | HuggingFace |
| Llama-3.1-8B-Instruct | q3f16_1 | HuggingFace |
| Llama-3.1-8B-Instruct | q4f16_1 | HuggingFace |
| Llama-3.1-8B-Instruct | q4f32_1 | HuggingFace |
| Llama-3.2-1B-Instruct | q0f16 | HuggingFace |
| Llama-3.2-1B-Instruct | q0f32 | HuggingFace |
| Llama-3.2-1B-Instruct | q4f16_0 | HuggingFace |
| Llama-3.2-1B-Instruct | q4f16_1 | HuggingFace |
| Llama-3.2-1B-Instruct | q4f32_1 | HuggingFace |
| Llama-3.2-3B-Instruct | q0f16 | HuggingFace |
| Llama-3.2-3B-Instruct | q0f32 | HuggingFace |
| Llama-3.2-3B-Instruct | q4f16_0 | HuggingFace |
| Llama-3.2-3B-Instruct | q4f16_1 | HuggingFace |
| Llama-3.2-3B-Instruct | q4f32_1 | HuggingFace |
| Hermes-2-Pro-Llama-3-8B | q0f16 | HuggingFace |
| Hermes-2-Pro-Llama-3-8B | q3f16_1 | HuggingFace |
| Hermes-2-Pro-Llama-3-8B | q4f16_1 | HuggingFace |
| Hermes-2-Pro-Llama-3-8B | q4f32_1 | HuggingFace |
| Hermes-2-Theta-Llama-3-70B | q0f16 | HuggingFace |
| Hermes-2-Theta-Llama-3-70B | q3f16_1 | HuggingFace |
| Hermes-2-Theta-Llama-3-70B | q4f16_1 | HuggingFace |
| Hermes-2-Theta-Llama-3-70B | q4f32_1 | HuggingFace |
| Hermes-2-Theta-Llama-3-8B | q0f16 | HuggingFace |
| Hermes-2-Theta-Llama-3-8B | q3f16_1 | HuggingFace |
| Hermes-2-Theta-Llama-3-8B | q4f16_1 | HuggingFace |
| Hermes-2-Theta-Llama-3-8B | q4f32_1 | HuggingFace |
| Hermes-3-Llama-3.1-8B | q0f16 | HuggingFace |
| Hermes-3-Llama-3.1-8B | q3f16_1 | HuggingFace |
| Hermes-3-Llama-3.1-8B | q4f16_1 | HuggingFace |
| Hermes-3-Llama-3.1-8B | q4f32_1 | HuggingFace |
| Phi-3-mini-128k-instruct | q0f16 | HuggingFace |
| Phi-3-mini-128k-instruct | q4f16_1 | HuggingFace |
| Phi-3-mini-128k-instruct | q4f32_1 | HuggingFace |
| Phi-3.5-mini-instruct | q0f16 | HuggingFace |
| Phi-3.5-mini-instruct | q4f16_0 | HuggingFace |
| Phi-3.5-mini-instruct | q4f16_1 | HuggingFace |
| Phi-3.5-mini-instruct | q4f32_1 | HuggingFace |
| Phi-3.5-vision-instruct | q0f16 | HuggingFace |
| Phi-3.5-vision-instruct | q3f16_1 | HuggingFace |
| Phi-3.5-vision-instruct | q4f16_1 | HuggingFace |
| Phi-3.5-vision-instruct | q4f32_1 | HuggingFace |
| Mistral-7B-Instruct-v0.3 | q0f16 | HuggingFace |
| Mistral-7B-Instruct-v0.3 | q3f16_1 | HuggingFace |
| Mistral-7B-Instruct-v0.3 | q4f16_0 | HuggingFace |
| Mistral-7B-Instruct-v0.3 | q4f16_1 | HuggingFace |
| Mistral-7B-Instruct-v0.3 | q4f32_1 | HuggingFace |
| Qwen1.5-0.5B-Chat | q0f16 | HuggingFace |
| Qwen1.5-0.5B-Chat | q4f16_1 | HuggingFace |
| Qwen1.5-0.5B-Chat | q4f32_1 | HuggingFace |
| Qwen1.5-1.8B-Chat | q0f16 | HuggingFace |
| Qwen1.5-1.8B-Chat | q4f16_1 | HuggingFace |
| Qwen1.5-1.8B-Chat | q4f32_1 | HuggingFace |
| Qwen2-0.5B-Instruct | q0f16 | HuggingFace |
| Qwen2-0.5B-Instruct | q0f32 | HuggingFace |
| Qwen2-0.5B-Instruct | q4f16_0 | HuggingFace |
| Qwen2-0.5B-Instruct | q4f16_1 | HuggingFace |
| Qwen2-0.5B-Instruct | q4f32_1 | HuggingFace |
| Qwen2-1.5B-Instruct | q0f16 | HuggingFace |
| Qwen2-1.5B-Instruct | q4f16_0 | HuggingFace |
| Qwen2-1.5B-Instruct | q4f16_1 | HuggingFace |
| Qwen2-1.5B-Instruct | q4f32_1 | HuggingFace |
| Qwen2-72B-Instruct | q0f16 | HuggingFace |
| Qwen2-72B-Instruct | q4f16_1 | HuggingFace |
| Qwen2-72B-Instruct | q4f32_1 | HuggingFace |
| Qwen2-7B-Instruct | q0f16 | HuggingFace |
| Qwen2-7B-Instruct | q4f16_1 | HuggingFace |
| Qwen2-7B-Instruct | q4f32_1 | HuggingFace |
| Qwen2-Math-1.5B-Instruct | q0f16 | HuggingFace |
| Qwen2-Math-1.5B-Instruct | q4f16_1 | HuggingFace |
| Qwen2-Math-1.5B-Instruct | q4f32_1 | HuggingFace |
| Qwen2-Math-72B-Instruct | q0f16 | HuggingFace |
| Qwen2-Math-72B-Instruct | q4f16_1 | HuggingFace |
| Qwen2-Math-72B-Instruct | q4f32_1 | HuggingFace |
| Qwen2-Math-7B-Instruct | q0f16 | HuggingFace |
| Qwen2-Math-7B-Instruct | q4f16_1 | HuggingFace |
| Qwen2-Math-7B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-0.5B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-0.5B-Instruct | q0f32 | HuggingFace |
| Qwen2.5-0.5B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-0.5B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-1.5B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-1.5B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-1.5B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-14B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-14B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-14B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-32B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-32B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-32B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-3B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-3B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-3B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-72B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-72B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-72B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-7B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-7B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-7B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-Coder-0.5B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-Coder-0.5B-Instruct | q0f32 | HuggingFace |
| Qwen2.5-Coder-0.5B-Instruct | q4f16_0 | HuggingFace |
| Qwen2.5-Coder-0.5B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-Coder-0.5B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-Coder-1.5B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-Coder-1.5B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-Coder-1.5B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-Coder-14B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-Coder-14B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-Coder-14B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-Coder-32B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-Coder-32B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-Coder-32B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-Coder-3B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-Coder-3B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-Coder-3B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-Coder-7B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-Coder-7B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-Coder-7B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-Math-1.5B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-Math-1.5B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-Math-1.5B-Instruct | q4f32_1 | HuggingFace |
| Qwen2.5-Math-72B-Instruct | q0f16 | HuggingFace |
| Qwen2.5-Math-72B-Instruct | q4f16_1 | HuggingFace |
| Qwen2.5-Math-72B-Instruct | q4f32_1 | HuggingFace |
| DeepSeek-V2-Lite-Chat | q0f16 | HuggingFace |
| DeepSeek-V2-Lite-Chat | q4f16_1 | HuggingFace |
| DeepSeek-V2-Lite-Chat | q4f32_1 | HuggingFace |
| Mixtral-8x7B-Instruct-v0.1 | q0f16 | HuggingFace |
| Mixtral-8x7B-Instruct-v0.1 | q4f16_1 | HuggingFace |
| Mixtral-8x7B-Instruct-v0.1 | q4f32_1 | HuggingFace |
| SmolLM-1.7B-Instruct | q0f16 | HuggingFace |
| SmolLM-1.7B-Instruct | q0f32 | HuggingFace |
| SmolLM-1.7B-Instruct | q4f16_1 | HuggingFace |
| SmolLM-1.7B-Instruct | q4f32_1 | HuggingFace |
| SmolLM-135M-Instruct | q0f16 | HuggingFace |
| SmolLM-135M-Instruct | q0f32 | HuggingFace |
| SmolLM-135M-Instruct | q4f16_1 | HuggingFace |
| SmolLM-135M-Instruct | q4f32_1 | HuggingFace |
| SmolLM-360M-Instruct | q0f16 | HuggingFace |
| SmolLM-360M-Instruct | q0f32 | HuggingFace |
| SmolLM-360M-Instruct | q4f16_1 | HuggingFace |
| SmolLM-360M-Instruct | q4f32_1 | HuggingFace |
| SmolLM2-1.7B-Instruct | q0f16 | HuggingFace |
| SmolLM2-1.7B-Instruct | q4f16_1 | HuggingFace |
| SmolLM2-1.7B-Instruct | q4f32_1 | HuggingFace |
| SmolLM2-135M-Instruct | q0f16 | HuggingFace |
| SmolLM2-135M-Instruct | q0f32 | HuggingFace |
| SmolLM2-135M-Instruct | q4f16_1 | HuggingFace |
| SmolLM2-135M-Instruct | q4f32_1 | HuggingFace |
| SmolLM2-360M-Instruct | q0f16 | HuggingFace |
| SmolLM2-360M-Instruct | q0f32 | HuggingFace |
| SmolLM2-360M-Instruct | q4f16_1 | HuggingFace |
| SmolLM2-360M-Instruct | q4f32_1 | HuggingFace |
| gemma-2-27b-it | q0f16 | HuggingFace |
| gemma-2-27b-it | q4f16_1 | HuggingFace |
| gemma-2-27b-it | q4f32_1 | HuggingFace |
| gemma-2-2b-it | q0f16 | HuggingFace |
| gemma-2-2b-it | q0f32 | HuggingFace |
| gemma-2-2b-it | q4f16_0 | HuggingFace |
| gemma-2-2b-it | q4f16_1 | HuggingFace |
| gemma-2-2b-it | q4f32_1 | HuggingFace |
| gemma-2-2b-jpn-it | q0f16 | HuggingFace |
| gemma-2-2b-jpn-it | q0f32 | HuggingFace |
| gemma-2-2b-jpn-it | q4f16_1 | HuggingFace |
| gemma-2-2b-jpn-it | q4f32_1 | HuggingFace |
| gemma-2-9b-it | q0f16 | HuggingFace |
| gemma-2-9b-it | q3f16_1 | HuggingFace |
| gemma-2-9b-it | q4f16_1 | HuggingFace |
| gemma-2-9b-it | q4f32_1 | HuggingFace |
| internlm2_5-1_8b | q0f16 | HuggingFace |
| internlm2_5-1_8b | q4f16_1 | HuggingFace |
| internlm2_5-1_8b | q4f32_1 | HuggingFace |
| internlm2_5-1_8b-chat | q0f16 | HuggingFace |
| internlm2_5-1_8b-chat | q4f16_1 | HuggingFace |
| internlm2_5-1_8b-chat | q4f32_1 | HuggingFace |
| internlm2_5-20b | q0f16 | HuggingFace |
| internlm2_5-20b | q4f16_1 | HuggingFace |
| internlm2_5-20b | q4f32_1 | HuggingFace |
| internlm2_5-20b-chat | q0f16 | HuggingFace |
| internlm2_5-20b-chat | q4f16_1 | HuggingFace |
| internlm2_5-20b-chat | q4f32_1 | HuggingFace |
| internlm2_5-7b | q0f16 | HuggingFace |
| internlm2_5-7b | q4f16_1 | HuggingFace |
| internlm2_5-7b | q4f32_1 | HuggingFace |
| internlm2_5-7b-chat | q0f16 | HuggingFace |
| internlm2_5-7b-chat | q4f16_1 | HuggingFace |
| internlm2_5-7b-chat | q4f32_1 | HuggingFace |