MLC LLM is a universal solution that allows any language model to be deployed natively on a diverse set of hardware backends and native applications, plus a productive framework for everyone to further optimize model performance for their own use cases. Everything runs locally with no server support and accelerated with local GPUs on your phone and laptop. Check out our GitHub repository to see how we did it. You can also read through instructions below for trying out demos.
This section contains the instructions to run large-language models and chatbot natively on your environment.
Try out this TestFlight page (limited to the first 9000 users) to install and use our example iOS chat app built for iPhone. Vicuna-7B takes 4GB of RAM and RedPajama-3B takes 2.2GB to run. Considering the iOS and other running applications, we will need a recent iPhone with 6GB for Vicuna-7B or 4GB for RedPajama-3B to run the app. The application is only tested on iPhone 14 Pro Max, iPhone 14 Pro and iPhone 12 Pro.
To build the iOS app from source, You can also check out our GitHub repo.
Note: The text generation speed on the iOS app can be unstable from time to time. It might run slow in the beginning and recover to a normal speed then.
Download the APK file here and install on your phone. You can then start a chat with LLM. When you first open the app, parameters need to be downloaded and the loading process could be slow. In future run, the parameters will be loaded from cache (which is fast) and you can use the app offline. Our current demo relies on OpenCL support on the phone and takes about 6GB of RAM, if you have a phone with the latest Snapdragon chip, you can try out out demo.
We tested our demo on Samsung Galaxy S23. It does not yet work on Google Pixel due to limited OpenCL support. We will continue to bring support and welcome contributions from the open source community. You can also check out our GitHub repo to build the Android app from source.
Check out our blog post for the technical details throughout our process of making MLC-LLM possible for Android.
We provide a CLI (command-line interface) app to chat with the bot in your terminal. Before installing the CLI app, we should install some dependencies first.
After installing all the dependencies, just follow the instructions below the install the CLI app:
# Create a new conda environment and activate the environment. conda create -n mlc-chat conda activate mlc-chat # Install Git and Git-LFS if you haven't already. # They are used for downloading the model weights from HuggingFace. conda install git git-lfs git lfs install # Install the chat CLI app from Conda. conda install -c mlc-ai -c conda-forge mlc-chat-nightly --force-reinstall # Create a directory, download the model weights from HuggingFace, and download the binary libraries # from GitHub. Select one of the following `LOCAL_ID` for a prebuilt LLM. mkdir -p dist/prebuilt git clone https://github.com/mlc-ai/binary-mlc-llm-libs.git dist/prebuilt/lib # Download prebuilt weights of Vicuna-7B cd dist/prebuilt git clone https://huggingface.co/mlc-ai/mlc-chat-vicuna-v1-7b-q3f16_0 cd ../.. mlc_chat_cli --local-id vicuna-v1-7b-q3f16_0 # Download prebuilt weights of RedPajama-3B cd dist/prebuilt git clone https://huggingface.co/mlc-ai/mlc-chat-RedPajama-INCITE-Chat-3B-v1-q4f16_0 cd ../.. mlc_chat_cli --local-id RedPajama-INCITE-Chat-3B-v1-q4f16_0
Please check out WebLLM, our companion project that deploys models natively to browsers. Everything here runs inside the browser with no server support and accelerated with WebGPU.
The pre-packaged demos are for research purposes only, subject to the model License.