Base line model testing

[Aisuko]

We want to run the model on consumer-grade hardware. It means we will support different accelerate methods on CPU and GPU.

For CPU, ggml it the best choice.
For GPU, we support distribution by using Hugging Face libs

And, we want the model as small as it can be. So, we should choose a base line model. And as HuggingFace transformers supports gguf(format of ggml). So, we have many of choices below:

• Phi-3 gguf -3.8B
• gpt-2 - 1.5B

The reason we choice the base line model size larger than 1B is that we should make sure the results of model are reasonable and useful. I will upload some notebooks or model test result to support this later.

@Micost

[Aisuko]

GGML format https://huggingface.co/aisuko/gpt-2-1.5B-ggml/tree/main. This is a demo model aims to show the GPT2-xl CPU accelerate with ggml. We will focus on HuggingFace transformers support phi and GPT-series gguf. More detail see the official blog.

Update:

It loads model to memory with fp32, it isn't what we want. So, we give up use Huggingface Transformers and their ecosystems on inference.

LLamacpp

For the original of llama.cpp(gguf), it already support GPT and Phi series.

So, we can check if these models are working with Hugging Face transformers

Here are GPT2 117M, 1.5B gguf models and the test result of these models. The reason why we need to pick up the model with parameters larger than 1B see below.

Furthermore, for the quantization method in llama.cpp, see the

• Difference in different quantization methods
• The issue about k_quant methods

The model repos

The reason why we use Instruct model see Model Training Techniques

• GPT2-117M
• GPT2-1.5B
• Phi3-3.8B

Hugging Face transformers with gguf

• Phi3-mini

Related issues

• [Feature]: Adapting pipeline to our base line language model kirin#153
• [Feature]: Support inference with the Q4_k_m quantization gguf models kirin#162
• [feature] pipeline support quantization directly kimchima#59
• [feature]: 4-bit inference on CPU kimchima#31
• [feature] Support multiple adapters for core small language models kimchima#15

RLHF notebooks

The out of memory of GPU error because Kaggle low GPU resource. We can fine-tuning on RMIT AWS if necessary.

• https://www.kaggle.com/code/aisuko/fine-tune-llama3-with-orpo
• https://www.kaggle.com/code/aisuko/producing-adapter-with-vera
• https://www.kaggle.com/code/aisuko/fine-tune-openelm-270m
• https://www.kaggle.com/code/aisuko/fine-tune-phi-1-5b-with-orpo

[Aisuko]

OpemELM's architecture can be seen a template of the customise small language models.