ApyHTML19/qwen-onnx-int8

Qwen2 with 1.5B ONNX INT8 — Mobile-Optimized Quantized Model

📌 Description

This is a quantized INT8 version of the Qwen language model, converted to ONNX format for efficient inference on low-resource devices such as mobile phones and edge hardware.

🎯 Objective

The goal of this project is to make Qwen accessible on devices with limited memory and compute power (e.g. iPhone 12, mid-range Android phones) without requiring high-end GPUs or cloud infrastructure.

⚡ Key Features

• Format: ONNX (INT8 quantized)
• Size: Reduced ~4x compared to FP32 original
• Target hardware: CPU-only devices, mobile, edge
• Inference engine: ONNX Runtime (onnxruntime)

🛠️ Usage

Install dependencies

pip install onnxruntime numpy

Run inference

import onnxruntime as ort

session = ort.InferenceSession( "model_int8.onnx", providers=["CPUExecutionProvider"] )

📁 Files

File	Description
model.onnx	Original FP32 model
model_int8.onnx	INT8 quantized model ✅
tokenizer.json	Tokenizer config
vocab.json	Vocabulary
quantize_int8.py	Quantization script