🏗️ Building on HF

Abdennacer Badaoui

badaoui

82 46 48

AI & ML interests

None yet

Recent Activity

updated a Space 1 day ago

badaoui/strix-halo-on-transformers

liked a Space 1 day ago

pollen-robotics/microduck-simulator

liked a model 21 days ago

Qwen/Qwen3.8-27B

View all activity

Organizations

posted an update about 2 months ago

Post

2412

432 GB of ultra-fast HBM4 and up to 23.3 TB/s of memory bandwidth on a single GPU 🤯.

Two weeks ago, we got early access to AMD's new Instinct MI455X, and our first goal was simple: make sure 🤗 Transformers works on day one.

Over the past few weeks, we worked closely with the AMD team to validate the platform, enable Flash Attention, add torchcodec support for multimodal models, and resolve issues uncovered during testing.

The result:
✅ 99.5% success rate across our 24 core Transformers model architectures - already on par with our daily CI on previous AMD and NVIDIA platforms.

The hardware is just as exciting. With 432 GB of HBM per GPU, our early capacity experiments showed more than 3× the concurrent long-context requests compared to MI300, thanks to the much larger KV cache capacity.

A huge thanks to the AMD team for the early access and the great collaboration!

Read the full blog 👇
https://huggingface.co/blog/badaoui/transformers-on-amd-mi455

1 reply

reacted to danielhanchen's post with 🔥 7 months ago

Post

5329

We collaborated with Hugging Face to enable you to train MoE models 12× faster with 35% less VRAM via our new Triton kernels (no accuracy loss). 🤗

Train gpt-oss locally on 12.8GB VRAM with our free notebooks: https://unsloth.ai/docs/new/faster-moe

1 reply

reacted to IlyasMoutawwakil's post with 🔥 7 months ago

Post

3381

Transformers v5 just landed! 🚀
It significantly unifies and reduces modeling code across architectures, while opening the door to a whole new class of performance optimizations.

My favorite new feature? 🤔
The new dynamic weight loader + converter. Here’s why 👇

Over the last few months, the core Transformers maintainers built an incredibly fast weight loader, capable of converting tensors on the fly while loading them in parallel threads. This means we’re no longer constrained by how parameters are laid out inside the safetensors weight files.

In practice, this unlocks two big things:
- Much more modular modeling code. You can now clearly see how architectures build on top of each other (DeepSeek v2 → v3, Qwen v2 → v3 → MoE, etc.). This makes shared bottlenecks obvious and lets us optimize the right building blocks once, for all model families.
- Performance optimizations beyond what torch.compile can do alone. torch.compile operates on the computation graph, but it can’t change parameter layouts. With the new loader, we can restructure weights at load time: fusing MoE expert projections, merging attention QKV projections, and enabling more compute-dense kernels that simply weren’t possible before.

Personally, I'm honored to have contributed in this direction, including the work on optimizing MoE implementations and making modeling code more torch-exportable, so these optimizations can be ported cleanly across runtimes.

Overall, Transformers v5 is a strong signal of where the community and industry are converging: Modularity and Performance, without sacrificing Flexibility.

Transformers v5 makes its signature from_pretrained an entrypoint where you can mix and match:
- Parallelism
- Quantization
- Custom kernels
- Flash/Paged attention
- Continuous batching
- ...

Kudos to everyone involved! I highly recommend the:
Release notes: https://github.com/huggingface/transformers/releases/tag/v5.0.0
Blog post: https://huggingface.co/blog/transformers-v5

3 replies

posted an update 10 months ago

Post

576

Building high-performance, reproducible kernels for AMD ROCm just got a lot easier.

I've put together a guide on building, testing, and sharing ROCm-compatible kernels using the Hugging Face kernel-builder and kernels libraries; so you can focus on optimizing performance rather than spending time on setup.

Learn how to:

- Use Nix for reproducible builds
- Integrate kernels as native PyTorch operators
- Share your kernels on the Hub for anyone to use with kernels.get_kernel()

We use the 🏆 award-winning RadeonFlow GEMM kernel as a practical example.

📜 Check out the full guide here : https://huggingface.co/blog/build-rocm-kernels

reacted to abidlabs's post with 👍 10 months ago

Post

11581

Why I think local, open-source models will eventually win.

The most useful AI applications are moving toward multi-turn agentic behavior: systems that take hundreds or even thousands of iterative steps to complete a task, e.g. Claude Code, computer-control agents that click, type, and test repeatedly.

In these cases, the power of the model is not how smart it is per token, but in how quickly it can interact with its environment and tools across many steps. In that regime, model quality becomes secondary to latency.

An open-source model that can call tools quickly, check that the right thing was clicked, or verify that a code change actually passes tests can easily outperform a slightly “smarter” closed model that has to make remote API calls for every move.

Eventually, the balance tips: it becomes impractical for an agent to rely on remote inference for every micro-action. Just as no one would tolerate a keyboard that required a network request per keystroke, users won’t accept agent workflows bottlenecked by latency. All devices will ship with local, open-source models that are “good enough” and the expectation will shift toward everything running locally. It’ll happen sooner than most people think.

8 replies

posted an update about 1 year ago

Post

448

🚀 Optimum libraries keep growing, and Optimum v2 is just around the corner!

I recently added ONNX export support for a bunch of new models in the optimum-onnx library, including: DeepSeek-V3, Cohere, Nemotron, Arcee, StableLM … and more!

⚡ With ONNX export, you can run your favorite models faster and more efficiently across different hardware backends, making deployment and experimentation much smoother.

💡 Have a model you’d love to see supported? Contributions are super welcome — let’s make Optimum even better together!

#ONNX #Optimum #HuggingFace #OpenSource #AI

replied to their post about 1 year ago

From the available metrics I currently have, I looked at how much quantizing only the mlp.down_proj layers impacts end-to-end perplexity compared to the original float model and to quantizing all linear layers.
I used asymmetric W4A16 quantization and evaluated on the same held-out dataset used for the sensitivity profiling.

Here’s a summary of the results:

Model	Original PPL	`mlp.down_proj` quantized PPL	All linear layers quantized PPL (except `lm_head`)
Gemma-2B	16.80	58.29	76.69
LLaMA-3.2-3B	13.73	15.11	17.32
Qwen-3-4B	20.10	22.73	29.70

Observations:

Gemma-2B: Quantizing only the mlp.down_proj layers already causes a large perplexity increase (from 16.80 -> 58.29), showing they’re a major bottleneck for precision loss.
LLaMA-3.2-3B: The impact is smaller (13.73 -> 15.11), but still noticeable.
Qwen-3-4B: The increase is modest (20.10 -> 22.73), and here the sensitivity is more evenly distributed — other linear layers are nearly as fragile as mlp.down_proj, so the relative contribution is less pronounced.

It would also be interesting to run evaluations of these models on downstream benchmarks to see how these perplexity changes translate into real-task performance differences.

replied to their post about 1 year ago

For a detailed breakdown of the divergence-based method used to measure layer sensitivity, you can read my blog post here: https://huggingface.co/blog/badaoui/sensitivity-aware-mixed-precision-quantizer-v1

posted an update about 1 year ago

Post

3261

Is there a "one-size-fits-all" recipe for quantizing Large Language Models? 🤔

As part of my ongoing work in mixed-precision quantization, I've been exploring this question by measuring layer-by-layer sensitivity. The goal is to see if we can find universal rules for which layers can be quantized aggressively without impacting performance.The results are fascinating and reveal two key insights:

1️⃣ Sensitivity profiles are like architectural "fingerprints." Models from the same family share strikingly similar sensitivity patterns. As you can see in the charts below for the Gemma and SmolLM families, the ranking and relative sensitivity of the layers remain remarkably consistent. This suggests that the underlying architecture is a primary driver of a model's quantization behavior.

2️⃣ A "universal" mixed-precision quantization strategy is challenging. While models within a family are similar, these "fingerprints" change dramatically when comparing different architectures like LLaMA, Qwen, and StableLM. This highlights the difficulty in creating a generalized mixed-precision configuration that works optimally across all model families.

However, there is one near-universal truth we uncovered: the mlp.down_proj layer consistently emerges as one of the most sensitive components across all models studied.
This finding strongly resonates with the work in "The Super Weight in Large Language Models" (by Mengxia Yu et al.). The paper identifies that functionally critical parameters, or "super weights," are concentrated in these down_proj layers. Our empirical results provide clear validation for this theory, showing these layers are highly intolerant to precision loss.

In short, while every architecture has a unique sensitivity profile, a fingerprint shaped not only by its core design but also by its specific training dataset and optimization approach, some components remain universally critical!
What are your thoughts?

4 replies

reacted to IlyasMoutawwakil's post with 🔥 about 1 year ago

Post

3535

🚀 Optimum: The Last v1 Release 🚀
Optimum v1.27 marks the final major release in the v1 series. As we close this chapter, we're laying the groundwork for a more modular and community-driven future:
- Optimum v2: A lightweight core package for porting Transformers, Diffusers, or Sentence-Transformers to specialized AI hardware/software/accelerators..
- Optimum‑ONNX: A dedicated package where the ONNX/ONNX Runtime ecosystem lives and evolves, faster-moving and decoupled from the Optimum core.

🎯 Why this matters:
- A clearer governance path for ONNX, fostering stronger community collaboration and improved developer experience..
- Enable innovation at a faster pace in a more modular, open-source environment.

💡 What this means:
- More transparency, broader participation, and faster development driven by the community and key actors in the ONNX ecosystem (PyTorch, Microsoft, Joshua Lochner 👀, ...)
- A cleaner, more maintainable core Optimum, focused on extending HF libraries to special AI hardware/software/accelerators tooling and used by our partners (Intel Corporation, Amazon Web Services (AWS), AMD, NVIDIA, FuriosaAI, ...)

🛠️ Major updates I worked on in this release:
✅ Added support for Transformers v4.53 and SmolLM3 in ONNX/ONNXRuntime.
✅ Solved batched inference/generation for all supported decoder model architectures (LLMs).

✨ Big shoutout to @echarlaix for leading the refactoring work that cleanly separated ONNX exporter logic and enabled the creation of Optimum‑ONNX.

📝 Release Notes: https://lnkd.in/gXtE_qji
📦 Optimum : https://lnkd.in/ecAezNT6
🎁 Optimum-ONNX: https://lnkd.in/gzjyAjSi
#Optimum #ONNX #OpenSource #HuggingFace #Transformers #Diffusers

Abdennacer Badaoui

AI & ML interests

Recent Activity

Organizations

badaoui's activity