Deploy GLM-5.2-FP8 as your open, frontier-level agent
When it comes to agentic coding tasks specifically, the quality gap is narrowing day by day. For instance, last month Z.AI released GLM-5.2, their new flagship model directly competing with the most capable closed models from Anthropic or OpenAI.
My goal with this article is to deploy GLM-5.2-FP8 and use it as a coding agent, proving that open models can deliver excellent performance on complex tasks.
Let's talk about hardware
Top performant AI models usually mean heavy hardware requirements (even when the authors claim they can run on so called “consumer hardware”), and reality is that for "local" inference this is still a massive entry barrier. We have seeing this over the past couple of years as RAM and accelerator prices haven't stopped rising.
Using smaller alternatives, such as GGUF-quantized models can help with this, but for top open models you still need a handful of GPUs. For example, unsloth/Kimi-K2.7-Code-GGUF still requires 304 GB for its smallest 1-bit GGUF quantization.
For the average user (myself included), owning the hardware required to run these large frontier open models is not feasible. This makes cloud compute or serverless providers the only realistic option for working with them. However, for small teams or labs willing to invest in hardware and prepared to deal with a variable degree of infra friction, I do believe that the "local paradigm" already makes sense and is worth the effort. You will have full control of your data, models and tokens, with potentially enough capacity to serve multiple users.
As mentioned above, I believe top models are sometimes overkill for most users and tasks. For a "smaller" expense, you can get yourself a capable GPU, a NVIDIA DGX Spark, or a Mac with sufficient unified memory. That would be enough to run excellent models such as google/gemma-4-31B-it or unsloth/Qwen3.6-35B-A3B-GGUF, which can serve you quite well on your daily tasks.
What I am using for this demo
In order to run GLM-5.2-FP8 with a max context length of 131k tokens, the amount of VRAM required is around 940 GB (estimated via hf mem).
For this demo, our partners at Dell provided me with access to one of their AI workstations: a Dell PowerEdge XE9680 8xH200. Each NVIDIA H200 GPU has ~144 GB of VRAM, giving us plenty of headroom to deploy the model with the described context configuration.
Deploying optimized models with Dell Enterprise Hub
Dell Enterprise Hub offers a broad catalog of AI models and apps. These are secure, deployment-ready Docker images built by Hugging Face and optimized for running on Dell hardware, making them ideal for this demo.
You can easily browse the DEH model catalog, filter by your hardware in the left-hand menu and get the Docker deployment snippet for your preferred model.
In this case, the deployment snippet for running GLM-5.2-FP8 on our Dell PowerEdge XE9680 8xH200 looks like this:
docker run \
-it \
-p 80:8000 \
--shm-size 1g \
--gpus 8 \
-e HF_TOKEN=hf_***** \
-e MODEL_ID=zai-org/GLM-5.2-FP8 \
-e TENSOR_PARALLEL_SIZE=8 \
-e MAX_MODEL_LEN=131072 \
-e ENABLE_AUTO_TOOL_CHOICE=true \
-e TOOL_CALL_PARSER=glm47 \
-e REASONING_PARSER=glm45 \
-e KV_CACHE_DTYPE=fp8 \
-e SPECULATIVE_CONFIG='{"method":"mtp","num_speculative_tokens":5}' \
registry.dell.huggingface.co/enterprise-dell-inference-zai-org-glm-5.2-fp8:vllm-v0.24.0
Remember to provide your Hugging Face token through the
HF_TOKENenvironment variable. You can create a new token here.
After deploying the model, GPUs usage looks like this:
Agent = model + harness
To add agentic behavior on top of a model (tool usage, MCP servers, session management, etc.) we have to connect it to a harness.
You can learn more about AI agents terminology in this article.
There are several options: On the proprietary side, Claude Code is an excellent tool that allows you to connect a model of your choice while giving you almost the same user experience as with its closed models. On the open-source side, Codex, OpenCode or Pi are among the most popular options.
Something important to note is that each harness has its own implementation and some LLMs might perform better on one than on the other. For example, Qwen models are optimized for Qwen-Code. However, most open-weights models, like GLM-5.2-FP8 in this case, can run seamlessly on Codex, Claude Code, OpenCode, etc.
For the purpose of this demo, I am using OpenCode. We will also connect the Hugging Face MCP Server so that our agent can interact with the Hugging Face ecosystem.
The final OpenCode configuration file looks like this:
# file @ (~/.config/opencode/open/opencode.json)
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"dell-enterprise-hub": {
"npm": "@ai-sdk/openai",
"options": {
"baseURL": "http://127.0.0.1:8080/v1"
},
"models": {
"zai-org/GLM-5.2-FP8": {
"name": "zai-org/GLM-5.2-FP8",
"tools": true
}
}
}
},
"mcp": {
"hf-mcp-server": {
"type": "remote",
"url": "https://huggingface.co/mcp",
"enabled": true
}
}
}
Time to test our agent!
To demonstrate the capabilities of this model, I will ask the agent to train a couple of small LMs of its choice, on a dataset of its choice and report which one performs the best.
This is a complex task. To address it, the agent must retrieve information about models and datasets from the Hugging Face hub, write a DPO training script using the TRL library, understand how Hugging Face Jobs works, launch the training jobs, track their progress with track.io, fetch the results and write a report summarizing the findings.
This translates into to the following prompts:
- Find the 2 best base LLMs for text generation under 1B parameters.
- Check TRL examples on the hub and find a small conversational preferences dataset for general chat alignment.
- Write a script for performing DPO with TRL for a given model and dataset. Track training with track.io under "dpo-demo" project.
- Now:
- Run a smoke-test hf job to make sure the script works. Stop it as soon as the job runs.
- Launch 2 hf jobs for training each model on the proposed dataset.
- Once finished, gather training and eval metrics and create a short report to see which model performs the best.
Before starting the session, I need to log into my Hugging Face account with hf auth login and install the HF skill with hf skills add --global so that the agent can use it.
Here you can see the beginning of the session with the agent:
Insights on the session with the agent
When asked to get the two best base LLMs for the task, the agent suggested google/gemma-3-270m and Qwen/Qwen2.5-0.5B, based on their popularity on the HF hub. Then it looked for a suitable preferences dataset and suggested trl-lib/Capybara-Preferences, since this collection is used in TRL examples. After that, it wrote a DPO training script with TRL based on snippets from the documentation.
The agent then iterated on the smoke-test job, adjusting the script, job instance and parameters until it found a working configuration (all on its own by retrieving and analyzing the job outputs). Next, the agent launched the two training jobs for the selected models. The training job for the gemma model failed with an out-of-memory error because its embedding size was larger than expected, so the agent reduced the batch size and relaunched the job.
Check results
After the jobs were complete, I asked the agent to upload both models to my Hugging Face account and generate a model card for each one. You can view both models in this collection.
For this article, I manually uploaded the report's markdown file to ChatGPT and asked for a more readable infographic.
As we can see, the models didn't really improve that much according to the training metrics. This simple experiment could be extended through a more interactive sesion with the agent, asking for example to explore different hyperparameters combinations, or perform a more robust quality assessment on targeted evaluation benchmarks.
Conclusions
Throughout this article, we have seen how a frontier open model can be as powerful as the closed AI providers we usually rely on. Although this specific setup is not affordable for everyone, I hope this post has provided useful insight and a bit of clarity about working with open AI. To me, owning your data and your models is worth the effort.
There are excellent open models out there that can help you get started. Explore the Hugging Face Hub and find the model that best fits your task and hardware.
Useful resources
- Open LLM leaderboard, where you can explore the best open LLMs by performance, size, hardware, etc.
- Artificial Analysis, another excellent resource for exploring and comparing models and AI providers.
- Dell Enterprise Hub, where you can browse a catalog of DEH-optimized models and apps ready to deploy.
- llama.cpp, for serving GGUF models on your device.
- Deploy GLM-5.2-FP8 on Foundry, an alternative version of this tutorial that deploys GLM-5.2-FP8 on a cloud instance.