Sudeep Pillai PRO

spillai

2 2 160

https://people.csail.mit.edu/spillai/

AI & ML interests

Self-supervised learning, Few-shot learning, Computer Vision, Robotics

Recent Activity

posted an update 3 days ago

We're excited to introduce VLM Run Gateway - a new unified OpenAI-compatible API for running open-weight VLMs, OCR VLMs and ViT-based vision models. https://vlm.run/gateway Full model catalog: https://vlm.run/gateway/models Blog post announcement: https://www.vlm.run/blog/introducing-gateway Try different models on the gateway simply by updating the model name. Free to use and no sign-up required for now (in alpha). $ uvx vlmrun gw models $ uvx vlmrun gw chat <doc>.pdf -m glm-ocr $ uvx vlmrun gw chat <doc>.pdf -m deepseek-ocr-2 $ uvx vlmrun gw chat <doc>.pdf -m pp-ocrv6 $ uvx vlmrun gw chat <img>.jpg -m qwen/qwen3.5-0.8b -p "describe the image" $ uvx vlmrun gw chat <vid>.mp4 -m qwen/qwen3.5-0.8b -p "describe the video"

published an article 3 days ago

VLM Run Gateway: Run open-weight OCR, VLM and vision models behind one API

reacted to nwaughachukwuma's post with 🔥 19 days ago

# VLM Run Gateway: Run GLM-OCR, DeepSeek-OCR-2, and Dots.mocr with an OpenAI Compatible API Open-weight OCR VLMs have advanced significantly over the past year, yet most teams still rely on frontier VLMs for document parsing because researching, evaluating, and deploying the right models remains challenging. So we built VLM Run Gateway: one OpenAI-compatible endpoint for open-weight OCR and VLM models. If you’re using frontier VLMs primarily for OCR/document parsing, open-weight OCR models can be dramatically cheaper and often very accurate. With a one-line change, you can switch between open-weight OCR VLMs (DeepSeek OCR 2, GLM-OCR, dots.mocr, Paddle OCR VL, PP-OCRv6, etc.) and process 100K+ pages for under $60. Try it out quickly via the CLI: ``` uvx vlmrun gw models uvx vlmrun config set --api-key '<VLMRUN_API_KEY>' # anon-user, rate-limited uvx vlmrun gw chat <doc>.pdf -m zai-org/glm-ocr uvx vlmrun gw chat <doc>.pdf -m zai-org/glm-ocr --json-mode uvx vlmrun gw chat <doc>.pdf -m deepseek-ai/deepseek-ocr-2 uvx vlmrun gw chat <doc>.pdf -m rednote-hilab/dots.mocr uvx vlmrun gw chat <doc>.pdf -m paddleocr/pp-ocrv6 ``` OpenAI SDK: ``` client = OpenAI( base_url="https://gateway.vlm.run/v1/openai", api_key="<VLMRUN_API_KEY>", ) response = client.chat.completions.create( model="rednote-hilab/dots.mocr", messages=[{ "role": "user", "content": [{ "type": "document_url", "document_url": {"url": "https://.../invoice.pdf"}, }], }], extra_body={"document_dpi": 72}, ) ``` Docs: https://docs.vlm.run/gateway Catalog: https://docs.vlm.run/gateway/models MCP: https://docs.vlm.run/gateway/mcp-server Colab Quickstart: https://colab.research.google.com/drive/1RkuVIyuc5Po-UlcSlFyJCam5tjCm9IHM?usp=sharing Read the full post here: https://huggingface.co/blog/vlm-run/intro-to-vlmrun-gateway

View all activity

Organizations

posted an update 3 days ago

Post

3154

We're excited to introduce VLM Run Gateway - a new unified OpenAI-compatible API for running open-weight VLMs, OCR VLMs and ViT-based vision models.

https://vlm.run/gateway
Full model catalog: https://vlm.run/gateway/models
Blog post announcement: https://www.vlm.run/blog/introducing-gateway

Try different models on the gateway simply by updating the model name. Free to use and no sign-up required for now (in alpha).

$ uvx vlmrun gw models
$ uvx vlmrun gw chat <doc>.pdf -m glm-ocr
$ uvx vlmrun gw chat <doc>.pdf -m deepseek-ocr-2
$ uvx vlmrun gw chat <doc>.pdf -m pp-ocrv6
$ uvx vlmrun gw chat <img>.jpg -m qwen/qwen3.5-0.8b -p "describe the image"
$ uvx vlmrun gw chat <vid>.mp4 -m qwen/qwen3.5-0.8b -p "describe the video"

reacted to nwaughachukwuma's post with 🔥 19 days ago

Post

3569

uvx vlmrun gw models
uvx vlmrun config set --api-key '<VLMRUN_API_KEY>' # anon-user, rate-limited
uvx vlmrun gw chat <doc>.pdf -m zai-org/glm-ocr

uvx vlmrun gw chat <doc>.pdf -m zai-org/glm-ocr --json-mode
uvx vlmrun gw chat <doc>.pdf -m deepseek-ai/deepseek-ocr-2
uvx vlmrun gw chat <doc>.pdf -m rednote-hilab/dots.mocr
uvx vlmrun gw chat <doc>.pdf -m paddleocr/pp-ocrv6

OpenAI SDK:

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key="<VLMRUN_API_KEY>",
)

response = client.chat.completions.create(
    model="rednote-hilab/dots.mocr",
    messages=[{
      "role": "user",
		  "content": [{
            "type": "document_url",
            "document_url": {"url": "https://.../invoice.pdf"},
      }],
    }],
    extra_body={"document_dpi": 72},
)

Docs: https://docs.vlm.run/gateway

Catalog: https://docs.vlm.run/gateway/models

MCP: https://docs.vlm.run/gateway/mcp-server

Colab Quickstart: https://colab.research.google.com/drive/1RkuVIyuc5Po-UlcSlFyJCam5tjCm9IHM?usp=sharing

Read the full post here: https://huggingface.co/blog/vlm-run/intro-to-vlmrun-gateway

reacted to nwaughachukwuma's post with 🤗 about 1 month ago

Post

682

Can a text-only model + a vision toolkit (mm-ctx) match a native vision model?

We benchmarked 4 setups on 23 multimodal tasks (image, video, audio, PDF):

• glm-5.2 (text-only) + mm-ctx: 88.4
• gemini-3.5-flash (vision): 83
• deepseek-v4-pro (text-only) + mm-ctx: 79.4
• qwen3.6-35b-a3b (vision): 44.3

The best text-only setup glm-5.2 + mm outperformed gemini-3.5-flash, the top vision model, by 5.4 points (6.5%). It was also:

• 1.5x faster (100s vs 150s mean per task)
• the only setup with zero timeouts (46/46 completed; gemini timed out 4x on bulk-image and long-video tasks)
• the only setup stable across runs (88.5 / 88.4)
• top on video (100.0), image (91.7), and PDF (90.0) tasks

The trade-offs: the toolkit consumed 3.3x more tokens (4.25M vs 1.28M), and lost on audio (85.6 vs 71.3).

On completed tasks alone the two are nearly identical (91.0 vs 88.4): the toolkit's edge is efficient extraction that keeps long media tasks inside the time budget.

Full report: https://huggingface.co/blog/vlm-run/text-only-models-with-mm

8 replies

reacted to nwaughachukwuma's post with 👀 about 2 months ago

Post

741

# One API for Every Visual & OCR Models.

The VLM Run Gateway is a fully compatible API for OpenAI chat completions for visual intelligence. If you’re building document extraction or visual understanding, the Gateway exposes OCR, VQA, and detection behind a single interface you already know.

Read the docs: https://docs.vlm.run/gateway/introduction.

We actively support the following recent OCR and VQA models, which you can try today at no cost:

* zai-org/glm-ocr
* rednote-hilab/dots.mocr
* paddleocr/pp-ocrv6
* qwen/qwen3.5-0.8b

## Quickstart

### CLI

uvx vlmrun gw models
uvx vlmrun config set --api-key '<VLMRUN_API_KEY>' # anon-user, rate-limited
uvx vlmrun gw chat <doc>.pdf -m zai-org/glm-ocr

uvx vlmrun gw chat <doc>.pdf -m zai-org/glm-ocr --json-mode
uvx vlmrun gw chat <doc>.pdf -m deepseek-ai/deepseek-ocr-2
uvx vlmrun gw chat <doc>.pdf -m rednote-hilab/dots.mocr
uvx vlmrun gw chat <doc>.pdf -m paddleocr/pp-ocrv6

### OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.vlm.run/v1/openai",
    api_key="<VLMRUN_API_KEY>",
)


response = client.chat.completions.create(
    model="zai-org/glm-ocr",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "document_url",
                    "document_url": {
                        "url": "https://storage.googleapis.com/vlm-data-public-prod/hub/examples/finance.sec-filings/tsla-8k.pdf"
                    },
                },
            ],
        }
    ],
    extra_body={"method": "markdown", "document_dpi": 150},
)

print(response.choices[0].message.content)

## Auth and limits

Anonymous auth is enabled, so you can omit the authorization header entirely, or send Bearer "" or Bearer vlmrun. Rate limits are 60 req/min and 1000 req/hr.

reacted to their post with 🔥 4 months ago

Post

8788

mm-ctx – fast, multimodal context for agents.

LLM-based agents handle text incredibly well, but images, videos, or PDFs with visual content are hard to interpret. mm-ctx gives your CLI agent multi-modal skills.

Try it interactively in Spaces: vlm-run/mm-ctx

Readme: https://vlm-run.github.io/mm/
PyPI: https://pypi.org/project/mm-ctx
SKILL.md: https://github.com/vlm-run/skills/blob/main/skills/mm-cli-skill/SKILL.md

mm-ctx is meant to feel familiar: the UNIX tools we already love (find/cat/grep/wc), rebuilt for file types LLMs can't read natively and designed to work with agents via the CLI.
- mm grep "invoice #1234" ~/Downloads searches across PDFs and returns line-numbered matches
- mm cat <document>.pdf returns a metadata description of the file
- mm cat <photo>.jpg returns a caption of the photo
- mm cat <video>.mp4 returns a caption of the video

A few things we obsessed over:
⚡ Speed: Rust core for the hot paths
🏠 Local-first, BYO model: Uses any OpenAI-compatible endpoint: Ollama, vLLM/SGLang, LMStudio with any multimodal LLM (Gemma4, Qwen3.5, GLM-4.6V).
🔗 Composable: stdin + structured outputs
🤖 Drops into any agent via mm-cli-skills: Claude Code, Codex, Gemini CLI, OpenClaw.

We’d love to hear your feedback! Especially on the CLI and what file types and workflows you would like to see next.

2 replies

posted an update 4 months ago

Post

8788

2 replies

Sudeep Pillai PRO

AI & ML interests

Recent Activity

Organizations

spillai's activity