Instructions to use infly/Infinity-Parser2-Pro with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use infly/Infinity-Parser2-Pro with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="infly/Infinity-Parser2-Pro") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoProcessor, AutoModelForImageTextToText processor = AutoProcessor.from_pretrained("infly/Infinity-Parser2-Pro") model = AutoModelForImageTextToText.from_pretrained("infly/Infinity-Parser2-Pro") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps
- vLLM
How to use infly/Infinity-Parser2-Pro with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "infly/Infinity-Parser2-Pro" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infly/Infinity-Parser2-Pro", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/infly/Infinity-Parser2-Pro
- SGLang
How to use infly/Infinity-Parser2-Pro with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "infly/Infinity-Parser2-Pro" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infly/Infinity-Parser2-Pro", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "infly/Infinity-Parser2-Pro" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "infly/Infinity-Parser2-Pro", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use infly/Infinity-Parser2-Pro with Docker Model Runner:
docker model run hf.co/infly/Infinity-Parser2-Pro
OOM and KV Cache Memory Shortage during Single H800 Inference with Infinity-Parser2
Subject: Troubleshooting "No available memory for the cache blocks" when running Infinity-Parser2 on a single H800 GPU
Recently, while deploying the Infinity-Parser2-Pro model, some developers encountered Out-Of-Memory (OOM) crashes when attempting to initialize the vLLM engine for inference on a single NVIDIA H800 (80GB) GPU.
- Error Analysis: Why isn't 80GB VRAM enough?
The typical error traceback looks like this:
Sometimes, this is accompanied by assertion errors related to batch sizes (e.g., something conceptually similar to assert num_cache_lines >= batch at the lower levels).
The root cause is that Infinity-Parser2-Pro is a high-resolution Multimodal Large Language Model (MLLM). The combination of its massive model weights and the necessary KV Cache footprint is enormous. When vLLM attempts to initialize the engine and allocate memory chunks (cache blocks), it finds that after loading the model weights, the remaining VRAM is insufficient to meet the minimum KV Cache requirements. A single 80GB H800 is practically exhausted just holding the model, leaving no room to actually serve requests.
- Solution: Scale out with Tensor Parallelism (TP)
Since we are hitting a physical hardware limit on a single card, the most effective and straightforward solution is to use Tensor Parallelism to distribute the model across multiple GPUs.
Recommendation: Abandon single-GPU inference and switch to a 2-GPU (TP=2) or 4-GPU (TP=4) setup.
Actionable Steps: Adjust the --tensor-parallel-size argument when starting the vLLM server or initializing the InfinityParser2 class.

