hf-internal-testing/tiny-ltx2-5-modular-pipe

This is a modular diffusion pipeline built with 🧨 Diffusers' modular pipeline framework.

Pipeline Type: LTX25AutoBlocks

Description: Auto blocks for LTX-2.5 supporting text-to-video, image-to-video, condition-to-video and in-context (IC-LoRA) generation (joint video + audio). Identical to LTX2AutoBlocks except that the video decoder is LTX2DiffusionVaeDecoderStep, since the diffusion decoder is the native default from LTX-2.5 on. To decode with the convolutional VAE instead, swap the decode block: blocks.sub_blocks["decode"] = LTX2AutoDecoderStep().

This pipeline uses a 8-block architecture that can be customized and extended.

Example Usage

[TODO]

Pipeline Architecture

This modular pipeline is composed of the following blocks:

1. prompt_enhancer (LTX2AutoPromptEnhancerStep)
   • Conditional prompt-enhancer step, run only when enable_prompt_enhancement is truthy.
2. text_encoder (LTX2TextConditioningStep)
   • Text-conditioning stage for LTX-2.X: encodes the prompt(s), then runs the text connectors to produce the video/audio-branch connector embeddings the denoiser consumes. Outputs stay at one row per prompt -- the denoise stage expands them by num_videos_per_prompt -- so they can be reused across denoise runs.
3. duration (LTX2AutoDurationStep)
   • Conditional duration-prediction step, run only when num_frames is omitted.
4. vae_encoder (LTX2AutoVaeEncoderStep)
   • VAE encoder step that encodes the reference image into latents for image-to-video.
5. condition_encoder (LTX2AutoConditionEncoderStep)
   • Conditional condition-encoder step, run only for the condition and in-context workflows.
6. reference_encoder (LTX2AutoReferenceEncoderStep)
   • Conditional reference-encoder step, run only when reference_conditions are supplied.
7. denoise (LTX2AutoCoreDenoiseStep)
   • Auto denoise block that selects the workflow based on inputs.
8. decode (LTX25AutoDecoderStep)
   • Auto decode block for LTX-2.5 that selects the decoder based on inputs.

Model Components

1. prompt_enhancer (PreTrainedModel)
2. processor (ProcessorMixin)
3. text_encoder (PreTrainedModel)
4. tokenizer (PreTrainedTokenizerBase)
5. connectors (LTX2TextConnectors)
6. duration_head (LTX2DurationHead)
7. vae (AutoencoderKLLTX2Video)
8. video_processor (VideoProcessor)
9. transformer (LTX2VideoTransformer3DModel)
10. scheduler (FlowMatchEulerDiscreteScheduler)
11. audio_vae (AutoencoderKLLTX2Audio)
12. guider (LTX2Guidance)
13. audio_guider (LTX2Guidance)
14. diffusion_decoder (LTX2VideoDiffusionDecoderModel)
15. vocoder (LTX2Vocoder)

Workflow Input Specification

text2video

• prompt (str): The prompt or prompts to guide image generation.

image2video

• prompt (str): The prompt or prompts to guide image generation.
• image (Image | list): Reference image(s) for denoising. Can be a single image or list of images.

condition

• prompt (str): The prompt or prompts to guide image generation.
• conditions (list, optional): LTX2VideoCondition (or list of them) placing image/video conditions at latent frame indices of the generated video.

in_context

• prompt (str): The prompt or prompts to guide image generation.
• num_frames (int, optional): The number of frames in the generated video. Omit to auto-predict via the duration_head (see LTX2AutoDurationStep).
• reference_conditions (list): LTX2ReferenceCondition (or list of them) whose videos are encoded into extra latent tokens the IC-LoRA adapter attends to.

Input/Output Specification

Inputs:

• prompt (str, optional): The prompt or prompts to guide image generation.
• conditions (list, optional): LTX2VideoCondition (or list of them) placing image/video conditions at latent frame indices of the generated video.
• enable_prompt_enhancement (bool, optional, defaults to False): Whether to run the prompt enhancer. Opt-in, matching the Lightricks reference pipelines.
• system_prompt (str, optional): System prompt for enhancement. Defaults to LTX2_5_I2V_DEFAULT_SYSTEM_PROMPT when a PIL.Image.Image condition frame is available, else LTX2_5_T2V_DEFAULT_SYSTEM_PROMPT.
• prompt_max_new_tokens (int, optional): Maximum number of new tokens to generate during prompt enhancement. Defaults to 600, the LTX-2.5 Gemma-4 enhancer's budget.
• prompt_enhancement_kwargs (dict, optional): Keyword arguments for the enhancer's .generate call. Defaults to greedy decoding.
• prompt_enhancement_seed (int, optional, defaults to 10): Random seed for prompt enhancement (inert under LTX-2.5's greedy decoding).
• generator (Generator, optional): Torch generator for deterministic generation.
• image (Image | list, optional): Reference image(s) for denoising. Can be a single image or list of images.
• negative_prompt (str, optional): The prompt or prompts not to guide the image generation.
• max_sequence_length (int, optional, defaults to 1024): Maximum sequence length for prompt encoding.
• min_seconds (float, optional, defaults to 1.0): Lower bound on the auto-predicted duration.
• max_seconds (float, optional, defaults to 20.0): Upper bound on the auto-predicted duration. Must be strictly greater than min_seconds.
• frame_rate (float, optional, defaults to 24.0): Frames per second of the generated video.
• height (int, optional, defaults to 512): The height in pixels of the generated image.
• width (int, optional, defaults to 704): The width in pixels of the generated image.
• image_crf (int, optional): H.264 CRF used to re-compress the conditioning image before VAE encode, matching the compression the model was trained against. None (default) resolves from the text-encoder generation (33 through LTX-2.3, 18 for LTX-2.5). Pass 0 to skip re-compression. Requires a PIL.Image.Image when re-compression runs.
• num_frames (int, optional): The number of frames in the generated video. Omit to auto-predict via the duration_head (see LTX2AutoDurationStep).
• reference_conditions (list, optional): LTX2ReferenceCondition (or list of them) whose videos are encoded into extra latent tokens the IC-LoRA adapter attends to.
• reference_downscale_factor (int, optional, defaults to 1): Ratio between the target and reference resolutions; 2 means the reference is preprocessed at half the target resolution. Spatial coordinates are scaled by this factor so the reference tokens land in the target coordinate space. Must match the factor the IC-LoRA was trained with.
• conditioning_attention_strength (float, optional, defaults to 1.0): Scalar in [0, 1] controlling how strongly the noisy tokens and reference tokens attend to each other. 1.0 (default) leaves attention unmasked.
• conditioning_attention_mask (Tensor, optional): Optional pixel-space mask of shape (1, 1, F, H, W) with values in [0, 1] giving spatially varying attention strength. Downsampled to the reference's latent grid and multiplied by conditioning_attention_strength.
• num_videos_per_prompt (int, optional, defaults to 1): The number of images to generate per prompt.
• condition_latents (list, optional): Per-condition normalized VAE latents of shape [1, C, F, H, W].
• condition_strengths (list, optional): Per-condition conditioning strengths.
• condition_indices (list, optional): Per-condition latent frame index at which the condition is applied.
• condition_pixel_frames (list, optional): Per-condition trimmed pixel frame count, used to clamp single-frame keyframe coords.
• reference_latents (Tensor, optional): Packed reference tokens of shape [1, total_reference_tokens, C], or None when no reference conditions were supplied (LTX2AutoReferenceEncoderStep is skipped).
• reference_coords (Tensor, optional): RoPE coordinates for the reference tokens.
• reference_token_counts (list, optional): Per-reference token counts, in reference_conditions order.
• latents (Tensor): Pre-generated noisy latents for image generation.
• noise_scale (float, optional): Initial noise level for the un-conditioned tokens. None (default) resolves to sigmas[0] when custom sigmas are supplied, else 1.0.
• sigmas (list, optional): Custom sigmas for the denoising process.
• reference_cross_mask (Tensor, optional): Per-reference-token noisy<->reference attention strengths of shape [1, num_ref_tokens].
• num_inference_steps (int): The number of denoising steps.
• timesteps (Tensor): Timesteps for the denoising process.
• audio_latents (Tensor): Optional pre-encoded audio latents; random noise is used when not provided.
• **denoiser_input_fields (None, optional): conditional model inputs for the denoiser: e.g. prompt_embeds, negative_prompt_embeds, etc.
• use_cross_timestep (bool, optional, defaults to True): Whether to condition the transformer on a separate per-token cross timestep (LTX-2.3+).
• attention_kwargs (dict, optional): Additional kwargs for attention processors.
• image_latents (Tensor, optional): VAE-encoded reference-image latents used for image-to-video conditioning.
• output_type (str, optional, defaults to pil): Output format: 'pil', 'np', 'pt'.

Outputs:

• videos (list): The generated videos.
• audio (Tensor): The generated audio waveform.