Haoxuan Wu

Diffusion models have shown remarkable success in image generation tasks. However, accurately interpreting and translating the semantic meaning of input text into coherent visuals remains a significant challenge. We observe that existing approaches often rely on enhancing attention maps in a pixel-based or patch-based manner, which can lead to issues such as non-contiguous regions, unintended region leakage, eventually causing attention maps with limited semantic richness, degrade output quality. To address these limitations, we propose CoRe Diffusion, a novel method that provides comprehensive regional guidance throughout the generation process. Our approach introduces a region-assignment mechanism coupled with a tailored optimization strategy, enabling attention maps to better capture and express semantic information of concepts. Additionally, we incorporate mask guidance during the denoising steps to mitigate region leakage. Through extensive comparisons with state-of-the-art methods and detailed visual analyses, we demonstrate that our approach achieves superior performance, offering a more faithful image generation framework with semantically accurate procedure. Furthermore, our framework offers flexibility by supporting both automatic region assignment and user-defined spatial inputs as conditional guidance, enhancing its adaptability for diverse applications.

Customizing diffusion models for multiple concepts remains challenging due to cross-concept interference. This paper introduces Multi-SBoRA, a novel method for customizing diffusion models for multiple concepts. By leveraging orthogonal standard basis vectors, Multi-SBoRA constructs low-rank matrices for LoRA fine-tuning, enabling regional and non-overlapping weight updates that effectively mitigate crosstalk between different concepts. This approach preserves the knowledge embedded in the pre-trained model and reduces interference between customized concepts, thereby ensuring that each concept is learned independently without compromising its integrity. The localized weight updates also reduce computational overhead and enhance model flexibility. Experimental results demonstrate the optimal quantitative performance of Multi-SBoRA, showcasing its efficacy in addressing multi-concept customization while maintaining independence via orthogonality and localized updates, and mitigating crosstalk effects.