Hugging Face's logo

Buckets:
hf-doc-build
/
doc-dev

Files
xet
hf-doc-build/doc-dev / diffusers /main /ko /using-diffusers /diffedit.html
download
raw
43.3 kB
<meta charset="utf-8" /><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;DiffEdit&quot;,&quot;local&quot;:&quot;diffedit&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Source와 target 임베딩 생성하기&quot;,&quot;local&quot;:&quot;source와-target-임베딩-생성하기&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2},{&quot;title&quot;:&quot;반전을 위한 캡션 생성하기&quot;,&quot;local&quot;:&quot;반전을-위한-캡션-생성하기&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2}],&quot;depth&quot;:1}">
 <link href="/docs/diffusers/main/ko/_app/immutable/assets/0.e3b0c442.css" rel="modulepreload">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/entry/start.df0b2e2f.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/chunks/scheduler.94020406.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/chunks/singletons.4482ca3c.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/chunks/index.8b553f6b.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/chunks/paths.1334326c.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/entry/app.4378a265.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/chunks/index.a08c8d92.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/nodes/0.96378992.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/chunks/each.e59479a4.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/nodes/39.ad827579.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/chunks/Tip.3b0aeee8.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/chunks/CodeBlock.b23cf525.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/chunks/DocNotebookDropdown.d8a25975.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/chunks/globals.7f7f1b26.js">
 <link rel="modulepreload" href="/docs/diffusers/main/ko/_app/immutable/chunks/EditOnGithub.b1bceb47.js"><!-- HEAD_svelte-u9bgzb_START --><meta name="hf:doc:metadata" content="{&quot;title&quot;:&quot;DiffEdit&quot;,&quot;local&quot;:&quot;diffedit&quot;,&quot;sections&quot;:[{&quot;title&quot;:&quot;Source와 target 임베딩 생성하기&quot;,&quot;local&quot;:&quot;source와-target-임베딩-생성하기&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2},{&quot;title&quot;:&quot;반전을 위한 캡션 생성하기&quot;,&quot;local&quot;:&quot;반전을-위한-캡션-생성하기&quot;,&quot;sections&quot;:[],&quot;depth&quot;:2}],&quot;depth&quot;:1}"><!-- HEAD_svelte-u9bgzb_END --> <p></p> <h1 class="relative group"><a id="diffedit" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#diffedit"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>DiffEdit</span></h1> <div class="flex space-x-1 absolute z-10 right-0 top-0"> <div class="relative colab-dropdown "> <button class=" " type="button"> <img alt="Open In Colab" class="!m-0" src="https://colab.research.google.com/assets/colab-badge.svg"> </button> </div> <div class="relative colab-dropdown "> <button class=" " type="button"> <img alt="Open In Studio Lab" class="!m-0" src="https://studiolab.sagemaker.aws/studiolab.svg"> </button> </div></div> <p data-svelte-h="svelte-2mpbmd">이미지 편집을 하려면 일반적으로 편집할 영역의 마스크를 제공해야 합니다. DiffEdit는 텍스트 쿼리를 기반으로 마스크를 자동으로 생성하므로 이미지 편집 소프트웨어 없이도 마스크를 만들기가 전반적으로 더 쉬워집니다. DiffEdit 알고리즘은 세 단계로 작동합니다:</p> <ol data-svelte-h="svelte-pey9sj"><li>Diffusion 모델이 일부 쿼리 텍스트와 참조 텍스트를 조건부로 이미지의 노이즈를 제거하여 이미지의 여러 영역에 대해 서로 다른 노이즈 추정치를 생성하고, 그 차이를 사용하여 쿼리 텍스트와 일치하도록 이미지의 어느 영역을 변경해야 하는지 식별하기 위한 마스크를 추론합니다.</li> <li>입력 이미지가 DDIM을 사용하여 잠재 공간으로 인코딩됩니다.</li> <li>마스크 외부의 픽셀이 입력 이미지와 동일하게 유지되도록 마스크를 가이드로 사용하여 텍스트 쿼리에 조건이 지정된 diffusion 모델로 latents를 디코딩합니다.</li></ol> <p data-svelte-h="svelte-kuwfj4">이 가이드에서는 마스크를 수동으로 만들지 않고 DiffEdit를 사용하여 이미지를 편집하는 방법을 설명합니다.</p> <p data-svelte-h="svelte-1dydg7a">시작하기 전에 다음 라이브러리가 설치되어 있는지 확인하세요:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-comment"># Colab에서 필요한 라이브러리를 설치하기 위해 주석을 제외하세요</span>
<span class="hljs-comment">#!pip install -q diffusers transformers accelerate</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-xrqotw"><code>StableDiffusionDiffEditPipeline</code>에는 이미지 마스크와 부분적으로 반전된 latents 집합이 필요합니다. 이미지 마스크는 <code>generate_mask()</code> 함수에서 생성되며, 두 개의 파라미터인 <code>source_prompt</code>와 <code>target_prompt</code>가 포함됩니다. 이 매개변수는 이미지에서 무엇을 편집할지 결정합니다. 예를 들어, <em>과일</em> 한 그릇을 <em>배</em> 한 그릇으로 변경하려면 다음과 같이 하세요:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->source_prompt = <span class="hljs-string">&quot;a bowl of fruits&quot;</span>
target_prompt = <span class="hljs-string">&quot;a bowl of pears&quot;</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-e4y31s">부분적으로 반전된 latents는 <code>invert()</code> 함수에서 생성되며, 일반적으로 이미지를 설명하는 <code>prompt</code> 또는 <em>캡션</em>을 포함하는 것이 inverse latent sampling 프로세스를 가이드하는 데 도움이 됩니다. 캡션은 종종 <code>source_prompt</code>가 될 수 있지만, 다른 텍스트 설명으로 자유롭게 실험해 보세요!</p> <p data-svelte-h="svelte-1rujngr">파이프라인, 스케줄러, 역 스케줄러를 불러오고 메모리 사용량을 줄이기 위해 몇 가지 최적화를 활성화해 보겠습니다:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch
<span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> DDIMScheduler, DDIMInverseScheduler, StableDiffusionDiffEditPipeline
pipeline = StableDiffusionDiffEditPipeline.from_pretrained(
 <span class="hljs-string">&quot;stabilityai/stable-diffusion-2-1&quot;</span>,
 torch_dtype=torch.float16,
 safety_checker=<span class="hljs-literal">None</span>,
 use_safetensors=<span class="hljs-literal">True</span>,
)
pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)
pipeline.inverse_scheduler = DDIMInverseScheduler.from_config(pipeline.scheduler.config)
pipeline.enable_model_cpu_offload()
pipeline.enable_vae_slicing()<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-w1crpy">수정하기 위한 이미지를 불러옵니다:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> diffusers.utils <span class="hljs-keyword">import</span> load_image, make_image_grid
img_url = <span class="hljs-string">&quot;https://github.com/Xiang-cd/DiffEdit-stable-diffusion/raw/main/assets/origin.png&quot;</span>
raw_image = load_image(img_url).resize((<span class="hljs-number">768</span>, <span class="hljs-number">768</span>))
raw_image<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-pmxeaq">이미지 마스크를 생성하기 위해 <code>generate_mask()</code> 함수를 사용합니다. 이미지에서 편집할 내용을 지정하기 위해 <code>source_prompt</code>와 <code>target_prompt</code>를 전달해야 합니다:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> PIL <span class="hljs-keyword">import</span> Image
source_prompt = <span class="hljs-string">&quot;a bowl of fruits&quot;</span>
target_prompt = <span class="hljs-string">&quot;a basket of pears&quot;</span>
mask_image = pipeline.generate_mask(
 image=raw_image,
 source_prompt=source_prompt,
 target_prompt=target_prompt,
)
Image.fromarray((mask_image.squeeze()*<span class="hljs-number">255</span>).astype(<span class="hljs-string">&quot;uint8&quot;</span>), <span class="hljs-string">&quot;L&quot;</span>).resize((<span class="hljs-number">768</span>, <span class="hljs-number">768</span>))<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-nvc4r3">다음으로, 반전된 latents를 생성하고 이미지를 묘사하는 캡션에 전달합니다:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->inv_latents = pipeline.invert(prompt=source_prompt, image=raw_image).latents<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-rr1tl7">마지막으로, 이미지 마스크와 반전된 latents를 파이프라인에 전달합니다. <code>target_prompt</code>는 이제 <code>prompt</code>가 되며, <code>source_prompt</code>는 <code>negative_prompt</code>로 사용됩니다.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->output_image = pipeline(
 prompt=target_prompt,
 mask_image=mask_image,
 image_latents=inv_latents,
 negative_prompt=source_prompt,
).images[<span class="hljs-number">0</span>]
mask_image = Image.fromarray((mask_image.squeeze()*<span class="hljs-number">255</span>).astype(<span class="hljs-string">&quot;uint8&quot;</span>), <span class="hljs-string">&quot;L&quot;</span>).resize((<span class="hljs-number">768</span>, <span class="hljs-number">768</span>))
make_image_grid([raw_image, mask_image, output_image], rows=<span class="hljs-number">1</span>, cols=<span class="hljs-number">3</span>)<!-- HTML_TAG_END --></pre></div> <div class="flex gap-4" data-svelte-h="svelte-6s9t77"><div><img class="rounded-xl" src="https://github.com/Xiang-cd/DiffEdit-stable-diffusion/raw/main/assets/origin.png"> <figcaption class="mt-2 text-center text-sm text-gray-500">original image</figcaption></div> <div><img class="rounded-xl" src="https://github.com/Xiang-cd/DiffEdit-stable-diffusion/blob/main/assets/target.png?raw=true"> <figcaption class="mt-2 text-center text-sm text-gray-500">edited image</figcaption></div></div> <h2 class="relative group"><a id="source와-target-임베딩-생성하기" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#source와-target-임베딩-생성하기"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>Source와 target 임베딩 생성하기</span></h2> <p data-svelte-h="svelte-9hjii0">Source와 target 임베딩은 수동으로 생성하는 대신 <a href="https://huggingface.co/docs/transformers/model_doc/flan-t5" rel="nofollow">Flan-T5</a> 모델을 사용하여 자동으로 생성할 수 있습니다.</p> <p data-svelte-h="svelte-nlvmrb">Flan-T5 모델과 토크나이저를 🤗 Transformers 라이브러리에서 불러옵니다:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch
<span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> AutoTokenizer, T5ForConditionalGeneration
tokenizer = AutoTokenizer.from_pretrained(<span class="hljs-string">&quot;google/flan-t5-large&quot;</span>)
model = T5ForConditionalGeneration.from_pretrained(<span class="hljs-string">&quot;google/flan-t5-large&quot;</span>, device_map=<span class="hljs-string">&quot;auto&quot;</span>, torch_dtype=torch.float16)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-n8m7wd">모델에 프롬프트할 source와 target 프롬프트를 생성하기 위해 초기 텍스트들을 제공합니다.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START -->source_concept = <span class="hljs-string">&quot;bowl&quot;</span>
target_concept = <span class="hljs-string">&quot;basket&quot;</span>
source_text = <span class="hljs-string">f&quot;Provide a caption for images containing a <span class="hljs-subst">{source_concept}</span>. &quot;</span>
<span class="hljs-string">&quot;The captions should be in English and should be no longer than 150 characters.&quot;</span>
target_text = <span class="hljs-string">f&quot;Provide a caption for images containing a <span class="hljs-subst">{target_concept}</span>. &quot;</span>
<span class="hljs-string">&quot;The captions should be in English and should be no longer than 150 characters.&quot;</span><!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1yqnb1t">다음으로, 프롬프트들을 생성하기 위해 유틸리티 함수를 생성합니다.</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">@torch.no_grad()</span>
<span class="hljs-keyword">def</span> <span class="hljs-title function_">generate_prompts</span>(<span class="hljs-params">input_prompt</span>):
 input_ids = tokenizer(input_prompt, return_tensors=<span class="hljs-string">&quot;pt&quot;</span>).input_ids.to(<span class="hljs-string">&quot;cuda&quot;</span>)
outputs = model.generate(
 input_ids, temperature=<span class="hljs-number">0.8</span>, num_return_sequences=<span class="hljs-number">16</span>, do_sample=<span class="hljs-literal">True</span>, max_new_tokens=<span class="hljs-number">128</span>, top_k=<span class="hljs-number">10</span>
 )
 <span class="hljs-keyword">return</span> tokenizer.batch_decode(outputs, skip_special_tokens=<span class="hljs-literal">True</span>)
source_prompts = generate_prompts(source_text)
target_prompts = generate_prompts(target_text)
<span class="hljs-built_in">print</span>(source_prompts)
<span class="hljs-built_in">print</span>(target_prompts)<!-- HTML_TAG_END --></pre></div> <div class="course-tip bg-gradient-to-br dark:bg-gradient-to-r before:border-green-500 dark:before:border-green-800 from-green-50 dark:from-gray-900 to-white dark:to-gray-950 border border-green-50 text-green-700 dark:text-gray-400"><p data-svelte-h="svelte-1cna4a8">다양한 품질의 텍스트를 생성하는 전략에 대해 자세히 알아보려면 <a href="https://huggingface.co/docs/transformers/main/en/generation_strategies" rel="nofollow">생성 전략</a> 가이드를 참조하세요.</p></div> <p data-svelte-h="svelte-n2xyim">텍스트 인코딩을 위해 <code>StableDiffusionDiffEditPipeline</code>에서 사용하는 텍스트 인코더 모델을 불러옵니다. 텍스트 인코더를 사용하여 텍스트 임베딩을 계산합니다:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch
<span class="hljs-keyword">from</span> diffusers <span class="hljs-keyword">import</span> StableDiffusionDiffEditPipeline
pipeline = StableDiffusionDiffEditPipeline.from_pretrained(
 <span class="hljs-string">&quot;stabilityai/stable-diffusion-2-1&quot;</span>, torch_dtype=torch.float16, use_safetensors=<span class="hljs-literal">True</span>
)
pipeline.enable_model_cpu_offload()
pipeline.enable_vae_slicing()
<span class="hljs-meta">@torch.no_grad()</span>
<span class="hljs-keyword">def</span> <span class="hljs-title function_">embed_prompts</span>(<span class="hljs-params">sentences, tokenizer, text_encoder, device=<span class="hljs-string">&quot;cuda&quot;</span></span>):
 embeddings = []
 <span class="hljs-keyword">for</span> sent <span class="hljs-keyword">in</span> sentences:
 text_inputs = tokenizer(
 sent,
 padding=<span class="hljs-string">&quot;max_length&quot;</span>,
 max_length=tokenizer.model_max_length,
 truncation=<span class="hljs-literal">True</span>,
 return_tensors=<span class="hljs-string">&quot;pt&quot;</span>,
 )
 text_input_ids = text_inputs.input_ids
 prompt_embeds = text_encoder(text_input_ids.to(device), attention_mask=<span class="hljs-literal">None</span>)[<span class="hljs-number">0</span>]
 embeddings.append(prompt_embeds)
 <span class="hljs-keyword">return</span> torch.concatenate(embeddings, dim=<span class="hljs-number">0</span>).mean(dim=<span class="hljs-number">0</span>).unsqueeze(<span class="hljs-number">0</span>)
source_embeds = embed_prompts(source_prompts, pipeline.tokenizer, pipeline.text_encoder)
target_embeds = embed_prompts(target_prompts, pipeline.tokenizer, pipeline.text_encoder)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1x8m5so">마지막으로, 임베딩을 <code>generate_mask()</code> 및 <code>invert()</code> 함수와 파이프라인에 전달하여 이미지를 생성합니다:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --> from diffusers import DDIMInverseScheduler, DDIMScheduler
 from diffusers.utils import load_image, make_image_grid
 from PIL import Image
pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)
 pipeline.inverse_scheduler = DDIMInverseScheduler.from_config(pipeline.scheduler.config)
img_url = &quot;https://github.com/Xiang-cd/DiffEdit-stable-diffusion/raw/main/assets/origin.png&quot;
 raw_image = load_image(img_url).resize((768, 768))
mask_image = pipeline.generate_mask(
 image=raw_image,
<span class="hljs-deletion">- source_prompt=source_prompt,</span>
<span class="hljs-deletion">- target_prompt=target_prompt,</span>
<span class="hljs-addition">+ source_prompt_embeds=source_embeds,</span>
<span class="hljs-addition">+ target_prompt_embeds=target_embeds,</span>
 )
inv_latents = pipeline.invert(
<span class="hljs-deletion">- prompt=source_prompt,</span>
<span class="hljs-addition">+ prompt_embeds=source_embeds,</span>
 image=raw_image,
 ).latents
output_image = pipeline(
 mask_image=mask_image,
 image_latents=inv_latents,
<span class="hljs-deletion">- prompt=target_prompt,</span>
<span class="hljs-deletion">- negative_prompt=source_prompt,</span>
<span class="hljs-addition">+ prompt_embeds=target_embeds,</span>
<span class="hljs-addition">+ negative_prompt_embeds=source_embeds,</span>
 ).images[0]
 mask_image = Image.fromarray((mask_image.squeeze()*255).astype(&quot;uint8&quot;), &quot;L&quot;)
 make_image_grid([raw_image, mask_image, output_image], rows=1, cols=3)<!-- HTML_TAG_END --></pre></div> <h2 class="relative group"><a id="반전을-위한-캡션-생성하기" class="header-link block pr-1.5 text-lg no-hover:hidden with-hover:absolute with-hover:p-1.5 with-hover:opacity-0 with-hover:group-hover:opacity-100 with-hover:right-full" href="#반전을-위한-캡션-생성하기"><span><svg class="" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 256"><path d="M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z" fill="currentColor"></path></svg></span></a> <span>반전을 위한 캡션 생성하기</span></h2> <p data-svelte-h="svelte-13u0kl2"><code>source_prompt</code>를 캡션으로 사용하여 부분적으로 반전된 latents를 생성할 수 있지만, <a href="https://huggingface.co/docs/transformers/model_doc/blip" rel="nofollow">BLIP</a> 모델을 사용하여 캡션을 자동으로 생성할 수도 있습니다.</p> <p data-svelte-h="svelte-t95mcn">🤗 Transformers 라이브러리에서 BLIP 모델과 프로세서를 불러옵니다:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">import</span> torch
<span class="hljs-keyword">from</span> transformers <span class="hljs-keyword">import</span> BlipForConditionalGeneration, BlipProcessor
processor = BlipProcessor.from_pretrained(<span class="hljs-string">&quot;Salesforce/blip-image-captioning-base&quot;</span>)
model = BlipForConditionalGeneration.from_pretrained(<span class="hljs-string">&quot;Salesforce/blip-image-captioning-base&quot;</span>, torch_dtype=torch.float16, low_cpu_mem_usage=<span class="hljs-literal">True</span>)<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-lp3q9g">입력 이미지에서 캡션을 생성하는 유틸리티 함수를 만듭니다:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-meta">@torch.no_grad()</span>
<span class="hljs-keyword">def</span> <span class="hljs-title function_">generate_caption</span>(<span class="hljs-params">images, caption_generator, caption_processor</span>):
 text = <span class="hljs-string">&quot;a photograph of&quot;</span>
inputs = caption_processor(images, text, return_tensors=<span class="hljs-string">&quot;pt&quot;</span>).to(device=<span class="hljs-string">&quot;cuda&quot;</span>, dtype=caption_generator.dtype)
 caption_generator.to(<span class="hljs-string">&quot;cuda&quot;</span>)
 outputs = caption_generator.generate(**inputs, max_new_tokens=<span class="hljs-number">128</span>)
<span class="hljs-comment"># 캡션 generator 오프로드</span>
 caption_generator.to(<span class="hljs-string">&quot;cpu&quot;</span>)
caption = caption_processor.batch_decode(outputs, skip_special_tokens=<span class="hljs-literal">True</span>)[<span class="hljs-number">0</span>]
 <span class="hljs-keyword">return</span> caption<!-- HTML_TAG_END --></pre></div> <p data-svelte-h="svelte-1y7rdgp">입력 이미지를 불러오고 <code>generate_caption</code> 함수를 사용하여 해당 이미지에 대한 캡션을 생성합니다:</p> <div class="code-block relative"><div class="absolute top-2.5 right-4"><button class="inline-flex items-center relative text-sm focus:text-green-500 cursor-pointer focus:outline-none transition duration-200 ease-in-out opacity-0 mx-0.5 text-gray-600 " title="code excerpt" type="button"><svg class="" xmlns="http://www.w3.org/2000/svg" aria-hidden="true" fill="currentColor" focusable="false" role="img" width="1em" height="1em" preserveAspectRatio="xMidYMid meet" viewBox="0 0 32 32"><path d="M28,10V28H10V10H28m0-2H10a2,2,0,0,0-2,2V28a2,2,0,0,0,2,2H28a2,2,0,0,0,2-2V10a2,2,0,0,0-2-2Z" transform="translate(0)"></path><path d="M4,18H2V4A2,2,0,0,1,4,2H18V4H4Z" transform="translate(0)"></path><rect fill="none" width="32" height="32"></rect></svg> <div class="absolute pointer-events-none transition-opacity bg-black text-white py-1 px-2 leading-tight rounded font-normal shadow left-1/2 top-full transform -translate-x-1/2 translate-y-2 opacity-0"><div class="absolute bottom-full left-1/2 transform -translate-x-1/2 w-0 h-0 border-black border-4 border-t-0" style="border-left-color: transparent; border-right-color: transparent; "></div> Copied</div></button></div> <pre class=""><!-- HTML_TAG_START --><span class="hljs-keyword">from</span> diffusers.utils <span class="hljs-keyword">import</span> load_image
img_url = <span class="hljs-string">&quot;https://github.com/Xiang-cd/DiffEdit-stable-diffusion/raw/main/assets/origin.png&quot;</span>
raw_image = load_image(img_url).resize((<span class="hljs-number">768</span>, <span class="hljs-number">768</span>))
caption = generate_caption(raw_image, model, processor)<!-- HTML_TAG_END --></pre></div> <div class="flex justify-center" data-svelte-h="svelte-1jq3ee2"><figure><img class="rounded-xl" src="https://github.com/Xiang-cd/DiffEdit-stable-diffusion/raw/main/assets/origin.png"> <figcaption class="text-center">generated caption: &quot;a photograph of a bowl of fruit on a table&quot;</figcaption></figure></div> <p data-svelte-h="svelte-1lrhtv2">이제 캡션을 <code>invert()</code> 함수에 놓아 부분적으로 반전된 latents를 생성할 수 있습니다!</p> <a class="!text-gray-400 !no-underline text-sm flex items-center not-prose mt-4" href="https://github.com/huggingface/diffusers/blob/main/docs/source/ko/using-diffusers/diffedit.md" target="_blank"><span data-svelte-h="svelte-1kd6by1">&lt;</span> <span data-svelte-h="svelte-x0xyl0">&gt;</span> <span data-svelte-h="svelte-1dajgef"><span class="underline ml-1.5">Update</span> on GitHub</span></a> <p></p>
<script>
 {
 __sveltekit_bmkzir = {
 assets: "/docs/diffusers/main/ko",
 base: "/docs/diffusers/main/ko",
 env: {}
 };
 const element = document.currentScript.parentElement;
 const data = [null,null];
 Promise.all([
 import("/docs/diffusers/main/ko/_app/immutable/entry/start.df0b2e2f.js"),
 import("/docs/diffusers/main/ko/_app/immutable/entry/app.4378a265.js")
 ]).then(([kit, app]) => {
 kit.start(app, element, {
 node_ids: [0, 39],
 data,
 form: null,
 error: null
 });
 });
 }
 </script>

Xet Storage Details

Size:
43.3 kB
·
Xet hash:
81a567dcdb03f69b941a5943082eb815ab13414010bb59906c3fb7013810b995

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.