Stable Diffusion Glossary

Stable Diffusion Glossary is our deep dive into the jargon-filled world of Stable Diffusion and Generative AI. In this instalment, we’re taking a closer look at some key terms to help you navigate this complex field with ease. So, let’s get started.

Stable Diffusion Glossary Terms and Terminology Part 1 (A – D)

.ckpt:
A .ckpt file, pronounced “checkpoint,” is a file format created by PyTorch Lightning, a research framework based on PyTorch. It contains a machine learning model used by Stable Diffusion to generate images.

.pt:
The .pt extension is used to denote a machine-learning model file created using PyTorch. These files contain algorithms that are designed to perform specific tasks automatically.

.Safetensors:
Safetensors is a file format specifically designed for Checkpoint models. It offers enhanced security by reducing the risk of embedded malicious code. This format provides a safer environment for storing and using Checkpoint models. For additional details, refer to the section on “Pickle.”

AGI:
AGI stands for Artificial General Intelligence. It represents a significant milestone in the field of AI, wherein artificial systems achieve or surpass human-level intelligence across various domains.

API:
API stands for Application Programming Interface. It refers to a set of functions, protocols, and tools that enable interactions between different software applications or components. APIs facilitate seamless communication and data exchange, allowing software systems to work together effectively.

Auto-GPT:
Auto-GPT is a prominent figure in the Stable Diffusion community. They are recognized as the creator of the popular WebUI graphical user interface, which provides a user-friendly interface for interacting with Stable Diffusion models. It is also an autonomous version of AI GPT.

Bard:
Bard is Google’s chatbot, powered by their LaMDA (Language Model for Dialogue Applications) model. It is designed to engage in conversational interactions and provide natural language responses.

Bing:
Bing is Microsoft’s chatbot, powered by ChatGPT, a language model developed by OpenAI. It is designed to generate human-like text responses based on user input.

CFG:
CFG stands for Classifier Free Guidance, also known as “Guidance Scale.” It is a parameter that controls the level of influence the text prompt has on the image generation process in Stable Diffusion. Adjusting the CFG value can determine how closely the generated image aligns with the provided text prompt.

Checkpoint:
A Checkpoint is the result of training on a large dataset of captioned images collected from various sources on the internet. It is a file that drives Stable Diffusion’s txt2img and img2img processes, allowing the model to generate images based on textual input.

Civitai (Civitai.com):
Civitai.com is a hosting site that provides a platform for hosting and sharing various AI-related files, including Checkpoint Models, Hypernets, Textual Inversion Embeddings, Aesthetic Gradients, and VAE (Variational Autoencoder) files.

CLIP:
CLIP is an open-source model developed by OpenAI. It is trained on a large dataset of images and captions and is designed to understand the relationship between images and textual descriptions. CLIP can evaluate how well a given caption describes an image.

Cmdr2:
Cmdr2 is a prominent figure in the Stable Diffusion community and is known for creating the EasyDiffusion one-click install graphical user interface. This user-friendly interface simplifies the process of interacting with Stable Diffusion models.

CodeFormer:
CodeFormer is a facial image restoration model used to enhance and restore the quality of blurry, grainy, or disfigured faces in images. It employs deep learning techniques to improve the appearance of facial features and details.

Colab
Collaboratory (Colab) is a product from Google Research that enables the execution of Python code directly through a web browser. It provides a cloud-based environment, allowing users to write and run code, particularly geared towards machine learning applications. Colab offers access to computational resources, such as CPUs, GPUs, and TPUs, and supports collaborative work and sharing of notebooks. It is commonly used for data analysis, machine learning experimentation, and research. More info

ComfyUI
ComfyUI is a popular and powerful modular user interface (UI) designed specifically for Stable Diffusion. It provides a workflow-oriented workspace, allowing users to interact with Stable Diffusion models in a user-friendly manner. ComfyUI offers a range of features and functionalities for image generation and manipulation. Compared to Auto1111 WebUI, ComfyUI provides a more complex and extensive set of tools. More info

CompVis
CompVis refers to the Computer Vision & Learning research group at Ludwig Maximilian University of Munich. They are involved in research related to computer vision and machine learning. The group hosts Stable Diffusion models on the Hugging Face platform, which provides access to various pre-trained models and tools for natural language processing and computer vision tasks.

Conda
Conda is an open-source package manager widely used in the Python ecosystem. It allows users to create isolated environments and easily install, manage, and update packages for different programming languages, including Python. Conda simplifies the management of dependencies and provides a convenient way to create reproducible environments for software development and data analysis.

ControlNet
ControlNet is an extension to the Auto1111 WebUI that enhances the capabilities of image manipulation within the interface. It provides additional functionalities to modify and transform images using various techniques and algorithms. ControlNet expands the possibilities for image editing and customization within the Stable Diffusion framework. More info

Convergence
Convergence in the context of image generation refers to the point at which the generated images stop changing significantly as the number of steps in the generation process increases. As the image generation progresses, the generated images become more refined and closer to the desired output. Convergence indicates that the model has reached a stable state, and further steps may not result in noticeable changes to the generated images.

CUDA
CUDA stands for Compute Unified Device Architecture, which is Nvidia’s parallel processing architecture. It is designed to leverage the computational power of Nvidia GPUs (Graphics Processing Units) for high-performance computing tasks. CUDA provides a programming model and a set of tools that enable developers to accelerate applications by offloading computationally intensive tasks to GPUs. It is widely used in various domains, including machine learning, scientific simulations, and graphics rendering.

DALL-E / DALL-E 2
DALL-E is a deep learning model developed by OpenAI that specializes in image generation based on textual descriptions. It uses a combination of unsupervised learning and generative modeling to create unique and coherent images from textual prompts. DALL-E 2 refers to an improved or updated version of the DALL-E model. These models have been trained on large datasets and can generate diverse and high-quality images based on user-provided text descriptions. They are available as a commercial image generation service.

Danbooru
Danbooru is an English-based imageboard website that focuses on sharing and discussing fan art, including erotic manga fan art (often labeled as NSFW – Not Safe For Work). It provides a platform for artists and enthusiasts to share and discover a wide range of artwork. Danbooru hosts a large collection of images, including various styles and genres, which can be used as a source of inspiration or training data for image generation models.

Danbooru Tag
Danbooru Tag refers to a system of keywords or tags applied to images on the Danbooru website. These tags describe the content, characteristics, and themes depicted in the images. When using Stable Diffusion models trained on Danbooru images, users can reference these tags in their prompts to guide the generation process and specify the desired content or style for the generated images.

DDIM (Sampler)
DDIM stands for Denoising Diffusion Implicit Models, which is a type of sampling method used in image generation models. DDIM samplers are based on the principles of diffusion models and aim to denoise and refine generated images through an iterative process. By gradually reducing noise and improving image quality, DDIM samplers can generate visually appealing and realistic images.

Deep Learning
Deep learning is a subfield of machine learning that focuses on the development and application of artificial neural networks inspired by the structure and function of the human brain. Deep learning models are designed to learn and extract meaningful patterns and representations from large amounts of data. They are capable of automatically discovering complex relationships and features, making them particularly effective in tasks such as image and speech recognition, natural language processing, and decision-making.

Deforum
Deforum represents a community of AI image synthesis developers, enthusiasts, and artists. They produce Generative AI tools and are most well-known for a Stable Diffusion WebUI video extension of the same name.

Denoising/Diffusion
Denoising or Diffusion refers to the process where random noise (also referred to as Seed) is iteratively reduced until the final image is produced.

depth2img
depth2img is a tool that infers the depth of an input image using an existing model, and subsequently generates new images based on the inferred depth.

Diffusion Model (DM)
A Diffusion Model (DM) is a type of generative model that is used to generate data similar to the data on which it has been trained.

DPM adaptive (Sampler)
The DPM adaptive is a Diffusion Probabilistic Model (Adaptive) sampler. It overlooks the step count during the generation process.

DPM Fast (Sampler)
DPM Fast refers to the Diffusion Probabilistic Model (Fast) sampler.

DPM++ 2M (Sampler)
DPM++ 2M denotes a Diffusion Probabilistic Model – Multi-step (2M) sampler. This model is capable of producing high-quality results within 15-20 steps.

DPM++ 2M Karras (Sampler)
DPM++ 2M Karras is a Diffusion Probabilistic Model – Multi-step (2M) Karras sampler. It also produces high-quality results within a span of 15-20 steps.

DPM++ 25 a Karras (Sampler)
DPM++ 25 a Karras refers to the Diffusion Probabilistic Model – Single-step (25a) Karras sampler that yields good quality results within 15-20 steps.

DPM++ 25a (Sampler)
DPM++ 25a is a Diffusion Probabilistic Model – Single-step (25a) sampler. It is known to produce good-quality results within 15-20 steps.

DreamArtist
DreamArtist is an extension to WebUI that allows users to create trained embeddings to steer an image towards a particular style or figure. It is a PyTorch implementation of the research paper “DreamArtist: Towards Controllable One-Shot Text-to-Image Generation via Contrastive Prompt-Tuning” by Ziyi Dong, Pengxu Wei, Liang Lin.

DreamBooth
DreamBooth is a deep-learning image generation model developed by Google Researchers. Its main function is to fine-tune existing models (checkpoints) and can be used to create custom models based on a set of images.

DreamStudio
DreamStudio is a commercial web-based image generation service. It has been created by Stability AI and employs Stable Diffusion models for its operations.

Stable Diffusion Glossary Part 2 | Buzzwords and Terminology (D – G)

EMA
EMA stands for Exponential Moving Average. A full EMA Checkpoint model contains additional training data which is not necessary for inference (generating images). However, these full EMA models can be utilized to further train a Checkpoint.

Emad
Emad Mostaque is the CEO and co-founder of Stability AI, one of the companies integral to the development of Stable Diffusion.

Embedding
Embeddings refer to additional file inputs that assist in guiding the diffusion model to produce images that align with the prompt. These inputs can denote a graphical style, representation of a person, or an object. See also Textual Inversion and Aesthetic Gradient.

DreamArtist
DreamArtist is an extension to WebUI that enables users to create trained embeddings, directing an image towards a specific style or figure. It is a PyTorch implementation of the research paper “DreamArtist: Towards Controllable One-Shot Text-to-Image Generation via Contrastive Prompt-Tuning” authored by Ziyi Dong, Pengxu Wei, and Liang Lin.

DreamBooth
DreamBooth is a deep-learning image generation model developed by Google Researchers. It is designed to fine-tune existing models (checkpoints) and can be utilized to create custom models based on a specific set of images.

DreamStudio
DreamStudio is a commercial web-based image generation service developed by Stability AI, making use of Stable Diffusion models.

GFPGAN
GFPGAN stands for Generative Facial Prior, a facial restoration model specifically designed for correcting blurry, grainy, or disfigured faces.

Git (GitHub)
Git (GitHub) is a hosting service used for software development. It offers version control, bug tracking, and documentation facilities.

GPT-3
GPT-3, or Generative Pre-trained Transformer 3, is a language model. It employs machine learning to generate human-like text based on an initial prompt.

GPT-4
GPT-4, or Generative Pre-trained Transformer 4, is a language model similar to GPT-3. It utilizes machine learning to generate human-like text based on an initial prompt. It represents a significant leap in performance and reasoning capability over GPT-3.

GPU
GPU is the abbreviation for Graphics Processing Unit. It is a type of processor engineered to perform rapid mathematical calculations, enabling it to render images and videos for display.

Gradio
Gradio is a web-browser-based interface framework, particularly designed for Machine Learning applications. Auto1111 WebUI operates in a Gradio interface.

Stable Diffusion Glossary Part 3 | Terminology (H – O)

Hallucinations (LLM)
In the context of Large Language Models (LLM) like ChatGPT, “Hallucinations” refer to instances where the model produces information that seems plausible but is nonsensical or entirely false.

Hash (Checkpoint model)
A Hash in the context of a Checkpoint model refers to an algorithm for verifying the integrity of a file. It generates an alphanumeric string unique to the file in question. Checkpoint models are hashed, and the resulting string can be used to identify that specific model. For example, The Ally’s Mix always has the hash c77ef05d.

Heun (Sampler)
Named after Karl Heun, the Heun sampler is a numerical procedure for solving ordinary differential equations.

Hugging Face
Hugging Face is a community/data science platform that provides tools for building, training, and deploying machine learning models.

Hypernetwork (Hypernet)
A Hypernetwork or Hypernet is a method that guides a Checkpoint model towards a specific theme, object, or character based on its own content, without requiring external data.

img2img
The img2img process is used to generate new images based on an input image and a txt2img prompt.

Inpainting
Inpainting is the practice of removing or replacing objects in an image based on a painted mask.

LAION
LAION is a non-profit organization that provides datasets, tools, and models for machine learning research.

LAION-5B
LAION-5B refers to a large-scale dataset for research purposes, consisting of 5.85 billion CLIP-filtered image-text pairs.

Lanczos
Named after its creator, Cornelius Lanczos, Lanczos is an interpolation method used to compute new values for sampled data. In this context, it is used to upscale images.

Large Language Model (LLM)
A Large Language Model (LLM) is a type of Neural Network that learns to write and converse with users. Trained on billions of pieces of text, LLMs excel at producing coherent sentences and responding to prompts in the correct context. They can perform tasks such as re-writing and summarizing text, chatting about various topics, and conducting research.

Latent Diffusion
Latent Diffusion refers to a type of diffusion model that contains compressed image representations instead of the actual images. This model allows the storage of a large amount of data that encoders can use to reconstruct images from textual or image inputs.

Latent Mirroring
Latent Mirroring applies mirroring to the latent images mid-generation, producing anything from subtly balanced compositions to perfect reflections.

Latent Space
Latent Space is the information-dense space where the diffusion model’s image representation, attention, and transformation are combined to form the initial noise for the diffusion process.

LDSR
LDSR, or Latent Diffusion Super Resolution upscaling, is a method used to increase the dimensions or quality of images.

Lexica
Lexica.art is a search engine dedicated to stable diffusion art and prompts.

LlamaIndex (GPT Index)
The LlamaIndex or GPT Index allows the connection of text data to an LLM via a generated “index”. More information here.

LLM
An LLM, or Large Language Model, is a type of Neural Network that learns to write and converse with users. Trained on billions of pieces of text, LLMs excel at producing coherent sentences and replying to prompts in the correct context. They can perform tasks such as re-writing and summarizing text, chatting about various topics, and conducting research.

LOCON
LOCON, or Low-Rank Adaptation, is a method of training for SD, much like Textual Inversion. It can capture styles and subjects, producing better results in a shorter time, with smaller output files, than traditional fine-tuning.

Merge (Checkpoint)
Merge, in the context of a Checkpoint, refers to a process by which Checkpoint models are combined (merged) to form new models. Depending on the merge method (see Weighted Sum, Sigmoid) and multiplier, the merged model will retain varying characteristics of its constituent models.

Metadata
Metadata is data that describes data. In the context of Stable Diffusion, metadata is often used to describe the Prompt, Sampler settings, CFG, steps, etc., which are used to define an image and stored in a .png header.

MidJourney
MidJourney is a commercial web-based image generation service, similar to DALL-E, or the free, open-source, Stable Diffusion.

Model
In the context of machine learning and AI, a Model is an alternative term for Checkpoint.

Negative Prompt
A Negative Prompt refers to keywords which guide a Stable Diffusion prompt by indicating what we don’t want to see in the generated image.

Neural Network
Neural Networks are mathematical systems that emulate the human brain, with layers of artificial “neurons” helping to find connections between data.

Notebook
A Notebook, also known as a Colab, is a Jupyter notebook service that provides access, free of charge, to computing resources including GPUs.

NovelAI (NAI)
NovelAI (NAI) is a paid, subscription-based AI-assisted story (text) writing service. It also has a txt2img model, which was leaked and is now incorporated into many Stable Diffusion models.

Olivio (Sarikas)
Olivio Sarikas produces excellent SD content on YouTube, and is considered one of the best SD news YouTubers out there! Check out his channel [here](https://www.youtube.com/@Olivio Sarikas).

OpenAI
OpenAI is an AI research laboratory consisting of the for-profit corporation OpenAI LP and the non-profit OpenAI Inc.

OpenPose
OpenPose is a method for extracting a “skeleton” from an image of a person, allowing poses to be transferred from one image to another. It is used by ControlNet.

Outpainting
Outpainting is the practice of extending the outer border of an image into blank canvas space while maintaining the style and content of the image.

Stable Diffusion Glossary Part 4 | Keywords and Terms (P – U)

Parameters (LLMs)
Parameters refer to numerical points across a Large Language Model’s (LLM) training data. These parameters dictate the model’s proficiency at its tasks. For instance, a 6 Billion (6B) parameter model will likely perform less well than a model with 13 Billion (13B) parameters.

Pickle
In the context of AI and machine learning communities, ‘Pickle’ is a slang term for potentially malicious code hidden within models and embeddings. To be “pickled” means to have unwanted code executed on your machine (i.e., to be hacked).

PLMS (Sampler)
PLMS stands for Pre-Trained Language Models in the context of Samplers.

Prompt
In the context of Stable Diffusion, a Prompt refers to the text input that describes the specifics of the image you would like as output.

Pruned
‘Pruned’ is a term that refers to a method of optimizing a Checkpoint model. This optimization is done to increase the speed of inference (prompt generation), reduce the file size, and lower the VRAM cost.

Python
Python is a popular, high-level, general-purpose programming language widely used in various fields, including data science, machine learning, and web development.

PyTorch
PyTorch is an open-source machine learning library created by META (formerly Facebook). It provides a wide range of algorithms for deep learning and uses the scripting language Lua.

Questianon
Questianon is the author of the widely used SD Resource Goldmine. More info here.

Real-ESRGAN
Real-ESRGAN is an image restoration method known for its effectiveness and high-quality outputs.

SadTalker
SadTalker is a framework for creating facial animations and lip-syncing based on audio input. More info here.

Samplers
Samplers are mathematical functions that provide different ways of solving differential equations. Each Sampler will produce a slightly or significantly different image result from the random latent noise generation.

Sampling Steps
Sampling Steps refer to the number of steps spent on generating (or diffusing) an image.

SD 1.4
SD 1.4 is a latent text-to-image (txt2img) model. At the time of its release, it was the default model for Stable Diffusion. It has been fine-tuned on 225k steps at a resolution of 512×512 on the laion-aesthetics v2 dataset.

SD 1.5
SD 1.5 is an updated version of the SD 1.4 model. It’s a latent text-to-image (txt2img) model, fine-tuned on 595k steps at a resolution of 512×512 on the laion-aesthetics v2 dataset.

SD UI
SD UI is a colloquial term for Cmdr2’s popular graphical interface for Stable Diffusion prompting.

SDXL
SDXL refers to Stability AI’s latest Stable Diffusion Model as of March 2023. It’s not available for offline use and can only be used for inference via certain subscription websites.

Seed
In the context of image generation, a ‘Seed’ is a pseudo-random number used to initialize the generation of random noise, from which the final image is built. Seeds can be saved and used along with other settings to recreate a particular image.

Shoggoth Tongue
A term coined in a humorous allusion to the language of the fictional monsters in the Cthulhu Mythos, “Shoggoth Tongue” refers to advanced ChatGPT commands. These commands can be particularly arcane and difficult to understand, but allow ChatGPT to perform advanced actions outside of the system’s intended operation.

Sigmoid (Interpolation Method)
Sigmoid is a method for merging Checkpoint Models. It’s based on a Sigmoid function – a mathematical function that produces an “S”-shaped curve.

Stability AI
Stability AI is a technology company specializing in AI, co-founded by Emad Mostaque. It is one of the companies behind the development of Stable Diffusion.

Stable Diffusion (SD)
Stable Diffusion is a deep learning, text-to-image model that was released in 2022. Its primary use is to generate detailed images based on provided text descriptions.

SwinIR
SwinIR is an image restoration transform. Its main purpose is to restore high-quality images from those of low quality.

teachyou.ai
Teachyou.ai is an alternate link to TheAlly’s Patreon. TheAlly, the author of this glossary, has created popular models and tutorials for Stable Diffusion.

Tensor
A Tensor is a container used for storing multi-dimensional data in the field of machine learning and deep learning.

Tensor Core
Tensor Core is a processing unit technology developed by Nvidia. It is designed specifically to carry out matrix multiplication, a crucial arithmetic operation in machine learning algorithms.

Textual Inversion
Textual Inversion is a technique used for capturing concepts from a small number of sample images. These captured concepts can then influence text-to-image (txt2img) results towards a particular face, object, or theme.

TheAlly
TheAlly is the creator of the popular TheAlly’s Mix models and Stable Diffusion (SD) tutorials. They are also the owner of teachyou.ai. More info here.

Token
In the context of machine learning and natural language processing, a Token is roughly equivalent to a word, punctuation mark, or a Unicode character in a prompt.

Tokenizer
A Tokenizer refers to the process or model through which text prompts are converted into tokens for processing.

Torch 2.0
Torch 2.0 is the latest version (as of March 2023) of PyTorch, an open-source machine learning library developed by Facebook’s AI Research lab.

Training Data
Training Data refers to a set of many images used to “train” a Stable Diffusion model or embedding. The model learns from this data to produce accurate outputs.

txt2img
Txt2img refers to a model or method of image generation via entry of text input.

txt2video
Txt2video refers to a model or method of video generation via entry of text input.

UniPC (Sampler)
UniPC is a recently released (as of March 2023) Sampler based on the Hugging Face Diffusers API Schedulers, and used in Stable Diffusion.

Upscale
Upscaling refers to the process of converting low-resolution media, such as images or video, into higher-resolution media.

UniPC (Sampler)
UniPC is a sampler used in Stable Diffusion. It’s a type of mathematical function that influences the way an image is generated during the diffusion process.

Stable Diffusion Glossary Part 5 | Terminology (V – Z)

VAE (Variational Autoencoder)
A Variational Autoencoder is a type of AI model that generates new data that’s similar to its training data. In Stable Diffusion, a .vae.pt file accompanies a Checkpoint model, providing additional detail improvements. Not all Checkpoints have an associated VAE file, and some VAE files are generic, which means they can be used to enhance any Checkpoint model.

Vector (Prompt Word)
In Stable Diffusion, a Vector is an attempt to represent the meaning of a word mathematically for processing. It’s a numerical interpretation of a word, which aids in generating more accurate outputs.

Venv (Virtual Environment)
A Python Virtual Environment, or Venv, allows multiple instances of Python packages to run independently on the same PC. This can prevent potential conflicts between different versions of the same Python package used by different software.

Vicuna
Vicuna is an open-source chatbot model. It was established by students and faculty from UC Berkeley, in collaboration with UC San Diego and Carnegie Mellon University. More info here.

Vladmandic
Vladmandic is a fork of Auto1111’s WebUI, featuring its own unique set of capabilities. It gained popularity around May 2023. You can find more about it on its GitHub page.

VRAM (Video Random Access Memory)
VRAM is a type of dedicated graphics card memory used to store pixels and other graphical processing data for display. The capacity of VRAM can influence the performance of graphic-intensive applications and tasks.

Waifu Diffusion
Waifu Diffusion is a popular text-to-image model specifically trained on high-quality anime images. It’s renowned for producing beautiful anime-style image outputs.

WebUI
WebUI is a colloquial term for Automatic1111’s WebUI, a popular graphical user interface for Stable Diffusion prompting.

Weighted Sum (Interpolation Method)
The Weighted Sum is an interpolation method used for merging Checkpoint Models. The method uses the formula Result = (A * (1 – M)) + (B * M), where A and B are the models being merged, and M is a weight parameter.

Weights
Weights are text files containing terms such as clothing types, cities, weather conditions, and so on. These can be automatically input into image prompts, allowing for the generation of a wide variety of dynamic images.

Wildcards
Wildcards is an optional library designed to speed up image generation. However, it has been somewhat superseded by new options implemented in Torch 2.0.

xformers
The term ‘xformers’ refers to transformers, which are models used in machine learning for various tasks. These tasks include natural language processing, image generation, and more.

yaml (YAML Ain’t Markup Language)
Yaml is a human-readable data-serialization programming language. It’s commonly used for configuration files. In Stable Diffusion, yaml files accompany Checkpoint models, providing additional information about the Checkpoint.

Frequently Asked Questions about Stable Diffusion Glossary Terms and Terminology