Ai generative art tools

Text to Image

There are a TON of shared Google Colab notebooks floating around for doing text to image with pre-trained GAN and diffusion models. I've been compiling the ones I come across and try out and find interesting. Please hit me up on twitter (@pharmapsychotic) if you know a cool notebook that I am missing! Stable Diffusion is most popular right now. 

• Stable Diffusion WebUI by automatic1111 - run SD local with lots of features and extensions
• StableStudio - local webui using Stability API for inference
• Deforum Stable Diffusion 0.7 - group effort for ultimate SD notebook (discord) (youtube tutorial) (guide)
• Disco Diffusion v5.6 by Somnai, gandamu, zippy721 (guide) (new guide) (youtube tutorial)
• Huemin Jax Diffusion 2.7 by nshepperd, huemin_art (guide) (stitching guide)
• pytti-tools v0.10 by DigThatData and sportsracer
• VQGAN+CLIP by remi_durant

• [2023/09/15] llamas.ipynb by @osanseviero QR ControlNet + SD1.5 for optical illusions ( tweet )
• [2023/04/28] DeepFloyd IF (huggingface) (github)
• [2023/04/05] Kandinsky 2.1 Batching+Dynamic prompting Colab by @jrobocat
• [2023/04/03] Kandinsky 2.1 (huggingface) (site)
• [2023/03/23] Image-to-text-to-image Colab by @jrobocat  - batch CLIP Interrogator + SD generations
• [2023/03/20] ModelScope text-to-video Colab by @camenduru (youtube) (github)
• [2023/03/18] ModelScope text-to-video huggingface space
• [2023/03/14] Unidiffuser - unified diffusion framework (github)
• [2023/02/20] Stable Diffusion Auto Stitching by @oleg_ai_art (guide)
• [2023/02/15] ControlNet - control Stable Diffusion with extra conditioning (youtube) (huggingface) (github) (models)
• [2023/02/14] Pix2Pix video with coherence by ​@johnowhitaker - stylize video inputs!
• [2023/01/30] Tune-a-Video - create short text2video sequences (github) (paper)
• [2023/01/21] KLMC2 Animation - @DigThatData's fork with lots of additions
• [2023/01/20] InstructPix2Pix - use text instructions to modify images (huggingface)
• [2023/01/19] Image Mixer by @Buntworthy - mix up to 5 images together with SD
• [2023/01/14] Latent Blending by @j_stelzer - smooth transition between SD latents (github)
• [2023/01/10] Custom Diffusion - fast SD finetune with multiple concepts (github)
• [2022/12/22] Karlo - unCLIP architecture like DALLE-2 (huggingface) (github)
• [2022/12/08] Stable Diffusion KLMC2 Animation by @RiversHaveWings
• [2022/11/30] BAOAB-limit sampler - new SD sampler that can also make anims hella fast (paper)
  
• [2022/11/25] Stable Diffusion 2.0 Web UI - by @anzorq (run SD 2.0 in colab using Diffusers)
• [2022/11/24] Stable Diffusion 2.0 w Diffusers - by @amrrs (youtube)
• [2022/11/08] Midjourney v4 Style - (dreambooth SD finetune on midjourney v4 outputs)
• [2022/11/03] All-in-one Private Diffusions Colab - fork and upgrades to WD notebook (website)
• [2022/10/25] Fast Dreambooth by TheLastBen (easy fast finetune of stable diffusion in colab)
• [2022/10/08] Stable Worlds by @NaxAlpha (create panoramas with SD!)
• [2022/09/29] MathRockDiffusion by ethansmith2000 (mods and improvements on Disco) ( guide )( cuts )
• [2022/09/29] robo_diffusion_v1 by @nousr (a DreamBooth fine tune of stable diffusion)
• [2022/09/27]  Video Killed The Radio Star Diffusion by @DigThatData (transform music videos from YouTube)        
• [2022/09/25] fast-stable-diffusion - automatic111 ui, hlky ui, github (+25% speed and low VRAM)
• [2022/09/18] Doohickey Diffusion by aicrumb (stable diffusion with CLIP guidance, perlin init, lots more)
• [2022/09/18] optimized colab by neonsecret (stable diffusion with nice gradio gui in colab)
• [2022/09/13] Stable Diffusion Batch by visoutre (includes tiled upscaling!) (tutorial)
• [2022/09/11] Easy Diffusion by WASasquatch and NOP (stable diffusion with lots of still image features)
• [2022/09/07] NMKD Stable Diffusion GUI (nice easy Windows GUI for stable by Noomkrad)
• [2022/08/30] Simple Stable Diffusion by @ai_curio (supports prompt weighting)
• [2022/08/29] Stable Diffusion WebUi by @altryne (fancy Gradio UI for stable diffusion)
• [2022/08/28] Prompt Parrot v2.0 by @KyrickYoung (train gpt2 on prompt list then generate with stable-diff)
• [2022/08/23] Stable Diffusion Interpolation by @ygantigravity (animate from own prompt to another!)
• [2022/08/23] Deforum Stable Diffusion (discord link) 🔥
• [2022/08/23] FunkyHorses Stable Diffusion by  Coskaiy/Corran (has neat import from spreadsheet)
• [2022/08/23] NOP's Stable Diffusion Colab v0.19 by NOP#1337
• [2022/08/23] Stable Diffusion Lite by @future__art (prompt queueing and seed mining)
• [2022/08/23] Interactive notebook for Stable Diffusion  
• [2022/08/22] Stable Diffusion HuggingFace space by stabilityai
• [2022/08/22] Stable Diffusion notebook by @pharmapsychotic 🔥 (easy to use and batch to gdrive) (tutorial)
• [2022/08/22] Official Stable Diffusion notebook - requires hugging face account
• [2022/08/22] DiscoStream v1.1 by @WASasquatch
• [2022/08/20] Disco Diffusion v5.6 with Inpainting by @cut_pow
• [2022/08/18] DiscoArt [w/ Batch Prompts + GPT3 generator] by Skquark
• [2022/08/16] WAS's Disco Diffusion v5.6-9 Portrait Generator Playground by WASasquatch
• [2022/08/08] Paint Pour Diffusion by @EclecticBeams (diffusion trained on paint pour art)
• [2022/07/31] Huemin Jax Diffusion 2.7 August 2022 by @huemin_art
• [2022/07/30] CLIP Prior + VQGAN by @RiversHaveWings and @jd_pressman (a new VQGAN notebook 😮)
• [2022/07/23] Textile Diffusion by @KaliYuga (diffusion trained on textiles)
• [2022/07/21] Floral Diffusion by @jags111 (fine tunes for floral)
• [2022/07/18] Liminal Diffusion v1 by @BrainArtLabs (diffusion trained on liminal photographs)
• [2022/07/18] DifNESfusion 1.35 by @LufiQ (fork or PixelArtDiffusion with NES dataset)
• [2022/07/18] Medieval Diffusion by @KaliYuga (diffusion trained on medieval art)
• [2022/07/17] FeiArt_Handpainted CG Diffusion by @FeiArt_AiArt
• [2022/07/17] Fantasy Diffusion by @LaVista (diffusion trained on fantasy art)
• [2022/07/15] Ukiyo-e Portrait Diffusion by @avantcontra
• [2022/07/15] Lithography Diffusion by @KaliYuga (diffusion trained on lithographic landscapes and portraits)
• [2022/07/06] Disco v5.2 Dynamic Prompting (dynamic prompt variations -  tutorial video )
• [2022/07/06] Watercolor Diffusion by @KaliYuga (diffusion trained on watercolor paintings)
• [2022/07/05] EnzymeZoo edits to Huemin Jax Diffusion by @EnzymeZoo (brought over masking from Majesty)
• see older notebooks in the archive

Upscaling / Super-resolution

Check out the Upscaling Guide

• Gigapixel AI by Topaz Labs (costs $99) <- voted #1
• Real-ESRGAN - ( github ) <- voted #2
• Real-ESRGAN Sber - a nice fine tuned ESRGAN model
• chaiNNer - node base tool that can batch process ESRGAN upscale and more
• Cupscale - Windows GUI for ESRGAN
• Latent-SR - Nightmare Ai latent diffusion super resolution (slow but nice!)
• PASD image super resolution - (github) pixel aware Stable Diffusion
• Neural Love - credit based system for diffusion upscaling 
• Stable Diffusion Upscaler - latest and greatest 🔥
• SuperRes Diffusion - Batch upscaling and super resolution with latent-diffusion
  
• SwinIR - Hugging Face space
  
• Upscale Model Database - big set of pretrained models for upscaling different types of content
• Waifu2x (github) - designed for anime / manga
• WaifuXL - newer and beats Waifu2x in quality
• LetsEnhance.io - credit based web service for image super resolution

StyleGAN

• [2022/08/23] Painting with StyleGAN by @jmoso13 (tutorial) - use VAE to navigate and animate!
• [2022/04/25] StyleGAN-Humans + CLIP modified by Diego Porres to use StyleGAN3
• StyleGAN2-ADA - train your own StyleGAN2 model from an image set you create
• StyleCLIP - Text-drive manipulation of StyleGAN imagery
• Structured Dreaming - Styledreams With helpers
• Structured Dreaming (CLIP+StyleGAN) by @ArYoMo (tweet)
  
• StyleGAN 2 pretrained models - can use these with Structured Dreaming
• StyleGAN 2 awesome pretrained models - BIG collection of models
• StyleGAN 3 training - train a StyleGAN and do interpolation video by @dvsch (currently busted)
• StyleGAN 3 music video generation - (tweet)
• StyleGAN 3 + CLIP by Annas
• StyleGAN3 + CLIP by @nshepperd1 and @RiversHaveWings
• StyleGANXL + CLIP by Eugenio Herrera and Rodrigo Mello
• Lucid Sonic Dreams - animate path through StyleGAN latent space with music (github)
  

Text

• GPT4All Chat - run local windows/linux/mac app like ChatGPT
• oobabooga text-generation-webui - it's like auto1111 sd web ui but for text models
• StableLM space - huggingface space for language model from Stability AI

• Goose.ai Playground - can use their playground to generate text with GPT-Neo
• GPT Neo Colab notebook - use GPT-neo 1.3B and 2.7B from Google colab
• GPT Neo HuggingFace - run GPT-neo 2.7B on HuggingFace
• Neuralism Generative Art Prompt Generator - generate prompts to use for text to image
• OpenAI GPT3 Playground - generate text with GPT-3 (requires free account)
• Textsynth Playground - text completion using large language models

• Dalle-2 Prompt Generator - nice site that let's you generate interesting text prompts
• Prompt Parrot by @KyrickYoung - train GPT2 on a list of your prompts
• MadLib Prompt Generator - makes interesting prompts for you, by @remi_durant
• Noodle Soup Prompts v2.1 by WASasquatch
  
• Neuralism Prompt Generator - generative art prompt generator

Video

Image to video

• [2023/11/22] Stable Video Colab by @mkshing

Text to video

• camenduru text-to-video Colabs - great collection of Zeroscope, potat1, modelscope notebooks
• AnimateDiff (colab) (github) - short video clubs with your own LoRA
• ModelScope (colab) (huggingface) - super fun but prominant shutterstock watermarks
• Text2Video-zero (colab) (github) (huggingface) (webui ext) - zero shot video from Stable Diffusion

Interpolation

• Video Enhance AI by Topaz Labs - commercial upscaling and frame interpolation <- excellent
• AnimationKit AI - video upscaling and interpolation tool <- great
• FILM colab - by @KyrickYoung has pause, loops, reverse <- my fave FILM
  
• 3D Ken Burns Effect from single image - animated video from 2D image
  
• 3D Photo Inpainting - cool 3D effects for 2D images
• Animating Pictures with Eulerian Motion Fields - code not out yet, looks like it'll be awesome
• DAIN colab - depth aware interpolation
• EbSynth - stylize video by giving it ai or hand painted key frames from video
• ESRGAN 4 Video - increase resolution of video with ESRGAN
  
• FILM: Frame Interpolation for Large Motion - (replicate link) smooth interpolation/morphing
  
• Flowframes - free Windows tool with patreon option, uses RIFE and other models
  
• PyTTI-Tools: FILM - @DigThatData 's version of FILM for video frames
• RIFE - smooth interpolation of video to increase frame rate
  
• Sequence Frame Interpolation - batch version of FILM
• Super Slomo - another way to increase frame rate of video
• Video Art and Styling Tools - by @Coskaiy (style transfer, interpolation, superres, and more)

Animation

• [2022/11/03] FrameSync.xyz - Automate Deforum Keyframe animations with waveforms
• [2022/10/26] Tulpa Prompter by @dreamingtulpa - helper to build animation prompts (tweet)
• [2022/08/15] AnimationPreview by @pharmapsychotic - quickly preview Deforum camera animations
• [2022/08/04] DALL-E 2.5D Depth Warped Zoom by @deKxi
• [2022/03/31] PyDub Audio to Disco Diffusion Keyframe Generator v0.1 by austinhquinn
• [2022/02/26] Wiggle animation key frame generator by @zippy731 
• [2022/02/23] audio-reactive-video - by @vsewall2motion, skip video frames based on volume
  
• Keyframe string generator for AI animation notebooks
• Audio to keyframe string generator for AI animation notebooks
  

Prompt Engineering

To get good results with CLIP guided diffusion and VQGAN+CLIP you need to find the right words and phrases that will direct the neural network to the content and style you are looking for.

Image to Text

• Antarctic-Captions by @dzryk
• BLIP image captioning HuggingFace space
• CLIP Interrogator by @pharmapsychotic - image to prompt! (huggingface) (lambda) (replicate) 🔥
  
• CLIP prefix captioning inference notebook (github)
• LLaVa: Large Language and Vision Assistant - ask vision model to describe image
• personality-clip by @dzryk
  
• PEZ: Prompts made EZ - prompt from image or long to short prompt  (huggingface) (colab)

Prompt Guides

• [2023/08/04] Stable Diffusion XL reference library - great guides for SDXL!!
• [2022/11/29] Stable Diffusion V2 CFG Scale Comparison - nice ref of samplers and cfg scale
• [2022/09/16] krea.ai search stable diffusion prompts and browse by modifiers
• [2022/09/07] libraire.ai search 10 million stable diffusion prompts and images
• [2022/08/28] Prompt Parrot v2.0 by @KyrickYoung (train gpt2 on prompt list then generate with stable-diff)
  
• [2022/08/24] Lexica stable diffusion prompt search engine
• [2022/08/13] Promptomania by @wszp - cool prompt building tool!
• [2022/08/08] Stable Diffusion Artist Studies by @proximasan @EErratica @KyrickYoung @surrailabs
• [2022/08/08] Stable Diffusion Modifier Studies by @proximasan +
  
• [2022/07/30] Disco Diffusion Portrait Study by @enviraldesign
• [2022/07/13] Dall-e 2 prompt book by @GuyP
• [2022/03/25] Disco Diffusion Modifiers Study by @KyrickYoung and @sureailabs
• [2022/03/21] DiscoDiffusion Model Comparison Study - by @KaliYuga
• [2022/03/05] Midjourney Artist Dump - spreadsheet or artists and example render
• [2022/02/26] Disco Diffusion 70+ Artist Studies
• A Guide to Writing Prompts for Text to Image - Google Doc guide and advice
  
• CLIP Retrieval Tool - see what kinds of images match strings for CLIP (wait a long time for it to load)
• CLIP Prompt Engineering for Generative Art - nice long guide by Matthew McAteer
• CLIP + VQGAN keyword comparison by @kingdomakrillic
• Artist Studies by @remi_durant - big collection of results using different artist names
• Art Movements and Styles as perceived by VQGAN + Clip (Imagenet 16k, ViT-B/32)
• Art Movements and Styles as perceived by VQGAN + Clip (Imagenet 16k, ViT-B/16)
  
• Art Movements and Styles as perceived by VQGAN + Clip (Imagenet 16k, RN50x16)
  
• Art Movements and Styles as perceived by VQGAN + CLIP (Imagenet 16k, RN50x4)

Other

• sdtools.org - cool wiki covering tools and methods related to Stable Diffusion
• JAX CLIP Guided Diffusion 2.7 Guide - Google doc from huemin
• Zippy's Disco Diffusion Cheatsheet - Google Doc guide to Disco and all the parameters
• EZ Charts - Google Doc Visual Reference Guides for CLIP-Guided Diffusion (see what all the parameters do!)
• Hitchhiker's Guide To The Latent Space  - a guide that's been put together with lots of colab notebooks too
• Resources for GAN Artists - another big Google Doc with notebooks and resources for AI art
• Way of the TTI Artist - pytti guide 
• Guide to install Disco Diffusion 5 on Windows with WSL - haven't tried this yet challenge is pytorch3d
• Great explanation of VQGAN+CLIP - https://ljvmiranda921.github.io/notebook/2021/08/08/clip-vqgan/
• Nice overview of lots of different optimization algorithms SGD, Adam, RMSProp etc and their differences (also covered in this lecture)
• Stanford's Convolutional Neural Networks class on YouTube - https://www.youtube.com/playlist?list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv 

• ClipMatrix - text controlled 3D mesh deformation and stylization
  
• CLIP-Mesh - text to 3D mesh with texture and normal map (still pretty simple and mixed results)
• DreamFields - latest text to 3D (github)
• ImageSorter by @pharmapsychotic - sort images by similarity (nice for StyleGAN/FiLM animated loops)
• PIFuHD Colab - Human photo to 3D mesh of the human
  
• Point-E - OpenAI's text to 3d point clouds (github)
• text2mesh - Kaggle notebook for text to 3D mesh
• Watermark images - little notebook to add text watermark to images
• Zero-Shot Text-Guided Object Generation with Dream Fields - text to 3D render
  

Learn to Code Generative Ai

• The Illustrated Stable Diffusion - really nice overview of Stable Diffusion and the pieces that make it up
• AIAIART - really nice ongoing youtube series and discussion in its Discord
  
• Deep Learning for Art, Aesthetics, and Creativity - MIT course available on youtube
  
• Dive into Deep Learning (online, free, interactive)
  
• Deep Learning Foundations to Stable Diffusion - 4 videos from the fast.ai class
• Generative Deep Learning: Teaching Machines to Paint, Write, Compose, and Play by David Foster [2019]
  Really enjoyed this and it's a great book! It's from 2019 so doesn't cover the very latest like VQGAN, CLIP, guided diffusion though.
• HuggingFace Diffusion Models Class - nice coverage of the diffusers library and Stable Diffusion 
• The Artist in the Machine: The world of AI-powered creativity by Arthur I. Miller [2020]
  Not very technical but engaging and inspiring view of many Ai art projects so far.
• ml4a.net - online textbook, classes, and learning resources

Cool Apps

No Code AI Art tools

• Artbreeder - StyleGAN model with "genes" (directions in latent space) for editing
• Artbreeder Collage - CLIP guided diffusion on top of simple collages
• Astria.ai - nice and easy Dreambooth training - upload images, and get finetuned SD model
• BlueWillow - text to image Discord like MidJourney (appears to use Stable Diffusion finetunes)
• CogView - text to image, Chinese model like DALL-E ( interview )
• conjure.art - new text to image site currently in beta
• craiyon - formerly known as dall-e mini, free and makes quick grids of 9 outputs
• Dall-e 2 - OpenAI's text to image
• DeepDreamGenerator - deep style, thin style, deep dream
• DreamStudio - easy to use text to image from creators of Stable Diffusion 🔥🔥🔥
• Genmo - short animations (looks like KLMC2)  🎥
• Kaiber - create short animations (looks like Deforum) 🎥
• midjourney - text to image via discord bot  🔥🔥🔥
• murf.ai - text to speech with Ai voices
• neural.love image-upscale - credit based image upscaling service
• NightCafe - style transfer, VQGAN, diffusion image generation
• Ostagram - style transfer
• Playform - style transfer, train stylegans, images morphs
• pollinations.ai - run lots of popular notebooks
• ProsePainter - interactive tool to "paint with words"
• runwayml - video editing powered by AI 🎥
• snowpixel - text to image and variations
• StarryAI - text to image with easy selection of styles
• synth.run - text to image app for iOS, Android, and web
• tokkingheads - animate portraits with Ai
• uberduck.ai - text to speech with lots of different voices
• Visions of Chaos - run the popular AI notebooks locally on Windows (see the Machine Learning setup steps)
• Wombo -  Super fast and free
• wzrd.ai - give it a music file and produce animation from big set of pretrained GANs

Create Game Assets

• Layer - 2D assets and variations for games
• Luma AI Imagine 3D - alpha test of text to 3D models
• MirageML - 3D assets and prototyping
• Scenario.gg - AI-generated game assets
• withpoly - AI-generated textures and materials
  

Online Galleries to Showcase Art

• OnCyber art galleries - https://oncyber.io - Cool 3D art gallery to showcase your art with links to NFT market
• Spatial - https://spatial.io

Has this guide been helpful to you? 🙂 You can follow me on twitter @pharmapsychotic and I'd love if you would