Five Penguins × Audrey Heyman

How Audrey and I made an animation out of 120 paintings:

1. Made a 3D animation that includes things she paints best
2. She painted one frame 48” x 16” acrylic on canvas
3. I trained an AI model on her painting
4. Used the AI to generate all frames of the animation in her style

First we created a 3D animation together using C4D. We wanted to make a primo version of a Five Penguins squad using elements she paints well. She’s from Albuquerque and her painting style uses New Mexican themes and landscapes so we went with a mountainous desert environment.

Once we finished the animation, she painted a frame from it. She painted this over a few months using acrylic and it’s on a nice big 48” x 16” canvas.

Next I used Stable Diffusion to train an “Audrey style” using textual inversion and about 20 images of the painting. I used wide shots and closeups, trying to capture a wide variety of details.

Then I used img2img to process the frames from the 3D render through SD to redraw them in the style of her painting. The prompt I used was: “a group of penguins walking along a road near mountains and a lake with hot air balloons flying in the sky, by (audrey5pengstyle_TI:1.1)”

After getting the raw output from SD, I doubled the resolution in Topaz AI and then did some compositing in After Effects. I wanted to bring back the balloons and also add texture and details to make it feel more like it was actually animated from 120 paintings on canvas.

The whole process was thrilling to us because we felt like we were able to use AI as a tool to enhance our own artistic capabilities, allowing us to make something that otherwise would have taken many months or even years to complete. Hope you enjoy 🐧