AI Graphics Python

An issue with AI image generation is that faces often look very similar, and you see the same characters turning up in images made in completely different contexts, by different people. Although it seems like we have a lot of ways of describing faces, we can't describe them in much detail when it comes to the exact shape of the features, so an image generator doesn't get much entropy from the prompt for the face itself. And even with a particular face in mind, it's very difficult to describe it exactly enough for it to come out of the image generator.

The solution is to generate faces from seeds, using a dedicated face generation application. Each seed gives a different person, and an image or video model can then make the rest of the picture around the face.

The application

FaceMaker is a Mac app that shows a grid of faces, each made from a seed by one of five face models. Everything runs on the Mac itself, with no server and no internet connection, through Microsoft's ONNX Runtime.

FaceMaker with StyleGAN2 FFHQ: ten faces from seeds 1000 to 1009, and the controls under them
FaceMaker with StyleGAN2 FFHQ: ten faces from seeds 1000 to 1009, and the controls under them
  • The grid shows any number of faces, from 4 to 100, from the seed and the seeds after it. Regenerate picks a new seed, and each face has its own re-roll button.
  • A face can be opened in its own window to zoom in on it, saved, copied, or dragged straight out into another app as a PNG. Right-clicking one copies its seed, saves its latent vector as a file, or uses the vector as the starting point for more faces.
  • The model is chosen from the menu, and its controls appear below: directions, attributes or face parts, depending on what the model has.
  • A photo dropped on the window, or opened with the app, finds the closest face the model can make.

The models

There are five face generators, each of which takes an input vector and turns it into a face image:

ModelSize and controlsDownload
MobileStyleGAN FFHQ
by Sergei Belousov
Up to 1024px
No controls
mobilestylegan_ffhq.ckpt
(Apache 2.0 code, trained to copy StyleGAN2 FFHQ)
StyleGAN2 FFHQ
by NVIDIA
Up to 1024px
Age, gender, smile, glasses, pose
stylegan2-ffhq-config-f.pkl, with InterFaceGAN's directions
(NVIDIA licence: research or evaluation only)
StyleGAN2 MetFaces
by NVIDIA
Up to 1024px
No controls
metfaces.pkl
(NVIDIA licence: research or evaluation only)
GAN-Control FFHQ
by Amazon
Up to 512px
Identity, expression, pose, light, age, hair colour
controller_age015id025exp02hai04ori02gam15
(Apache 2.0 code, FFHQ non-commercial)
SemanticStyleGAN CelebAMask-HQ
by ByteDance
Up to 512px
28 face parts
CelebAMask-HQ-512x512.pt
(CC BY-NC-SA 4.0)

And there are two networks that work the other way, taking a face image in. The e4e encoder turns a face image into the vector StyleGAN2 FFHQ takes as its input, so that StyleGAN2 draws that face. ArcFace measures how alike two faces are:

ModelInput and jobDownload
e4e encoder
by Omer Tov and others
A 256px face image
Turns it into StyleGAN2 FFHQ's input
e4e_ffhq_encode.pt
(MIT code, FFHQ non-commercial)
ArcFace
by InsightFace
Two 112px face images
How alike they are
buffalo_l.zip, for w600k_r50.onnx
(InsightFace non-commercial research)

ArcFace is how FaceMaker finds a face image in a model's latent space, and the only way for the four models that have no encoder. It starts from the closest of 200 faces the model has already made. Then it tries six small random changes to the vector around that point, generates each face and keeps any change that ArcFace says is more alike, taking bigger steps while that keeps working and smaller ones when it doesn't, so it moves through the latent space until it settles where no nearby change brings the face any closer.

These are the authors' original checkpoints, mostly PyTorch, so they can't be run as they are. models/download.sh downloads every one of them from where its authors publish it, and converts each one into the ONNX files the app reads, using the authors' own model code. It takes about 15 minutes and 5 GB of downloads, including PyTorch, and leaves 2 GB of models. Nothing is redistributed, so each model keeps its own licence. All of them are either limited to non-commercial use or trained on data that is, and NVIDIA's licence is narrower still: research or evaluation only.

Here are ten faces from each, from the same seeds, 1000 to 1009. The times are for making all ten, on a MacBook Pro with an M2 Max (12 CPU cores) and 64 GB of memory, on the CPU:

MobileStyleGAN FFHQ

Ten faces from MobileStyleGAN FFHQ
Ten faces from MobileStyleGAN FFHQ: 5.2 seconds, 0.52 each

MobileStyleGAN is a small network trained to copy StyleGAN2 FFHQ, so the same seed gives the same person as StyleGAN2, in half the time. The faces are a little softer, and the backgrounds and edges less detailed, but for picking a face it's hard to tell them apart. It has no controls apart from the seed and the latent vector.

StyleGAN2 FFHQ

Ten faces from StyleGAN2 FFHQ
Ten faces from StyleGAN2 FFHQ: 10.6 seconds, 1.06 each

NVIDIA's StyleGAN2, trained on FFHQ's 70,000 photos from Flickr, is the best quality of the five, with a wide spread of ages and faces. Every so often one comes out broken, like 1005 here, half lost in a costume. It has five directions to edit along, described under Latent space.

StyleGAN2 MetFaces

Ten faces from StyleGAN2 MetFaces
Ten faces from StyleGAN2 MetFaces: 10.6 seconds, 1.06 each

The same network, trained on 1,336 portraits from the Metropolitan Museum of Art: paintings, drawings and sculpture. The faces come in every style from oils to pastels to an Egyptian mask, which makes it good for faces that don't need to look like photos. It has no controls.

GAN-Control FFHQ

Ten faces from GAN-Control FFHQ
Ten faces from GAN-Control FFHQ: 5.9 seconds, 0.59 each

Amazon's GAN-Control is trained on the same photos, at 512 pixels, so it's twice as fast as StyleGAN2. The faces vary more in pose, down to full profiles, and are a little less polished. Its strength is its controls: pose in degrees, age in years, hair colour, lighting and expression, which work much better than StyleGAN2's directions.

SemanticStyleGAN CelebAMask-HQ

Ten faces from SemanticStyleGAN CelebAMask-HQ
Ten faces from SemanticStyleGAN CelebAMask-HQ: 6.2 seconds, 0.62 each

SemanticStyleGAN was trained on CelebAMask-HQ, 30,000 photos of celebrities, and it shows: the faces are clean and well lit, but they're nearly all young, glamorous and made up, much like the faces a general image generator tends towards. It's the least varied of the five. What it has instead is control over each part of the face separately.

Generating faces

All five generators work the same way. A seed is turned into 512 random numbers, called z. A small mapping network, eight layers deep, turns z into W, the latent vector that the rest of the model is built around. The synthesis network, which is most of the model, then draws the face from W.

From a seed to a face: z, the mapping network, W, and the synthesis network
From a seed to a face: z, the mapping network, W, and the synthesis network

The 512 numbers come from the seed with a small random number generator and the Box-Muller transform, which turns evenly spread random numbers into the bell-shaped spread the models were trained on:

static func gaussianLatent(seed: UInt64, count: Int = 512) -> [Float] {
    var rng = SplitMix64(state: seed == 0 ? 0x1234_5678_9ABC_DEF : seed)
    var values: [Float] = []
    values.reserveCapacity(count)
    while values.count < count {
        var u1 = 0.0
        repeat { u1 = unit(&rng) } while u1 <= .leastNonzeroMagnitude
        let u2 = unit(&rng)
        let radius = sqrt(-2 * log(u1))
        let angle = 2 * Double.pi * u2
        values.append(Float(radius * cos(angle)))
        if values.count < count {
            values.append(Float(radius * sin(angle)))
        }
    }
    return values
}
FaceLocal.swift: a seed to z

The same seed always gives the same 512 numbers, and the models' noise inputs are fixed, so a seed always gives the same face, on any Mac. That's what makes a seed worth writing down.

W is 512 numbers too, but the synthesis network takes a copy of it for each layer: 18 rows at 1024 pixels, 16 at 512. Giving different layers different rows mixes two faces, the coarse layers deciding the shape and pose and the fine layers the colouring. That's called W+, and it's how a photo is matched, below.

Latent space

The space of all the Ws is called latent space. Faces close together in it look alike, and moving through it changes the face smoothly. The controls a model has are learned directions in latent space: lines along which one thing about the face changes, so that moving W along them changes that and, ideally, nothing else.

Every slider has a Range tick box. With it ticked, the grid shows the same face over and over, each with a random value between the slider and a second value, and each face is labelled with its value instead of its seed. That's the quickest way to see what a slider does.

StyleGAN2 FFHQ's five sliders are directions found by InterFaceGAN. A classifier sorted thousands of generated faces, for example into smiling and not smiling, and a straight boundary was fitted between the two groups in W. The direction across that line is the smile, and the slider adds a multiple of it to every row of W:

StyleGAN2 FFHQ: seed 1006, with the Smile slider anywhere from -3 to 3
StyleGAN2 FFHQ: seed 1006, with the Smile slider anywhere from -3 to 3

These directions are kind of meaningful, but not that good. They were found for the original StyleGAN, not StyleGAN2, so they only roughly fit. The smile works, gently, with the light shifting along with it, but the age direction hardly changes the face at all.

GAN-Control was trained to have its controls. Its W is split into slices of 64 or 128 numbers, for identity, expression, pose, light, age and hair, and a small controller network for each attribute turns a value you'd understand, like an age in years or a yaw in degrees, into that slice. The groups are off until you tick them, because a ticked group sets its slice for every face: ticking Pose, for example, turns every face to look straight at the camera. Age works properly here:

GAN-Control FFHQ: seed 1003, aged anywhere from 15 to 80
GAN-Control FFHQ: seed 1003, aged anywhere from 15 to 80

SemanticStyleGAN gives each part of the face its own two rows of W, one for the shape and one for the texture, so a slider changes just that part: the hair, the eyes, the mouth, the clothes or the background, plus two codes for the overall layout. Each slider moves its part towards the average face or away from it:

SemanticStyleGAN: seed 1003, with the first layout code anywhere from average to exaggerated
SemanticStyleGAN: seed 1003, with the first layout code anywhere from average to exaggerated

The first layout code hardly changes her at all, just a little of the hair colour. Most of the part sliders are like this: they nudge a face rather than make a new one.

MobileStyleGAN and MetFaces have no learned directions, just the 512-number vector itself. Edit… opens a window with a slider for each of the 512 numbers, but on its own each number doesn't mean anything you could name: every one of them changes a little of everything.

Seeds and exploration

In seed mode, the grid shows the seed and the seeds after it, each one a different person. Vector mode starts from one face instead: either the current seed (From seed), or any face in the grid (right-click, Use Vector). The first face in the grid is that face, and the rest are found around it in latent space, each one a random step away from it. Radius sets how far the steps go: small, and they're close relatives; larger, and they're cousins.

Too low a radius and the faces all look the same, too high and they all look completely different. 0.3, the default, gives faces that are clearly related but each a little different. Here's each model at 0.3:

MobileStyleGAN FFHQ in vector mode: seed 1007 first, then nine faces around it at a radius of 0.3
MobileStyleGAN FFHQ in vector mode: seed 1007 first, then nine faces around it at a radius of 0.3
StyleGAN2 FFHQ in vector mode: seed 1006 first, then nine faces around it
StyleGAN2 FFHQ in vector mode: seed 1006 first, then nine faces around it
StyleGAN2 MetFaces in vector mode: seed 1009 first, then nine faces around it
StyleGAN2 MetFaces in vector mode: seed 1009 first, then nine faces around it
GAN-Control FFHQ in vector mode: seed 1003 first, then nine faces around it, with the Pose group ticked
GAN-Control FFHQ in vector mode: seed 1003 first, then nine faces around it
SemanticStyleGAN CelebAMask-HQ in vector mode: seed 1003 first, then nine faces around it
SemanticStyleGAN CelebAMask-HQ in vector mode: seed 1003 first, then nine faces around it

The same radius goes further in some models than others. MetFaces stays closest, nearly the same sitter in the same paint, and SemanticStyleGAN mostly changes the hair. StyleGAN2 and MobileStyleGAN wander furthest: their faces are related, but now and then one turns into someone else entirely. GAN-Control keeps the face and changes the details, like hair, glasses and age.

The sliders work in vector mode too, on top of the radius, and with a Range ticked the grid goes back to showing the one face with different values.

You can also start from an existing face: drop a photo on the window, or open it with the app. FaceMaker finds the face with Apple's Vision framework, lines it up the way the models' training photos were, and then looks for it in one of three ways:

  • Quick compares it with 200 faces the model has already made, using ArcFace, which measures how alike two faces are, and starts from the closest. It's instant, but only ever vaguely similar.
  • Search starts there and keeps trying small changes, keeping any that make the face more alike, live in the grid until you stop it. It gets clearly similar, slowly, and never all the way.
  • Encoder (StyleGAN2 FFHQ only) runs e4e, a network trained to go straight from a photo to the W+ that draws it, which gives a real likeness straight away.
A 1947 photograph of Albert Einstein, matched with the encoder, and the faces around it
A 1947 photograph of Albert Einstein (Orren Jack Turner, public domain), matched with the encoder, and the faces around it

This has proved less successful than starting from a generated face. A real face doesn't tend to sit somewhere nice in latent space, with lots of similar faces around it: the faces nearby are more like distorted versions of the same face. Realism pulls them towards the average face, which keeps them looking real, and Coherent moves every row of W+ the same way, so they vary like faces rather than like noise. Both help, but a generated face is still the better place to start.

Usage

The best way to use FaceMaker is actually to keep generating random faces until you find one near enough to what you need, then search around it in latent space:

  1. Pick a model, set the grid to 20 or 30 faces, and press Regenerate until one is close.
  2. Right-click it and choose Use Vector, which makes it the first face and fills the grid with faces around it.
  3. Bring the Radius down as the faces get closer to what you want, and pick again. Use the model's sliders for anything they're good at, like GAN-Control's age and pose.
  4. Save the face, or drag it straight into the image or video tool, as the face for the character.

Writing down the seed, or saving the face's vector (Save Vector… on the face, or Export… in vector mode), means the same face can always be made again: Import… reads the file back.

Code organisation

FaceMaker is a Swift app with no Xcode project: swift build compiles it, and build.sh wraps it up as a .app. The models run through the C interface of ONNX Runtime, called directly from Swift, on the CPU, apart from ArcFace, which runs on Apple's Neural Engine through Core ML. Python is only used once, to convert the models.

  • Sources/FaceMaker/: the app.
    • FaceLocal.swift holds the list of models and their controls, and the engine that runs them: seed to z, the mapping and synthesis networks, the directions, controllers and part scales.
    • ORT.swift is a small wrapper around ONNX Runtime's C interface: a session, tensors in and out.
    • FaceInversion.swift finds a photo's face: Vision, alignment, ArcFace, the 200-face banks, the search and the encoder.
    • AppModel.swift holds everything the window shows and works out each face's latent and slider values.
    • RootView.swift, FaceCard.swift, FaceViewer.swift and VectorEditor.swift are the windows.
  • models/: download.sh, and the converters in models/convert/, one Python script for each model, written against the authors' own code.
  • Vendor/: download.sh fetches ONNX Runtime from Microsoft's releases.

To build it:

./Vendor/download.sh       # ONNX Runtime
./models/download.sh       # the models: about 15 minutes, and Python 3.10 to 3.13
make run
Building and running FaceMaker

The converters had to work around the same few things in every model. The custom GPU operations that StyleGAN2's PyTorch versions use are replaced by plain PyTorch, so they run on a Mac, and the modulated convolution at the heart of StyleGAN2 is rewritten so its weights have a fixed shape that ONNX can follow. Each conversion is checked against the original PyTorch model, and they all agree to within 0.0002, on pixel values from -1 to 1.

Source code

The whole app: the Swift sources, the model download and conversion scripts, and the build files. The models themselves come from models/download.sh.

The app

  • FaceMakerApp.swift Swift · 72 lines · the windows, and opening photos with the app
  • AppModel.swift Swift · 798 lines · everything the window shows, and each face's latent and slider values
  • FaceLocal.swift Swift · 658 lines · the models, their controls, and the engine that runs them
  • ORT.swift Swift · 189 lines · ONNX Runtime's C interface, wrapped for Swift
  • FaceInversion.swift Swift · 482 lines · from a photo to the closest face: Vision, ArcFace, the search and the encoder
  • RootView.swift Swift · 575 lines · the main window: the grid and the controls
  • FaceCard.swift Swift · 160 lines · one face in the grid, its menu and dragging it out
  • FaceViewer.swift Swift · 311 lines · the window for one face, with zoom, and saving faces and vectors
  • VectorEditor.swift Swift · 118 lines · the 512 sliders of the latent vector
  • Persistence.swift Swift · 54 lines · remembering the window's size and split
  • Theme.swift Swift · 81 lines · sizes and fonts
  • FaceMakerTests.swift Swift · 168 lines · the tests

The models

  • download.sh Shell · 188 lines · downloads every model from its authors and converts it
  • requirements.txt Text · 13 lines · the Python packages the converters use
  • export.py Python · 82 lines · exporting to ONNX, and checking the result against PyTorch
  • cpu_ops.py Python · 147 lines · StyleGAN2's GPU operations in plain PyTorch
  • stylegan2.py Python · 101 lines · StyleGAN2 FFHQ and MetFaces
  • mobilestylegan.py Python · 60 lines · MobileStyleGAN FFHQ
  • gancontrol.py Python · 124 lines · GAN-Control FFHQ and its five controllers
  • semantic.py Python · 65 lines · SemanticStyleGAN CelebAMask-HQ, with its part scales
  • e4e.py Python · 56 lines · the e4e encoder and the average face
  • directions.py Python · 25 lines · InterFaceGAN's directions
  • preview.py Python · 26 lines · one face from a converted model, to check it

Building

  • Makefile Makefile · 65 lines · make, make run, make test
  • build.sh Shell · 115 lines · compiles with SwiftPM and assembles the .app
  • Package.swift Swift · 51 lines · the Swift package
  • Vendor/download.sh Shell · 31 lines · fetches ONNX Runtime
  • module.modulemap Module map · 5 lines · makes ONNX Runtime's C header a Swift module
  • shim.h C · 2 lines · the header it includes
  • Info.plist XML · 53 lines · the app's details, and opening images
  • FaceMaker.entitlements XML · 15 lines · what the app is allowed to do
  • make-icon.swift Swift · 75 lines · renders the app icon at every size

See also