Mindbender

The safe post-production pipeline



  • Keywords: Film, games, content creation, pipeline

  • Objective:

    1. Provide a unified framework within which artists at Mindbender may work efficiently.
    2. Enable extending of said framework for future improvements and unique requirements.
    3. Inspire further expansion upon a series of basic ideas.

  • Requirements: Core functionality must be editable and extensible by the technical director.

  • Technology: Mindbender is built upon Pyblish, Python and bindings for Qt, and depends upon a Windows, Linux or MacOS operating system with Autodesk Maya, The Foundry Nuke, SideFX Houdini and other DCC content creation applications.



Target Audience

To make the most out of this document, some knowledge and experience is assumed.

minimal recommended
title technical director pipeline technical director
experience 1 year in advertisements or games 5+ years in feature film
software windows, linux or macos maya





Introduction

Welcome to mindbender-core, the production pipeline at Mindbender Animation Studio.


Table of contents



Features

This project currently boasts the following features.

Feature Description
Open Source Anyone can take part in testing, using and developing the pipeline on GitHub.
Version Control Entire pipeline is reviewed and tested prior to any change being made.
Issue Tracker Feature requests, bugs and questions are reported, assigned and solved openly.
Single-command install and update via Git Zero dependencies make for easy install and updates.
Environment management The developer manages the environment without distrupting the end user
Directory management Folders are automatically created as per the developers setup.
Application launcher Control over which version of which software is launched along with its initial configuration.
Asset versioning Every asset, including shots, are explicitly versioned - e.g. v001, v034
Asset subsets Each asset, such as Tarzan, contains one or more subsets, such as model, rig and look.
Asset and shot uniformity Shots are assets too.
Per project/shot settings Globally configure the framerate of a given project.
Per family publishing of assets Individually manage the validation and export assets of specific types, such as model, rig and animation.
Per family loading of assets Individually manage loading of the same specific types of assets.
Asset Metadata Relevant information is stored alongside each published version
Tools Graphical user interfaces for Creating, Publishing, Loading and Managing assets
Menus Application menus for modeling, rigging and animation
Support for off-site freelancers/studios No dependence on the local environment, the pipeline can be installed anywhere.
Support for Qt 4 and Qt 5 bindings Transparently uses the most desirable binding of Qt wherever the pipeline is used.



Install

Mindbender takes the form of a "git" repository. Therefore, you will need to install Git. Once installed, open up a terminal (Start > "cmd" > Enter) and type this in.

$ git clone https://github.com/mindbender-studio/setup mindbender-setup --recursive
$ cd mindbender-setup
$ git checkout 1.0 -b 1.0
$ git submodule update
$ start .

Windows Explorer appears! Double-click mb.bat and off you go!

image

untitled

Congratulations!

You are now fully equipped to handle Mindbender tasks!

Next Step



Update

To update mindbender-setup, run the update.bat file supplied within the mindbender-setup directory.

Alternatively, you may type this.

$ cd mindbender-setup
$ git pull
$ git submodule update --recursive

It is safe to run these as many times as you'd like.



Customisation

The above is all that is required to get started with the Mindbender Pipeline. If you are at home and are just looking to try things out, feel free to play around. Create a few assets, publish them and familiarise yourself with the workflow.

If you are at a studio, you will likely want to use your own projects with the Mindbender Launcher.

The next step is telling mb.bat about your local projects. From your terminal, type this.

$ cd mindbender-setup
$ notepad mb.bat

Notepad appears.

At the bottom of this file, replace %REPLACE_ME% with the absolute path to where your projects are located, such as m:\f03_projects. Now type mb.bat again to access your own projects. Optionally, you may also create a shortcut of mb.bat to your Desktop, Start Menu or any location you prefer.



Usage

Mindbender is initialised by calling install() with an interface for your host.

from mindbender import api, maya
api.install(maya)

Supported hosts

From here, you model, rig and animate as per the contract below.



How to read this guide

Here are a few of the conventions used throughout this guide.

  • bold words are used to augment important aspects of a sentence
  • UPPERCASE words are unique terminology, each detailed under Terminology below.
  • (1), (2) are used to division important aspects in a sentence.
  • code is used to highlight words that occur in code.



Description

Mindbender is a content creation studio focusing on photo-realistic, yet exaggerated cartoons. They work across continents in Sweden, Brazil and Spain and therefore require an underlying pipeline that facilitate these activities.

Pyblish Mindbender works exclusively with computer generated imagery - which means no focus is put on video, sound or the integration of 3d and live action footage. It has been designed for use both locally and remotely, to bridge the gap between artists working together but in different locations.

Overview

This pipeline covers the entire pipeline at Mindbender, including asset and shot creation. A PROJECT is partitioned into SHOTs where each shot consists of one or more ASSETs.

This pipeline includes tools and Pyblish plug-ins for 4 common types of ASSETs in a typical production pipeline.

  • Modeling
  • Rigging
  • Animation
  • Look

These implement (1) a contract for each FAMILY of ASSETs and (2) their interface towards each other.



Batteries

Mindbender hosts a series of graphical user interfaces that aid the user in conforming to the specified contracts.

Name Purpose Description
creator control what goes out Manage how data is outputted from an application.
loader control what goes in Keep tabs on where data comes from so as to enable tracking and builds.
manager stay up to date Notification and visualisation of loaded data.


Creator

Associate content with a family.

The family is what determins how the content is handled throughout your pipeline and tells Pyblish what it should look like when valid.

API

The creator respects families registered with Mindbender.

from mindbender import api

api.register_family(
    name="my.family",
    help="My custom family",
)

For each family, a common set of data is automatically associated with the resulting instance.

{
    "id": "pyblish.mindbender.instance",
    "family": {chosen family}
    "name": {chosen name}
}

Additional common data can be added.

from mindbender import api

api.register_data(
    key="myKey",
    value="My value",
    help="A special key"
)

Data may be associated with a family.

from mindbender import api

api.register_family(
    name="my.family",
    data=[
        {"key": "name", "value": "marcus", "help": "Your name"},
        {"key": "age", "value": 30, "help": "Your age"},
])



Loader

Visualise results from api.ls().

from mindbender import api

for asset in api.ls():
    print(asset["name"])

API

The results from api.ls() depends on the currently registered root.

from mindbender import api
api.register_root("/projects/gravity")

The chosen ASSET is passed to the load() function of the currently registered host.

from mindbender import api, maya
api.register_host(maya)

A host is automatically registered on api.install().



Manager

Visualise loaded assets.

from mindbender import maya

for container in maya.ls():
    print(container["name"])

# The same is true for any host; houdini, nuke etc.

API

The results from ls() depends on the currently registered host, such as Maya.

from mindbender import nuke
api.register_host(nuke)










API

Mindbender exposes a series of interrelated APIs to the end-user.

Name Purpose
Terminal API Defines how the artist interacts with tasks
Filesystem API Defines how the developer interact with data on disk
Mindbender API Defines how the developer interacts with Mindbender
Host API Defines how the host interacts with Mindbender



Terminology

Mindbender reserves the following words for private and public use. Public members are exposed to the user, private ones are internal to the implementation.

Term Public Description Example
PROJECTS X Parent of projects m:\f03_projects
PROJECT X Root of information Gravity, Dr. Strange
ASSET X Unit of data Ryan, Bicycle, Flower pot
VERSION X An ASSET iteration v1, v034
REPRESENTATION A data format Maya file, pointcache, thumbnail
FORMAT A file extension .ma, .abc, .ico, .png
FAMILY X A type of ASSET model, rig, look, animation
WORKSPACE X Private data Scenefile for v034 of Ryan
SILO Data repository Ryan resides in assets, caches in film.
INSTANCE Inverse of a file modelDefault_SET
STAGE Transient data Outgoing VERSION from scenefile
SHARED X Public data v034 of Ryan
PRODUCER Creator of data You
CONSUMER User of data Me



Terminal API

mb.bat (pronounced "embee-bat") is the Mindbender Launcher. It is how artists launch applications, such as Maya and Nuke. It establishes important environment variables used when producing and publishing data, along with exposing artists to tools relevant a given project.

untitled


project.bat

mb.bat is where artists enter the pipeline. A user may then enter any of the available projects, via a project.bat file.

The project file is located in your PROJECTS directory

@echo off
call _mkproject %~dp0 %~n0 %1

asset.bat

@echo off
call _mkproject %~dp0 %~n0 %1

The layout is as follows.

  1. {PROJECT} (with TAB-completion)
  2. {ASSET} or Shot (with TAB-COMPLETION)
  3. {APPLICATION} {TASK}

The given TASK is automatically created, unless it already exists.

Environment variables

Variable Description
PROJECT Nice name of PROJECT, e.g. Gifts for Greta
PROJECTDIR Absolute path to PROJECT, e.g. m:\f01_projects\p999_Gifts_for_Greta
ROOT Top level directory of either shot or asset, e.g. ..\Greta



Filesystem API

Data is organised into files and folders.

Some files and folders have special meaning in Mindbender.

image


Information Hierarchy

The mental and physical model for files and folders look like this.

temp_03



Private, Public and Stage

During the course of the creation of any ASSET, data moves between 2 of 3 states.

temp

  • The pipeline does not take into consideration the workspace and is therefore workspace-agnostic.
  • The staging area is both implicit and transparent to the PRODUCER and CONSUMER, except for debugging purposes. This is where automatic/scheduled garbage collection may run to optimise for space constraints.
  • The shared space is where ASSETs ultimately reside once published.


Private and Public separation

A naive approach to content creation might be to refer to ASSETs straight from another artists workspace. At Mindbender, data is separated between work-in-progress (private) and data exposed to others (public).

Private data resides in work, public data resides in publish.

Private data is highly mutable and typically private to an individual artist.

  • Mutable implies transient data that is likely to change at any given moment.
  • Private implies personal, highly irregular and likely invalid data.

Public data on the other hand is immutable, correct and impersonal.

  • Immutable implies that the data may be dependent upon by other data.
  • Correct implies passing validation of the associated family.
  • Impersonal implies following strict organisational conventions.



Each ASSET reside within a top-level ROOT and SILO directory as follows.

Hierarchy Example
hasset hassetex


Each ASSET contains a work and publish directory. work is where artists save their own files while working, whereas publish is where published files go.

The publish directory contains 0 or more SUBSETS which in turn contains 0 or more VERSIONS

Hierarchy Example
hsubset hsubsetex


Each VERSION contains 1 or more REPRESENTATIONS.

Example
hrepresentationex


All data is written in 3 stages.

  1. To a local /temp directory.
  2. To a "staging" directory
  3. To the final destination.

Data is first written locally, so as to not burden a potentially remote filesystem with sporadic writes performed by an application - for example, caching may be bound by parsing of geometry, and not I/O bound. Which means many small writing requests are scattered when exporting a large file.

It also alleviates the remote system from times where writing is cancelled or otherwise fails.

The staging area is where files are written next; the stage resides within the users local workspace and offers the end-user a chance to inspect previous successful and unsuccessful publishes. On a successful export from an application, this is where disparate files are assembled and injected with additional metadata

Should this succeed, the are then moved to their final destination.

Hierarchy Example
usr1 usr2

Each directory will contain everything that did extract successfully, along with its metadata, for manual inspection and debugging.


ls()

Communication with the filesystem is made through ls().

ls() returns available assets - relative the currently registered root and silo directories - in the form of JSON-compatible dictionaries. Each dictionary is strictly formatted according to four distinct "schemas".

Example

from mindbender import api

for asset in api.ls():
    for subset in asset["subsets"]:
        for version in asset["versions"]:
            for representation in version["representations"]:
                pass

See below for a full list of members.


Schema

image

Available schemas are organised hierarchically, with the former containing the latter.


asset.json

A part of a PROJECT, such as a Character or Shot.

Key Value Description
name str Name of directory
subsets list 0 or more subset.json


subset.json

A part of an ASSET, such as a model or a rig.

Key Value Description
name str Name of directory
versions list 0 or more version.json


version.json

An immutable iteration of a SUBSET.

Key Value Description
version int Number of this VERSION
path str Unformatted path, e.g. {root}/
time str ISO formatted, file-system compatible time.
author str User logged on to the machine at time of publish.
source str Original file from which this VERSION was made.
representations list 0 or more representation.json

Versions are immutable, in that they never change once made. This is in stark contrast to mutable versions which is when one version may be "updated" such that the same file now contains new information.


representation.json

One of many representations of a VERSION.

Key Value Description
format str File extension
path str Unformatted path

Think of a representation as one way of storing the same set of data on disk. For example, an image may be stored as both PNG and JPEG. Different files, same data. It could also be stored as a description. "A picture of my computer." Much less information is ultimately stored, but it is nonetheless the exact same original data in a different (albeit lossy) representation. The image could also be represented by a feeling (warm, mystical) or a spoken word (muah!).

Representation are very powerful and lie at the heart of assets that are more than just a single file.

As a practical example, a Look is stored as both an MA scene file and a JSON. The JSON stores the shader relationships, whereas the MA file stores the actual shaers. Same data, different representations.

container.json

An imported VERSION, as yielded from api.registered_host().ls().

Key Value Description
schema str Contains "mindbender-core:container-1.0"
name str Pipeline name of this container
objectName str Internal name used in an application
asset str Source ASSET of this container
subset str Source SUBSET of this container
version int Source VERSION of this container
path str Path to loaded VERSION
source str Absolute path to original scene
author str Author of VERSION



Mindbender API

Mindbender provides a stateful API.

State is set and modified by calling any of the exposed registration functions, prefixed register_*.


Public members of mindbender.api

Member Returns Description
install(host) str Install Mindbender into the current interpreter session
uninstall() str Revert installation
ls() generator List available assets, relative root
registered_root() str Absolute path to current working directory
format_staging_dir(root, name) str Return absolute path or staging directory relative arguments
format_shared_dir(root) str Return absolute path of shared directory
format_version(version) str Return file-system compatible string of version
find_latest_version(versions) int Given a series of string-formatted versions, return the latest one
parse_version(version) str Given an arbitrarily formatted string, return version number
register_root(root) Register currently active root
register_host(host) Register currently active host
register_plugins() Register plug-ins bundled with Pyblish Mindbender
deregister_plugins()
registered_host() module Return currently registered host
registered_families() list Return currently registered families
registered_data() list Return currently registered data
registered_root() str Return currently registered root



Host API

A host must implement the following members.

Member Returns Description
ls() generator List loaded assets
create(name, family, options=None) str Build fixture for outgoing data (see instance), returns instance.
load(asset, subset, version=-1, representation=None) None Import external data into container
update(container, version=-1) None Update an existing container
remove(container) None Remove an existing container


Information hierarchy

Loaded data is stored in a container. A container hosts a loaded asset along with metadata used to associate assets that use other assets, such as a Wheel asset used in a Car asset.

Host data relationship

Id

Internally, Pyblish instances and containers are distinguished from native content via an "id". For example, in Maya, the id is a user-defined attribute.

Name Description Example
pyblish.mindbender.container Unit of incoming data ...:model_GRP, ...:rig_GRP
pyblish.mindbender.instance Unit of outgoing data Strange_model_default



Contract

Mindbender defines these families.

Family Definition Link
mindbender.model Geometry with deformable topology Spec
mindbender.rig An articulated mindbender.model for animators Spec
mindbender.animation Pointcached mindbender.rig for rendering Spec
mindbender.lookdev Shaded mindbender.model for rendering Spec


mindbender.model

A generic representation of geometry.

aud Workflow

  1. Create a new Model INSTANCE.
  2. Add the ROOT group
  3. Publish

aud Target Audience

  • Texturing
  • Rigging
  • Final render

req Requirements

  • All DAG nodes must be parented to a single top-level transform
  • Normals must be unlocked

dat Data

  • name (str, optional): Pretty printed name in graphical user interfaces

set Sets

  • geometry_SEL (geometry): Meshes suitable for rigging
  • aux_SEL (any, optional): Auxilliary meshes for e.g. fast preview, collision geometry





mindbender.rig

The mindbender.rig contains the necessary implementation and interface for animators to animate.

aud Workfow

  1. Add the ROOT group
  2. Add animatable controllers to an objectSet called controls_SET
  3. Add cachable meshes to an objectSet called out_SET

Publishing.

  1. Create a new Rig INSTANCE
  2. Add ROOT
  3. Add controls_SET
  4. Add out_SET
  5. Publish

aud Target Audience

  • Animation

req Requirements

  • All DAG nodes must be parented to a single top-level transform
  • Must contain an objectSet for controls and cachable geometry

dat Data

  • name (str, optional): Pretty printed name in graphical user interfaces

set Sets

  • in_SEL (geometry, optional): Geometry consumed by this rig
  • out_SEL (geometry): Geometry produced by this rig
  • controls_SEL (transforms): All animatable controls
  • resources_SEL (any, optional): Nodes that reference an external file



mindbender.animation

Point positions and normals represented as one Alembic file.

aud Workflow

The animator workflow is simplified by the fact that an INSTANCE is automatically created upon loading a rig.

  1. Load rig
  2. Publish

aud Target Audience

  • Lighting
  • FX
  • Cloth
  • Hair

req Requirements

  • None

dat Data

  • name (str, optional): Pretty printed name in graphical user interfaces

set Sets

  • None



mindbender.lookdev

aud Workflow

Shaders are exported relative the meshes in an INSTANCE.

  1. Create a new Look INSTANCE
  2. Add the transform of each shaded mesh
  3. Publish

aud Target Audience

  • Lighting

req Requirements

  • None

dat Data

  • name (str, optional): Pretty printed name in graphical user interfaces

set Sets

  • None




Legend

Title Description
aud Target Audience Who is the end result of this family intended for?
req Requirements What is expected of this ASSET before it passes the tests?
dat Data End-user configurable options
set Sets Collection of specific items for publishing or use further down the pipeline.