dropbox3




Reference

This section covers high-level aspects of Avalon in an information-oriented fashion.

Looking for API reference?

See the auto-generated documentation for the Avalon API



Content Life Cycle

Data in Avalon is either persistent or in transit. Persistent data resides in either a file-system or database, whereas data in transit is in one of three states.

image

Note

For each state, there is an API for developers and at least one GUI for users.

Create

Creation is the process of introducing new data into a project and is divided into two parts; asset and subset creation.

Assets are abstract representations of the data used throughout a project - such as sequences, shots, characters and props - whereas Subsets represents data per asset - such as geometry, textures or rigs.

Asset and subset creation is governed by the Project Inventory API and Creator API respectively via one or more plug-ins associated to named "families" of data, such as model, look or render.

Assets are created via the Project Inventory API and subsets are generally created via the use of a Digital Content Creation package, such as Autodesk Maya or The Foundry Nuke.

API Example

from avalon import api

class CreateModel(api.Creator):
    """Polygonal geometry for animation"""

    label = "Create Avalon Model"
    name = "modelDefault"
    family = "avalon.model"

More information


Import

Import is the process of parsing persistent data from disk and into the memory of a running application.

Due to data being either localised or referenced, import is referred to as loading, a process governed by the Loading API through one or more plug-ins associated to named families of data.

API Example

from avalon import api

class LoadModel(api.Loader):
    """Load data of family avalon.model"""

    label = "Load Avalon Model"
    families = ["avalon.model"]
    representations = ["ma"]

    def process(self, name, namespace, context):
        from maya import cmds
        from avalon import maya

        with maya.maintained_selection():
            nodes = cmds.file(self.fname)

        self[:] = nodes

More information



Export

Export is the process of transforming in-memory data native to an application into something that can persist on disk. During export, data is funneled through a validation mechanism that check for consistency. Because of this additional mechanism, export is referred to as publishing.

The manner in which data is validated and written is governed by a series of plug-ins, orchestrated by the Publishing API and associated to families of data.

API Example

from pyblish import api

class ExtractAvalonModel(api.InstancePlugin):
    """Produce a stripped down Maya file from instance"""

    label = "Extract Avalon Model"
    order = api.ExtractorOrder
    hosts = ["maya"]
    families = ["avalon.model"]

    def process(self, instance):
        from maya import cmds
        from avalon import maya

        with maya.maintained_selection(), maya.without_extension():
            cmds.select(instance, noExpand=True)
            cmds.file(path, typ="mayaAscii", exportSelected=True)

More information


Persist

Once exported, data resides in one or two locations - as files in a file-system, or as documents in a database. The exact location within the file-system is governed by the Project Configuration API via path "templates" - a string encoded with placeholder variables associated to the various objects in the object model, customisable per-project.


Example

{
    "work": "{root}/{project}/{asset}/work/{task}/{user}/{app}",
    "publish": "{root}/{project}/{asset}/publish/{subset}/v{version:0>3}/{subset}.{representation}"
}



Object Model

Wherever data is stored, it is stored as a hierarchy of increasingly granular objects, representative of the division of labour in each project created through Avalon.


image


Assets represents the most course grained division of labour and is typically used for shots and builds, e.g. the 6th shot and the hero character.

Each asset contain one or more Subsets which are typically used for individual models and animation caches for a build or shot.

Each subset contain one or more Versions which are immutable sources of data containing the final element of the object model; the Representation.

Representations are the storage method of a version, such as a .png thumbnail, an .obj geometry file or .mp4 turntable of a hero model. Both of which represents the same set of data in three different ways.

Each object containing a series of members defined by an explicit schema, enforced via jsonschema and organised hierarchically with the former containing the latter.

Read more

Metadata

Every object in the model contains a dictionary member called .data.

The conceptual difference between top-level members and members of .data is that top-level members can be assumed to exist in every application, whereas members of .data are optional.

Optional members facilitate flexible code at the cost of having more of it and therefore more to maintain.




Database

Avalon stores data in two separate locations, on disk and in a database. The separation is made due to performance and search capabilities offered by databases.

MongoDB was chosen due to the inherent simplicity and similarity to Python's built-in dictionary type, and performance great enough to enable graphical user interfaces to be built without asynchronousity in mind.

Inside of MongoDB, data is stored as Collections containing many Documents. In Avalon, each Collection represents a project and documents make up the Object Model.

  • Asset
  • Subset
  • Version
  • Representation

These form a hierarchy, where each contain the latter. Assets make up the top-level object within a project, and can represent anything from characters, shots to levels and more.

Asset Description
Hulk A bulky fellow
Bruce The hero of the film
1000 First shot
1200 Second shot

Subsets is the asset broken down into smaller sets of information, such as a rig or a model.

Subset Description
model Hulk's model
rig Hulk's rig
lookdev Hulk's look
animation Hulk's point cached geometry

A subset must have a least one Version, which is typically immutable.

Version Comment
v001 Initial version
v002 Fixed whole in mesh
v003 Increased the size of pecs

Finally, in each version there is at least one Representation; typically a file or sequence of files.

Representation Description
ma Maya rig
mov Turntable
abc Still frame of mesh used in rig

Read more about the kinds of objects in Schemas below.


Schemas

All data within the database and on disk follow a strict layout, known as a "schema".

Project

A project is a top-level object that cannot be contained elsewhere, but contains everything else.

project-2.0.json 

{
    "config": "Document metadata",
    "data": "Document metadata",
    "name": "Name of directory",
    "parent": "Unique identifier to parent document",
    "schema": "Schema identifier for payload",
    "type": "The type of document"
}

Example

{
    "config": {
        "apps": [
            {
                "label": "Autodesk Maya 2016",
                "name": "maya2016"
            },
            {
                "label": "The Foundry Nuke 10.0",
                "name": "nuke10"
            }
        ],
        "schema": "avalon-core:config-1.0",
        "tasks": [
            {
                "name": "model"
            },
            {
                "name": "render"
            },
            {
                "name": "animate"
            },
            {
                "name": "rig"
            },
            {
                "name": "lookdev"
            },
            {
                "name": "layout"
            }
        ],
        "template": {
            "publish": "{root}/{project}/{silo}/{asset}/publish/{subset}/v{version:0>3}/{subset}.{representation}",
            "work": "{root}/{project}/{silo}/{asset}/work/{task}/{app}"
        }
    },
    "data": {
        "fps": 24,
        "height": 1080,
        "width": 1920
    },
    "name": "hulk",
    "parent": "592c33475f8c1b064c4d1696",
    "schema": "avalon-core:project-2.0",
    "type": "project"
}


Asset

A part of a project, such as a Character or Shot.

asset-2.0.json 

{
    "data": "Document metadata",
    "name": "Name of asset",
    "parent": "Unique identifier to parent document",
    "schema": "Schema identifier for payload",
    "silo": "Group or container of asset",
    "type": "The type of document"
}

Example

{
    "data": {
        "key": "value"
    },
    "name": "Bruce",
    "parent": "592c33475f8c1b064c4d1696",
    "schema": "avalon-core:asset-2.0",
    "silo": "assets",
    "type": "asset"
}


Subset

A part of an Asset, such as a model or a rig.

subset-2.0.json 

{
    "data": "Document metadata",
    "name": "Name of directory",
    "parent": "Unique identifier to parent document",
    "schema": "The schema associated with this document",
    "type": "The type of document"
}

Example

{
    "data": {
        "frameEnd": 1201,
        "frameStart": 1000
    },
    "name": "shot01",
    "parent": "592c33475f8c1b064c4d1696",
    "schema": "avalon-core:subset-2.0",
    "type": "subset"
}


Version

An immutable iteration of a Subset.

Versions are immutable, in that they never change once made. This is in stark contrast to mutable versions which is when one version may be "updated" such that the same file now contains new information.

version-2.0.json 

{
    "data": "Document metadata",
    "locations": "Where on the planet this version can be found.",
    "name": "Number of version",
    "parent": "Unique identifier to parent document",
    "schema": "The schema associated with this document",
    "type": "The type of document"
}

Example

{
    "data": {
        "author": "marcus",
        "families": [
            "avalon.model"
        ],
        "source": "{root}/f02_prod/assets/BubbleWitch/work/modeling/marcus/maya/scenes/model_v001.ma",
        "time": "20170510T090203Z"
    },
    "locations": [
        "data.avalon.com"
    ],
    "name": 12,
    "parent": "592c33475f8c1b064c4d1696",
    "schema": "avalon-core:version-2.0",
    "type": "version"
}


Representation

One of many representations of a Version.

Think of a representation as one way of storing the same set of data on disk. For example, an image may be stored as both PNG and JPEG. Different files, same data. It could also be stored as a description. "A picture of my computer." Much less information is ultimately stored, but it is nonetheless the exact same original data in a different (albeit lossy) representation. The image could also be represented by a feeling (warm, mystical) or a spoken word (muah!).

Representations are very powerful and lie at the heart of assets that are more than just a single file.

As a practical example, a Look is stored as both an MA scene file and a JSON. The JSON stores the shader relationships, whereas the MA file stores the actual shaders. Same data, different representations.

representation-2.0.json 

{
    "context": "Summary of the context to which this representation belong.",
    "data": "Document metadata",
    "dependencies": "Other representation that this representation depends on",
    "name": "Name of representation",
    "parent": "Unique identifier to parent document",
    "schema": "Schema identifier for payload",
    "type": "The type of document"
}

Example

{
    "context": {
        "asset": "Bruce",
        "project": "hulk",
        "representation": "ma",
        "silo": "assets",
        "subset": "rigDefault",
        "version": 12
    },
    "data": {
        "label": "Alembic"
    },
    "dependencies": [
        "592d547a5f8c1b388093c145"
    ],
    "name": "abc",
    "parent": "592c33475f8c1b064c4d1696",
    "schema": "avalon-core:representation-2.0",
    "type": "representation"
}


Container

An imported Version, as yielded from api.registered_host().ls().

container-2.0.json 

{
    "id": "Identifier for finding object in host",
    "loader": "Name of loader plug-in used to produce this container",
    "name": "Internal object name of container in application",
    "namespace": "Internal namespace of container in application",
    "objectName": "Name of internal object, such as the objectSet in Maya.",
    "representation": "Unique id of representation in database",
    "schema": "Schema identifier for payload"
}

Example

{
    "id": "pyblish.avalon.container",
    "loader": "ModelLoader",
    "name": "modelDefault_01",
    "namespace": "Bruce_",
    "objectName": "Bruce_:rigDefault_CON",
    "representation": "59523f355f8c1b5f6c5e8348",
    "schema": "avalon-core:container-2.0"
}



Software

Avalon assumes content is created within an application of some kind and manages the execution of each application via Launcher.

Apps

Launcher is responsible for launching "apps", such as Maya. "App" is the term used for a pre-configured application in Avalon.

Problem

It could call on c:\Program Files\Autodesk\Maya2017\bin\maya.exe directly, but doing so is problematic because..

  1. It assumes a particular operating system
  2. It assumes a particular installation directory
  3. It assumes a particular app is what you want for your project(s)
  4. It assumes no customisation of environment prior to launch

Solution

The Project Executable API addresses this by splitting the problem it into three independently configurable parts.

  1. Apps are assumed to be available on your PATH, e.g. maya.sh or maya.bat
  2. Configuration is performed per application in an individual configuration file, e.g. maya.toml
  3. Apps are associated per project, e.g. Hulk uses Maya and Nuke.




Library API

Public members of avalon.api. See API Documentation for full details.

Member Description
install Install host into the running Python session.
uninstall Undo all of what install() did
schema Wrapper around :mod:jsonschema
Loader Load representation into host application
Creator Determine how assets are created
Action A custom action available
InventoryAction A custom action for the scene inventory tool
Application Default application launcher
discover Find and return subclasses of superclass
Session dict() -> new empty dictionary
session dict() -> new empty dictionary
on Call callback on event
after Convenience to on() for after-events
before Convenience to on() for before-events
emit Trigger an event
publish Shorthand to publish from within host
create Create a new instance
load Use Loader to load a representation.
update Update a container
switch Switch a container to representation
remove Remove a container
data dict() -> new empty dictionary
update_current_task Update active Session to a new task work area.
get_representation_path Get filename from representation document
loaders_from_representation Return all compatible loaders for a representation.
register_host Register a new host for the current process
register_plugin_path Register a directory of one or more plug-ins
register_plugin Register an individual obj of type superclass
register_root Register currently active root
registered_root Return currently registered root
registered_plugin_paths Return all currently registered plug-in paths
registered_host Return currently registered host
registered_config Return currently registered config
deregister_plugin Oppsite of register_plugin()
deregister_plugin_path Oppsite of register_plugin_path()
logger
time Return file-system safe string of current date and time



Host API

A host must implement the following members.

Member Returns Description
ls generator List loaded assets
create str Build fixture for outgoing data (see instance), returns instance.
load None Import external data into container
update None Update an existing container
remove None Remove an existing container


Information hierarchy

Imported data is stored in a container. A container hosts a loaded asset along with metadata used to associate assets that use other assets, such as a Wheel asset used in a Car asset.

Host data relationship

Id

Internally, Pyblish instances and containers are distinguished from native content via an "id". For example, in Maya, the id is a user-defined attribute.

Name Description Example
pyblish.avalon.container Unit of incoming data ...:model_GRP, ...:rig_GRP
pyblish.avalon.instance Unit of outgoing data Strange_model_default



Project Inventory API

The inventory contains all ASSETs of a project, including metadata.

.inventory.toml

# Mandatory, do not touch
schema = "avalon-core:inventory-1.0"

# Project metadata
label = "The Hulk"
fps = 24
resolution_width = 1920
resolution_height = 1080

# Available assets
[[assets]]
name = "Batman"

[[assets]]
name = "Bruce"
label = "Bruce Wayne"  # (Optional) Nicer name
group = "Character"  # (Optional) Visual grouping
icon = "gear"  # (Optional) Icon from FontAwesome

[[assets]]
name = "Camera"

# Available shots
[[film]]
name = "1000"
edit_in = 1000
edit_out = 1202

[[film]]
name = "1200"
edit_in = 1000  # Optional metadata per shot
edit_out = 1143

The above is an example of an "inventory". A complete snapshot of all available assts within a given project, along with optional metadata.



Project Configuration API

The project configuration contains the applications and tasks available within a given project, along with the template used to create directories.

.config.toml

# Mandatory, do not touch
schema = "avalon-core:config-1.0"

# Available tasks to choose from.
[[tasks]]
name = "modeling"
label = "Character Modeling"
icon = "video-camera"

[[tasks]]
name = "animation"

# Available applications to choose from, the name references
# the executable API (see below)
[[apps]]
name = "maya2016"
label = "Autodesk Maya 2016"

[[apps]]
name = "python"
label = "Python 3.6"
args = ["-u", "-c", "print('Something nice')"]

# Directory layouts for this project.
[template]
work = "{root}/{project}/{silo}/{asset}/work/{task}/{user}/{app}"
publish = "{root}/{project}/{silo}/{asset}/publish/{subset}/v{version:0>3}/{subset}.{representation}"

The directory layout have the following members available.

Member Type Description
{app} str The current application directory name, defined in Executable API
{task} str Name of the current task
{user} str Currently logged on user (provided by getpass.getuser())
{root} str Absolute path to root directory, e.g. m:\f01_project
{project} str Name of current project
{silo} str Name of silo, e.g. assets
{asset} str Name of asset, e.g. Bruce
{subset} str Name of subset, e.g. modelDefault
{version} int Number of version, e.g. 1
{representation} str Name of representation, e.g. ma



Project Executable API

Every executable must have an associated Application Definition file which looks like this.

maya2016.toml

# Required header, do not touch.
schema = "avalon-core:application-1.0"

# Name of the created directory, available in the 
# `template` of the Configuration API
application_dir = "maya"

# These directories will be created under the
# given application directory
default_dirs = [
    "scenes",
    "data",
    "renderData/shaders",
    "images"
]

# Name displayed in GUIs
label = "Autodesk Maya 2016x64"

# Arguments passed to the executable on launch
arguments = [ "-proj", "{AVALON_WORKDIR}",]

# Name of the executable on the local computer.
# This name must be available via the users `PATH`.
# That is, the user must be able to type this into
# the terminal to launch said application.
executable = "maya2016"
description = ""

# Files copied into the application directory on launch
[copy]
"{AVALON_CORE}/res/workspace.mel" = "workspace.mel"

# The environment variables overrides any previously set
# variables from the parent process.
[environment]
MAYA_DISABLE_CLIC_IPM = "Yes"  # Disable the AdSSO process
MAYA_DISABLE_CIP = "Yes"  # Shorten time to boot
MAYA_DISABLE_CER = "Yes"
PYTHONPATH = [
    "{PYBLISH_MAYA}/pyblish_maya/pythonpath",
    "{AVALON_CORE}/avalon/maya/pythonpath",
    "{PYTHONPATH}"
]


Project Template API

The vast majority of data managed by Avalon is stored as files on disk, and every file has a path. The path is an integral part of any production and requires fine-grained control.

Avalon provides this control in the form of templates.

A template is a string of {placeholders}, each placeholder representing some data.

template = "{project}/{asset}/myfile_v{version:0>3}.ma"

The keywords within the {} are known as placeholders and may be dynamically replaced at run-time by your code.

Example

On saving a file, this template could be used to construct a path.

template = "{project}/{asset}/myfile_{version:0>3}.ma"
fname = template.format(
    project="hulk",
    asset="bruce",
    version=13
)
print(fname)

hulk/bruce/myfile_013.ma

Provided that pass the same data, the template itself can vary, whilst leaving your code intact.

template = "c:/assets/{asset}/{project}/myfile_version{version:0>2}.mb"

Viola, a drastically different directory layout, with identical data and code to process it. Notice how this templates includes reference to c:\ which not only asserts a particular operating system (Windows) but also leaves no room for an alternative "root". See Root Template below for an example of how to overcome this limitation.

This is how Avalon works.

In this way, you can establish convention at a high-level whilst still remaning specific in places where you need to either read from or write to disk. More importantly, this is the aspect which allows Avalon to not make assumptions about your directory layout; it is up to you to write this template.

Having said that, though Avalon doesn't make assumptions about where data is written, it does make assumptions on the types of data written - referred to as the Avalon Object Model - and your directory layout must accommodate for these types at minimum.

For any project, Avalon assumes a work template to be available in your Project Configuration. This template is used to generate the initial directory layout for your work-in-progress files.

The layout and ordering of placeholders in this template is within your control, and these are the ones available to you.

{
    "root": "Currently registered root, e.g. 'c:/projects'",
    "project": "Current project, e.g. 'hulk'",
    "silo": "Current silo, e.g. 'assets', 'film', 'episodes'",
    "asset": "Current asset, e.g. 'Bruce'",
    "task": "Current task",
    "app": "Current app, e.g. 'maya2019'",
    "user": "Current user, e.g. 'rick'"
}

Here's an example of a work template.

template = "{root}/{project}/{silo}/{asset}/work/{task}/{user}/{app}"

Template Access

Templates are generally stored in your project configuration.

# Untested
from avalon import io
project = io.find_one({"type": "project"})
template = project["config"]["template"]["work"]

Root Template

Every path has a "root"; the first part of a path.

c:\
/root
/projects/

When a template includes a root explicitly, like any of the above, then a template is limited to (1) a particular platform and (2) a particular mount on that platform.

For example, if your studio is exclusively on Windows and all share the drive z:\, then it is safe to embed this into your template.

template = "z:/projects/{project}/{asset}/..."

However, when there are multiple operating systems and/or multiple mount points - e.g. z:\ on some machines and and x:\ on others - then the above template no longer works.

To work around this, Avalon provides the idea of a "root" directory.

# Windows box, from our London branch
from avalon import api
api.register_root(r"c:\projects")

At run-time, a root directory is registered relative your platform and mount point, such that your template can look like..

template = "{root}/{project}{asset}..."

Such that, on another platform or machine, the root registration can change to account for a differing root.

# Linux box, in LA
from avalon import api
api.register_root("/mnt/projects")

As a result, your folder creation and publishing code can remain as-is, whilst assets are being read and written all over the place.




Environment Variables

Avalon uses environment variables extensively. For example, simply running an application involves two layers of information exchange.

  1. Your shell > Launcher
  2. Launcher > Application

In this documentation, environment variables are UPPERCASE and setting one is similar across all platforms.

set MY_VARIABLE=value

export MY_VARIABLE=value

Variables may also be set from within Python, by altering the os.environ dictionary.

import os
os.environ["MY_VARIABLE"] = "value"

Avalon Environment

These are all environment variables relevant in some way to Avalon.

session-1.0.json 

{
    "AVALON_APP": "Name of application",
    "AVALON_ASSET": "Name of asset",
    "AVALON_CONFIG": "Name of Avalon configuration",
    "AVALON_CONTAINER_ID": "Unique identifier for a loaded representation in a working file",
    "AVALON_DB": "Name of database",
    "AVALON_DEADLINE": "Address to Deadline",
    "AVALON_DEBUG": "Enable debugging mode. Some applications may use this for e.g. extended verbosity or mock plug-ins.",
    "AVALON_INSTANCE_ID": "Unique identifier for instances in a working file",
    "AVALON_LABEL": "Nice name of Avalon, used in e.g. graphical user interfaces",
    "AVALON_MONGO": "Address to the asset database",
    "AVALON_PASSWORD": "Generic password",
    "AVALON_PROJECT": "Name of project",
    "AVALON_PROJECTS": "Absolute path to root of project directories",
    "AVALON_SENTRY": "Address to Sentry",
    "AVALON_SILO": "Name of asset group or container",
    "AVALON_TASK": "Name of task",
    "AVALON_TIMEOUT": "Wherever there is a need for a timeout, this is the default value.",
    "AVALON_UPLOAD": "Boolean of whether to upload published material to central asset repository",
    "AVALON_USERNAME": "Generic username",
    "AVALON_WORKDIR": "Current working directory of a host, such as Maya's location of workspace.mel"
}

Example

{
    "AVALON_APP": "maya2016",
    "AVALON_ASSET": "Bruce",
    "AVALON_CONFIG": "polly",
    "AVALON_CONTAINER_ID": "avalon.container",
    "AVALON_DB": "avalon",
    "AVALON_DEADLINE": "http://192.168.99.101",
    "AVALON_DEBUG": "True",
    "AVALON_INSTANCE_ID": "avalon.instance",
    "AVALON_LABEL": "Mindbender",
    "AVALON_MONGO": "mongodb://localhost:27017",
    "AVALON_PASSWORD": "abc123",
    "AVALON_PROJECT": "Hulk",
    "AVALON_PROJECTS": "/nas/projects",
    "AVALON_SENTRY": "https://5b872b280de742919b115bdc8da076a5:8d278266fe764361b8fa6024af004a9c@logs.mindbender.com/2",
    "AVALON_SILO": "assets",
    "AVALON_TASK": "modeling",
    "AVALON_TIMEOUT": "1000",
    "AVALON_UPLOAD": "True",
    "AVALON_USERNAME": "myself",
    "AVALON_WORKDIR": "/mnt/projects/alita/assets/vector/maya"
}


Inheritance and Persistence

Setting a variable as above only affects the current instance of the process, such as cmd.exe and python. That's because variables are inherited.

Inheritance in this context means that any process launched from a parent process will contain a duplicate of the parent environment, and no change you make to the environment of this child process will affect the parent process.

set MY_VARIABLE=value
cmd
echo %MY_VARIABLE%
:: value
set CHILD_VARIABLE=1
echo %CHILD_VARIABLE%
:: 1
exit
echo %CHILD_VARIABLE%
:: %CHILD_VARIABLE%

export MY_VARIABLE=value
cmd
echo $MY_VARIABLE
# value
export CHILD_VARIABLE=1
echo $CHILD_VARIALBE
# 1
exit
echo $CHILD_VARIABLE
#

Notice how MY_VARIABLE is available in the child process, but the variable created within the child process it not accessible from the parent process.

This concept is incredibly powerful and is how Avalon manages the individual environment variables for your projects and applications.


PATH

Some environment variables have special meaning to your operating system, PATH is one of them. It contains absolute paths from which executables are accessed.

That is, when you type git on the command-line, your operating system performs a search in each of the paths listed in PATH until it finds it.

If you wanted to expose an executable of your own, you can add the directory containing the executable to the PATH.


PYTHONPATH

Like PATH, PYTHONPATH is where Python looks for files during import.




Events

Attach callbacks to critical events throughout the use of Avalon.

from avalon import api

def on_event():
    print("An event happened")

api.on("event", on_event)
api.emit("event")

An event happened

Some events are called by Avalon.

  • init Called as early as possible during the initialisation of an application.
  • new Called upon the creation of a new document/scene
  • save Called upon saving a document/scene



Custom Events

Aside from the built-in events, you can emit your own events too.

from avalon import api

def create_alembic():
    # Creating alembic..
    api.emit("alembic_created")