Film industry partners Lucasfilm and Sony Imageworks have teamed up to create a compact exchange format called Alembic. The effort was announced last year and version 1.0 made its debut at Siggraph 2011. The developers say performance has exceeded their goals thanks to data de-duplication.

Last year Lucasfilm and Imageworks announced the creation of Alembic, their partnership to create an exchange format for content. The goal was to create an extensible, data representation scheme to allow artists to share files. The two companies realized they should collaborate on a format when they recognized that they were both working on the same types of problems. Tommy Burnette of Lucasfilm Singapore, and Rob Bredow of Sony Pictures Imageworks led the development team.

Their first concern was for 3D animation content but they recognize the problem is a broad one in the industry and Alembic is designed as a container format that can accommodate different types of data files. An alembic is a device used in alchemy. It’s a still used to distill magical formulas, and that pretty much is what the companies see themselves as doing. Alembic distills complex animated scenes into application independent files with baked geometric results. The resulting files can be imported using supporting software.

There are few forces more powerful for industry cooperation than customers with problems they want solved. Lucasfilm and Imageworks are big customers and they have big friends. The Alembic effort has the enthusiastic backing of Autodesk, the Foundry, Luxology, Side Effects Software, and others. Alembic has gotten this support because it’s not an exchange format in the sense that it allows files to be exchanged and edited between software products. It protects proprietary content but allows further work to be done on files. For instance an animated scene can by handed off for lighting and rendering or additional work such as flesh simulation, or cloth can be done on an animated character. Physical simulation can be added to animations, and so on.

Burnett said that once they announced the standard and got the participation of the software vendors, they expected to be able to come up with a format within a few months. That didn’t happen. It took the companies a year to figure out how to distill the huge amounts of data created for digital effects and animation into portable files. Key to the effort, is the idea of de-duplication. The ability to identify geometric shapes and store them only once.

Alembic 1.0 automatically recognizes repeated shapes and only writes a single instance to disk. The shapes are accessed as referenced data. As a result Alembic is compact. Its developers describe it as fast, efficient, and reliable. At the Siggraph event introducing Alembic 1.0, Rob Bredow said it’s being used for the new Men in Black and they saw a 48% reduction in disc space used. The savings are even more dramatic comparing the proprietary file formats used by the studios to share files between different groups involved in a show. The developers say that in the case of a hero deforming humanoid character including hair, shot caches have been reduced by more than 70% and they’re seeing reductions of up to 98% for the kinds of “complex, deeply hierarchical and mostly rigid assets” used to create the Transformer characters.  The format has the ability to speed up production because images can be written to disc faster and they can be accessed faster letting artists get back to work creating art.

Alembic is an open standard. It’s available for download at http://www.alembic.io. Bredow says they’re looking forward to the industry coming together to further improve the format. At Siggraph, Autodesk showed Alembic support in its Maya software and will include support in its 2012 Subscription Advantage Pack just announced at the conference. Luxology will support Alembic in Modo and the company showed a technical preview. The Foundry will include support in Katana, which will come out in Beta in September, and Houdini’s Side Effects 11.1 which is being released at the show will also have Alembic support. Nvidia and Pixar Animation’s Renderman will also support Alembic.

What do we think?

Although content creators dream of the day when they can freely exchange files like napkin sketches, the reality is that the challenges can be very specific. Furthermore, artists may want to freely exchange files but studios want to protect their data. Collada, the collaboration format maintained by the Khronos Group is seeing adoption by the game industry and companies like Daz and Smith Micro are seeing it as a useful format for the emerging consumer 3D market. However, game developers have devised their own ways of sharing data and often they have tweaked Collada to suit their internal practices.

Alembic has been developed to address the specific problems experienced around exchanging animation and visual effects files in the movie industry, but its developers hope it is flexible enough to be used for a broad range of digital content. Proprietary files like Autodesk’s FBX are going to continue to exist as well. Clearly, there is no magical format that can handle all the challenges inherent in exchanging digital content.

The film industry, for all its security concerns and secrecy, is also very well structured to drive an initiative like this. Collaboration is the name of the game and creative people in the film industry know each other and work together across studios. Alembic has the enthusiastic support of the in-crowd and they will make it work. The Alembic team is pledging that their format will remain consistent and compatible. – K.M.