Wednesday, April 14, 2010

iXMF & GSoC

This year I applied to GStreamer's GSoC proposing to add iXMF support, consequently XMF.

For those who don't know iXMF, it's an interative audio format/container. It's a low-overhead meta file format capable of bundling collections of different data resources into a single file.

It's designed minding games sounds/music, audio skins, artist autonomy, playback instructions and many other aspects.

To quote iXMF's background description:
This new file format will put artistic control into the hands of the artists, keep programmers from having to make artistic decisions, eliminate rework for porting to new platforms, and reduce production time, cost, and stress.
Sounds powerful, doesn't it?

Now back to the proposal, my initial idea is to start by adding XMF meta file support (parsers and writers elements). This would be useful for packing multiple audio files (possibly with different formats) into a single file.

Once done, the fun part jumps in. Being an interactive audio format, iXMF playback is designed to be somehow event-driven (or signal-driven), e.g. a game application signals play (shot_sound). Additionally, these packed audio resources can contain scripts that allow transitions (e.g. crossfade) and playback parameters (e.g. volume, pan, etc). For example, the shot_sound resource could specify a play cross-fade effect.

Concerning this engine, I have discussed some implementation details Thiago and Stefan,
elder GStreamer developers. I have been pointed to http://gstreamer.freedesktop.org/wiki/SampleBin, which seems to be a good idea to be aggregated here. I also gathered lots of mxmf samples (mobile XMF), which can validate the parser/writer elements.

Eager as I am, I already done some research on simultaneous play of multiple audio files (with different formats) and wrote a simple element called decoderadder which does the basic job of decoding and adding the different resources. A simple test pipeline could be
gst-launch filesrc location=background.mp3 ! decoderadder name=a ! pulsesink audiotestsrc freq=100 ! a. audiotestsrc freq=200 ! a.
which plays an mp3 audio file (e.g. a game background music) and two sin waves with frequencies 100Hz and 200Hz.

Currently I'm researching how to implement seek in decoderadder and adding signals for scheduling plays / stopping resources. I'm also researching a way for caching/predecoding samples for almost-instant play (Stefan pointed me to buzztard's memoryaudiosrc element).

I must say I'm very excited about this project, so, fingers crossed!