André Dieb

xmf/ixmf plugin updates

2010-09-07T13:12:00.002-03:00

For the past few months I've implemented a few XMF plugins for GStreamer's GSoC. The GSoC objectives and results are here:

Add XMF support into GStreamer (both creation and parsing of XMF files)
Have a base for XMF sub types (mXMF - mobile ringtones, iXMF - games, interactivity, among others)
Perform iXMF experiments
Get to know the GStreamer community, make professional contacts, learn the technologies in question (GStreamer, GObject, GLib, etc)
Get in touch with the XMF/iXMF working group

I'm glad to say we managed to accomplish these objectives. From the code POV, we have the following plugins:

xmfmux: capable of creating XMF files
xmfdemux: capable of demuxing XMF files
xmfbin: worth-mentioning experiment in which we tested how to perform some iXMF features within the gst context.

It's important to note that me and my mentor will maintain these plugins and keep them updated according to newly published specs. Although they're in a near-stable state, we're in the process of rounding them before pushing to -bad.

Concerning the community, we've held a conference call with key members of the iXMF working group. It was a great talk and they're definitely backing up our project. I joined the group and I may attend to following conference calls.

xmfbin experiments, ixmfmux

2010-07-19T00:15:00.006-03:00

This is the third report for the iXMF GSoC project. In this phase I've done experiments on an element called xmfbin, that would implement similar behavior of an iXMF engine. Me and my mentor have discussed the difficulties and aspects of this element for defining what we'll do next.

The experiments were all about learning more about the runtime environment required for ixmf. xmfbin was able to play simultaneously mp3 files embedded in a xmf file, created using our own xmfmux component. We had some problems when playing a mp3 file two times while using pulsesink, debug shows there may be a problem with discontinuity but alsasink did not present the same problem. We're still looking into it.

There are also other ideas regarding how ixmf playback could be done using GStreamer, we have stashed them and we'll probably revisit them on the next few weeks.

Now I'll start coding ixmfmux, which is just an extension of xmfmux. The addition is that it embeds special data structures used for ixmf effects (e.g fading, transitions, volume control, etc).

Project Schedule
Week	Date	Event	Status
0	May 18th	First component delivered	xmfdemux delivered
1	May 24th	GSoC officially begins	Designing xmfmux
2	May 31th	Coding xmfmux	xmfmux, unit tests
3	Jun 7th	Coding xmfmux	muxing of complex folder structures
4	Jun 14th	Second component delivered	xmfmux delivered, start designing ixmf playback engine and muxing requirements
5	Jun 21th	iXMF bin design and prototype	extended
5	Jun 28th	xmfbin experiments	xmfbin initial implementation
6	Jul 5th	xmfbin experiments	simultaneous playback tests with formats other than (e.g. mp3)
7	Jul 12th	xmfbin experiments	seeking, samplebin-like behavior
8	Jul 16th	GSoC Mid-term	GSoC midterm evaluations
8	Jul 19th	ixmf startup	start coding ixmfmux component for samples generation

GSoC Project Report 2 - xmfmux

2010-06-21T21:31:00.011-03:00

This is the second report for the iXMF GSoC project. In this phase I have created the xmfmux component, the remaining requirement for starting diving into ixmf.

Concerning GSoC timeline, here's how we're scheduled:

Project Schedule
Week	Event	We're at
0 (May 18th)	First component delivered	xmfdemux delivered
1 (May 24th)	GSoC officially begins	Designing xmfmux
2 (May 31th)	Coding xmfmux	xmfmux, unit tests
3 (Jun 7th)	Coding xmfmux	muxing of complex folder structures
4 (Jun 14th)	Second component delivered	xmfmux delivered, start designing ixmf playback engine and muxing requirements
5 (Jun 21th)	iXMF bin design and prototype	In progress

Important information:

Project repository: http://gitorious.org/gst/ixmf
Contact

Me: André Dieb Martins (andredieb on IRC-freenode) - andre.dieb@gmail.com
Mentor: Thiago Sousa Santos (thiagoss on IRC-freenode) - thiagossantos@gmail.com

XMF - introduction about the base format for iXMF
iXMF - introduction about interactive XMF
iXMF IASIG - interactive XMF working group (contains the draft spec)
mXMF - mobile XMF
Blog post by me about XMF/iXMF and the GSoC project
First report: xmfmux

Now to the report itself.

As you can see from our schedule above, we spent about 2-3 weeks coding the xmfmux component. The component is currently in its most basic form (doesn't support metadata yet) but it's able to fully mux xmf files and build any node structure specified (for instance, multiple folders, subfolders). This was a functional requirement for iXMF, as it specifies multiples folders for storing its different resources descriptors. Here we show some usage:

1. Single node structure, one root node that is actually the file (most common mxmf files have this layout):

filesrc location=file1.midi ! xmfmux ! filesink location=output.xmf

2. Two nodes owned by a root folder node:

filesrc location=file1.midi ! xmfmux name=muxer ! filesink location=output.xmf filesrc location=file2.midi ! muxer.

3. Subfolders:

Desired layout:

Root

Folder1

Folder2

file1

Pipeline:

filesrc location=file1.midi name=file1 xmfmux name=muxer ! filesink location=output.xmf file1 ! muxer.Folder1:Folder2:file1

Note that in this pipeline we specified the folder structure through the pad names, folders separated with : tokens.

Now on week 5, I'm studying and designing how the ixmf bin will be implemented. Concerning the xmfmux and xmfdemux components, they still lack a couple functionalities required by ixmf:

xmfdemux: Pull pads for srcpads created
xmfmux: metadata

Week 5 will be about implementing these remaining functionalities and keep designing the ixmf bin. Next report will expose the design study results and the implementation roadmap.

Vuvuzela Filter

2010-06-16T05:26:00.006-03:00

Here we'll create a filter that removes the vuvuzela sound. We'll use GStreamer (free software) and some of its plugins (also free :)).

Requirements:

GStreamer 0.10
GStreamer Base Plugins
GStreamer Good Plugins

Ubuntu users: install gstreamer0.10-plugins-base and gstreamer0.10-plugins-good through Synaptic or apt-get.

Used files:

As you might have heard about over the web, Vuvuzela main harmonics are at frequencies 233 Hz, 466 Hz, 932 Hz and 1864 Hz. This filter is basically an array of band-reject filters for attenuating (or completely removing) the signal level on those frequencies. For this, we'll be using the audiochebband plugin, part of GStreamer Good Plugins. It's a Chebyshev band pass and band reject filter but here we'll use it in band reject mode.

Vuvuzela complete filter (run in command-line):

gst-launch-0.10 uridecodebin uri="file:///home/dieb/Desktop/vuvuzela_antes.mp3" ! audioconvert ! audiochebband mode=band-reject lower-frequency=223 upper-frequency=243 type=2 ripple=50 ! audiochebband mode=band-reject lower-frequency=456 upper-frequency=476 type=2 ripple=50 ! audiochebband mode=band-reject lower-frequency=922 upper-frequency=942 type=2 ripple=50 ! audiochebband mode=band-reject lower-frequency=1854 upper-frequency=1874 type=2 ripple=50 ! audioconvert ! autoaudiosink

Note that:

This filter is attenuating a bandwidth of 20 Hz around the target frequencies. You can tune that bandwidth down (e.g. to 10 Hz, 5 Hz) for a better output sound by setting lower-frequency and upper-frequency properties to -bandwidth/2 and +bandwidth/2, respectively (e.g. 10 Hz bandwidth at 233 Hz, lower=228, upper=238)
You can also tune the ripple for a better sound. 50dB is a bit too much, perhaps you can find a smaller value for your setup

Note that this pipeline can be used for filtering live your TV sound through your computer:

gst-launch-0.10 autoaudiosrc ! audioconvert ! audiochebband mode=band-reject lower-frequency=223 upper-frequency=243 type=2 ripple=50 ! audiochebband mode=band-reject lower-frequency=456 upper-frequency=476 type=2 ripple=50 ! audiochebband mode=band-reject lower-frequency=922 upper-frequency=942 type=2 ripple=50 ! audiochebband mode=band-reject lower-frequency=1854 upper-frequency=1874 type=2 ripple=50 ! audioconvert ! autoaudiosink

Connect your TV sound output to your computer microphone/audio input (usually pink or blue connectors) and then your PC output (green connector) back to the TV (most modern TV's have blue-colored audio input jacks on the back panel). That's it. You have vuvuzela-free world cup games :-), for free.

GSoC Project Report 1 - xmfdemux

2010-05-18T23:39:00.007-03:00

This is the first report for the iXMF GSoC project. First let's have some useful information about it:

Project repository: http://gitorious.org/gst/ixmf

Contact:

Me: André Dieb Martins (andredieb on IRC-freenode) - andre.dieb@gmail.com
Mentor: Thiago Sousa Santos (thiagoss on IRC-freenode) - thiagossantos@gmail.com

iXMF related resources:

XMF - introduction about the base format for iXMF
iXMF - introduction about interactive XMF
iXMF IASIG - interactive XMF working group (contains the draft spec)
mXMF - mobile XMF
Blog post by me about XMF/iXMF and the GSoC project

For achieving the bigger goal (iXMF), GStreamer needs to support the base format (XMF). For a better understanding of the format, we decided to start the project writing a XMF demuxer element.

I started implementing the demuxer element about a week ago and today we played for the first time a XMF file. The demuxer is capable of parsing a mxmf file (mobile XMF - audio/mobile-xmf, bundle of two audio files: mDLS + MIDI) and creating sometimes-pads for each internal file.

It seems that GStreamer does not yet support mDLS, so the test concerned only the internal MIDI file (using wildmidi MIDI decoder plugin). The pipeline used was:

filesrc location=sample.mxmf ! xmfdemux ! wildmidi ! pulsesink

Still, there are a few TODO's for this element:

Pull mode for created pads: when working in sink pull mode, it would be nice to let our created pads also work in pull mode. This can be tested with the timidity plugin which supports only pull-mode.
Finish implementing file references handlers: currently we're only parsing in-line resources data. There are a few more types of reference that should be implemented before moving forward to the next element.

I'll be working on these TODO's in the next week and there should be a report concerning the next steps, which are currently under discussion with Thiago.

I'd also like to call out our fellow GStreamer developers for help, insights, suggestions, reviews, etc. It would be great to have feedback from the community.

iXMF & GSoC

2010-04-14T22:08:00.007-03:00

This year I applied to GStreamer's GSoC proposing to add iXMF support, consequently XMF.

For those who don't know iXMF, it's an interative audio format/container. It's a low-overhead meta file format capable of bundling collections of different data resources into a single file.

It's designed minding games sounds/music, audio skins, artist autonomy, playback instructions and many other aspects.

To quote iXMF's background description:

This new file format will put artistic control into the hands of the artists, keep programmers from having to make artistic decisions, eliminate rework for porting to new platforms, and reduce production time, cost, and stress.

Sounds powerful, doesn't it?

Now back to the proposal, my initial idea is to start by adding XMF meta file support (parsers and writers elements). This would be useful for packing multiple audio files (possibly with different formats) into a single file.

Once done, the fun part jumps in. Being an interactive audio format, iXMF playback is designed to be somehow event-driven (or signal-driven), e.g. a game application signals play (shot_sound). Additionally, these packed audio resources can contain scripts that allow transitions (e.g. crossfade) and playback parameters (e.g. volume, pan, etc). For example, the shot_sound resource could specify a play cross-fade effect.

Concerning this engine, I have discussed some implementation details Thiago and Stefan,

elder GStreamer developers. I have been pointed to http://gstreamer.freedesktop.org/wiki/SampleBin, which seems to be a good idea to be aggregated here. I also gathered lots of mxmf samples (mobile XMF), which can validate the parser/writer elements.

Eager as I am, I already done some research on simultaneous play of multiple audio files (with different formats) and wrote a simple element called decoderadder which does the basic job of decoding and adding the different resources. A simple test pipeline could be

gst-launch filesrc location=background.mp3 ! decoderadder name=a ! pulsesink audiotestsrc freq=100 ! a. audiotestsrc freq=200 ! a.

which plays an mp3 audio file (e.g. a game background music) and two sin waves with frequencies 100Hz and 200Hz.

Currently I'm researching how to implement seek in decoderadder and adding signals for scheduling plays / stopping resources. I'm also researching a way for caching/predecoding samples for almost-instant play (Stefan pointed me to buzztard's memoryaudiosrc element).

I must say I'm very excited about this project, so, fingers crossed!