In this proof of concept (currently only tested and working in Chrome 24+) I've split the mp3 in 25 parts by using the unix split command. The moment the first part is loaded then the playback starts immediately and it loads the second part. When the second part is loaded then then I create a new AudioBuffer by combining the old and the new, and I change the buffer of the AudioSourceNode with the new one. At that point I start playing again from the new AudioBuffer.

A lot of thanks to Paul (@aerotwist) for his suggestions and to Theo for letting me use his awesome track Breathe In.

You can check the source code on Github or check the demo.