I have my custom soundtrack all in AAC (.m4a) format, and the music plays just fine, I think the game should support anything the ps3 supports playing from the xmb menu.

Did you try ripping a CD on the ps3 and then using the playlist from that?