Inter-Sentient Radio Communication

Lately I've been thinking a lot more about how other sentient life forms out there beyond our solar system are supposed to interpret any communication broadcasts out to the universe. Audio RF is a very simple modulation that eats up a lot of spectrum, to correct this we will be replacing a lot of FM channels with a purely digital solution in a smaller frequency range using a proprietary codec. The same for broadcast television. Where there was once a somewhat complex video transmission standard there is now an incredibly complex and proprietary (at least to the planet Earth) standard that offers much higher resolution in a relatively smaller frequency range. To test this I'd like to put some of our brightest physicists, radio astronomers, and electrical engineers in a room together with a generic but unidentified radio stream and ask them to make sense of it. I doubt they could - we create things therefore we have both the encoder and decoder right off the bat. Have fun doing it backwards without knowing what it is you're expecting.

I work in Voice over Internet Protocol projects. Typically VoIP is rarely transmitted over radio waves that could reach out beyond our atmosphere however when it does leak out there, I'm guessing that if whoever cares out there wants to decode the OSI model well enough to get at the packet data they would then have to understand the proprietary codecs used by VoIP providers like Google Voice and Skype.

Imagine you're on the spaceship Enterprise approaching an unknown space fairing vessel and you get hailed. Why do you know about how they communicate? It's easier on television writers, that's for sure. But really what universal method is there for communication? Are we all supposed to purchase transponders from a race of inter-galactic ware hawkers that just happen to be in the hood at the time we're jaunting about in space?

You would think telepathy might be a good way to go. It possibly suffers from some of the same problems that RF would, different chemistry and biological makeup may be a factor, meeting on a common plane could be a problem.

So far I can think of only transmitting at a certain frequency and registering a bias with a long transmit and a long silence. Then of course just putting binary out there. Are aliens big or little endian? When defining a float what determines the decimal place?

I'm guessing it will all have to come down to something like a super fast quantum computer that will define a billion very useful procedures to decoding radio communication and run through them all until it finds a suitable match. Even though these computers don't exist yet, this seems like the most feasibleness idea. Heck, only one side of the communication would need one.