My experience showed that the Nokia 770 cannot play any video I give to it, unless it has been encoded specially. And even then it has problems with long videos (read: movies).
Encoding instructions can be found in the Wiki:

http://maemo.org/maemowiki/VideoEncoding

I hope the player will be improved...