Short answer:
Without a USB video input box and a Linux driver, the N900 has no video input hardware.

**** around with your phone answer:
The N900's noise floor is pretty high for pure audio, and plus there is almost certainly filtering and compressing and echo-cancelling on the audio input. Treating video as an audio signal would result in those signal adjustments applying to the video and looking nasty.

However, even if you connect the yellow video-out cable to the N900's microphone-in (believe this is the "white ring" on your N900 video cable), at no point is a circuit converting this "audioized" video to RGB video for a computer screen. I don't think you can tell mencoder "this is an audio-digitized RCA signal of X voltage, convert to RGB".

It's a fun idea, since everything from Commodore 64 to modern games (Bangai-O for Nintendo DS) have used varieties audio signals to transfer arbitrary data. If you like trying stuff, go for it. If you just want video-in, use a USB device, build the source module, be ready to overclock your phone for performance, and cross your fingers.