Well it's getting late for me, and I'm running out of ideas here. Only thing I can remotely think of right now is to check your logs, maybe you'll see something thats just not right.

In console.app, vine servers main logs are at ~/Library/Logs/OSXvnc-server.log

also it couldnt hurt to check console.log as well as I think some various generic vnc things get logged there as well

If I can think of anything else I'll give a post.

Aha, success!! I realized that I was still connected to my VPN the whole time (I was trying to access a university system earlier and forgot to disconnect). So I disconnected that, and now it works fine.
So thank you Rob, your solution of turning off the conflicting Apple Remote Desktop server did the trick.

Thanks to the rest of you for posting tips too

Weird. I just use the built in ard vnc server and have no problems. Don't know about the screen saver part though. It works just fine at work with the couple hundred different macs I've tried it with.

I've had no problem with the screen saver after using vineserver for a while now...
Interesting thing i noticed this morning, though... i rebooted my computer last night, and this morning when i tried to start vineserver, it told me mac osx vnc server was conflicting on the same port... i had not gotten that error before. So i went into activity manager and killed osx's vnc server, and vineserver worked fine... but not before i tested connecting to it without vineserver, and surprise surprise, it worked fine....
So... very weird. but solution for mac vnc problems with built in server: just reboot :-/

I've been running Vine Server for ever here on my G5 And I cannot dismiss the screen saver dialog box for anything. It tells me the password is incorrect every time. is your password all lowercase?

Ahaaaa, It doesn't like the shift key from the pop up keyboard. Dumb.