I’ve seen this a lot of the connection to the x11vnc server isn’t terminated properly.
If you’ve used that vnc install that’s on the forums, you can restart the server by logging in over ssh and running this:
/etc/init.d/x11vnc stop
/etc/init.d/x11vnc start
For a permanent fix, x11vnc has some arguments you can use when starting the server to allow multiple connections etc.
I believe -shared and -forever are applicable.