* Linux Mint comes with Vino pre-installed, which is a nice VNC server but quite bare-bones and requires Gnome. Worst of all, it's incapable of renaming the exported Avahi service name, meaning that Mac OS clients will see duplicate entries for the machine in their sidebar (one for file shares and one for the desktop share). Therefore I uninstalled Vino and experimented with x11vnc until I had a perfect setup that works beautifully together with Macs on the network. Here is the guide for what I did.
* First, let's install x11vnc, an extremely light-weight, powerful, fast and secure VNC server, which (like Vino) is intended for connecting to the local display/keyboard/mouse and allowing complete control over the machine as if you sat at the console. But unlike Vino, we can actually configure this thing to do what we want! In this case, we'll be setting it up to run a single server that controls the entire machine as if we sat at the console.
# apt-get update
# apt-get install x11vnc
* Now create a hashed password file only readable by root. Only the first 8 characters of the password will be used due to VNC protocol limitations!
# x11vnc -storepasswd /etc/x11vnc.pwd
# chmod 600 /etc/x11vnc.pwd
* We want the x11vnc server to start at the same time as Linux Mint's MDM login manager, which is when we can be sure that the desktop X server has been started. This also allows the VNC user to connect during the login stage and actually log in without physically being at the console. In fact, the user can even log out and log back in as another user, since the x11vnc server continues running as root at all times.
* Alright, so to make it launch as root in conjunction with MDM, you need to edit the MDM Init script.
# vi /etc/mdm/Init/Default
* Go down to the bottom of the file and insert the following right before the last "exit 0" line:
if [ -z "$(pidof x11vnc)" ]; then nohup x11vnc -norc -forever -shared -rfbauth /etc/x11vnc.pwd -allow 192.168.0. -autoport 5900 -avahi -env X11VNC_AVAHI_NAME=`hostname` -desktop "`hostname`'s Remote Desktop" -scale_cursor 0.5 -repeat -bg -o /var/log/x11vnc.log 2>/dev/null 1>&2; fi
Warning: Change 192.168.0. to whatever your local network prefix is if that's not your prefix. For instance, you might be on 192.168.1. or 10.0.0. so just change it to whatever your network is. This option basically prevents unauthorized internet connections since by only accepting local-network connections. If you don't want that security, you can remove the whole "-allow 192.168.0." part.
* That's it. Reboot to start the VNC server. Here's an explanation of what all of the options above are doing:
if [ -z "$(pidof x11vnc)" ]; then ...; fi (the mdm-Init script is executed twice on boot, so this statement ensures that we don't launch x11vnc twice if it's already running)
nohup x11vnc (executes the server process without listening to the "HUP" hangup signal; this ensures that the process keeps running even when the launching "terminal" shuts down (this simply protects against the MDM process dying and taking x11vnc down with it, but is actually useless because MDM never dies on a normal system even when switching users, and may even be harmful because IF MDM somehow died then we'd probably want the VNC server to die too, but oh well). note that some people think that "nohup" protects it from dying when you log out from one user and return to the login screen; that's not true. like I said, it has to do with the actual MDM process itself dying (which should never happen). and just to clarify; the x11vnc process will of course never die just because you log out of the current user. however, x11vnc WILL do some weird stuff internally when it switches active X session, including stopping the mDNS service, starting it again, and wiping the output log file, but it all takes place within the exact same x11vnc process; no process spawning is going on (I verified with both -bg and & to see that it was indeed the same process still running))
-norc (don't search for a config file on the disk; take all options from the command line instead)
-forever (keep listening even after the first client disconnects; extremely important otherwise the server quits when the first client disconnects)
-shared (allow more than one viewer to connect at the same time; extremely important because the Mac OS VNC client makes multiple connections per session)
-rfbauth /etc/x11vnc.pwd (tell it to use the password file we created earlier (only readable if process is spawned as root, which it is since MDM runs as root))
-allow 192.168.0. (only allow clients on the same local IPv4 subnet, this also blocks ipv6 clients since we aren't adding any v6 addresses)
-autoport 5900 (try to listen on port 5900, otherwise keep incrementing until we find a free port)
-avahi (use mDNS service discovery to announce the VNC server address and port to the network)
-env X11VNC_AVAHI_NAME=`hostname` (if your computer is called "MediaCenter" for instance, then the automatic mDNS service name would be MediaCenter:0, so we force it to just plain "MediaCenter" (hostname) to merge it neatly into the Finder sidebar so that it merges itself with any exported file shares from the same machine and doesn't take up two sidebar entries)
-desktop "`hostname`'s Remote Desktop" (replaces the default "MediaCenter:0" desktop name shown as the title bar inside your VNC viewer, with a nicer title)
[[-display :0 (the physical X server display we want to share; we could also have used -find to autodetect the display) <-- not needed because MDM has already set the correct "main screen" DISPLAY environment variable in the Init script
[[-auth guess -env FD_XDM=1 (these two commands will try to find the X authority file for the login manager/current X session) <-- not needed because MDM has already set the XAUTHORITY environment variable in the Init script
[[-reopen (avoids issues with being killed by the login manager, but shouldn't be needed; in fact, the readme says that many login managers kill x11vnc as soon as the user logs in, but that it should be fine to leave this option out as long as you log in within 45 seconds - so, I tried leaving it out and waited for about a minute and a half, and had no issues with the process being killed, so I can only assume it's not needed at all when using MDM)
-scale_cursor 0.5 (this reduces the scale of the mouse pointer to half, which is useful for turning larger-than-normal HTPC cursors back to normal desktop size over VNC, but most people probably do not want this option)
-repeat (this is used for compatibility with Mac OS X's built-in VNC viewer. first some backstory: all keystrokes received by x11vnc are simply forwarded to the X server, which in turn works by looking at the time that it received the key-down event, and then waiting for a key-up event. while waiting, it will apply auto-repeats at regular intervals until it gets the key-up event. the issue with this is that x11vnc might receive a key-down event, forward it to the X server, and then become busy with a heavy screen refresh, and as a result it might not get to forwarding the key-up event in time. the X server will then think that the key has been held down for all this time, and will repeat it. this means that over high-latency connections and in certain heavy-screen-refresh situations, it might be near impossible to type over VNC. the chosen workaround was for x11vnc to default to the -norepeat flag, which tells the X server to disable all autorepeating for as long as anyone is connected over VNC. you can then use specially implemented VNC clients that do different client-side autorepeating by sending alternating down/up/down/up events for as long as the key is being held down, rather than just sending down/down/down/up. this workaround works perfectly if your client uses the down/up/down/up pattern, and supports viewer-side autorepeat while avoiding all of the latency repeat issues. however, the issue with this is that the Mac OS X VNC viewer sends repeats using the regular down/down/down/up pattern. therefore, we must tell x11vnc to keep the X server's autorepeat enabled in order for it to interpret repeated keys properly from Mac clients. however, even when load was so high that the screen refreshes hung and all keypresses became queued for seconds at a time, I never ever encountered issues with characters becoming repeated, suggesting that the -norepeat default is no longer needed for libvncserver and that the back-end queue in the latter has been fixed (x11vnc is mainly a wrapper around libvncserver and the author isn't a god; he makes mistakes with his libvncserver usage like any other human))
-bg (takes all of our given command line parameters and spawns a background process, then exits the main process, thus allowing the MDM Init script to proceed; could also have been solved with a regular "&" to spawn a background process, but I suppose it looks neater to use the built-in function)
-o /var/log/x11vnc.log (clears any existing log contents and then sends all status messages here, thereby giving us per-reboot logfiles)
2>/dev/null 1>&2 (redirects stderr to /dev/null and stdout to stderr, thus removing all nohup and x11vnc terminal output during our initial launch (the log file will still be populated by the background child process of course); this redirection also ensures that nohup won't be writing stdout of the spawned process to nohup.out, which it would otherwise have done)
* Alright, now enjoy the near-flawless VNC server! Performance and reliability is great, and it even supports clipboard sharing!
- The only known issue is that anything that triggers massive amounts of screen redraws (such as a terminal window scrolling text like crazy) can break the ability to send further input (clicks/text). The solution is to disconnect and reconnect from the VNC server.
* Sidenote: Don't be tempted to try the -inetd mode, because that is a special socket mode which spawns one process per connection and doesn't support mDNS, etc; it's useless. Also, don't try to launch this from anywhere but the mdm-Init script, because it's the best possible place for hooking into the same DISPLAY/XAUTHORITY as the main window manager.
* The only area where we could improve things is if we could somehow set it to automatically re-launch the daemon if it dies for some reason. I guess we could do that using the -loopbg option but I don't care enough to get into testing that setup.
* Oh and Linux Mint's "New Login" menu item ("log in as a new user without logging out") should be avoided at all costs, since this action launches a separate X server for the new user while pausing your old X server in the background, thereby preventing you from controlling the computer via VNC again until you either reboot or log out from the new user. The same thing happens if you hit the regular "Logout" menu item and then click "Switch User," and should be avoided as well. If you want to switch user the proper way, you must completely log out from the current user via the regular Logout action.