The VNC Heisenbug

Update 16.07.: Apparently the VNC viewer also works when I copy the executable to a folder on the local system, rather than executing it from the desktop which is located on a network share. It will also work while on the desktop when I give the user full access to the entire home share (\\<server>\benutzer\home\). Giving him full access rights to just his own home directory (\\<server>\benutzer\home\<department>\<user>) results in the VNC viewer still failing with the "getaddinfo" error.

---

Two of our departments recently got a new system for their number ticket system. You press a button on a touch panel and get a ticket with a number on it. When that number is called, you proceed to the counter.

The company who delivered the new system pre-installed VNC (at this point I only knew it was *some-kind-of* VNC, later I found out it was a TightVNC server) on the machines responsible for displaying the button to request a new ticket and print the ticket, and the machines responsible for showing which numbers are up next.


I got them a VNC viewer (since I did not know what VNC server they were using I went with RealVNC) so they could connect to these machines to administrate them themselves. Testing the connection worked fine with my user. But half an hour after I gave them the viewer I got a call that they were getting an error and were not able to establish a connection. I tried again on my own computer and a test machine, everything worked fine. After hours I tested it on a remote machine where the error occurred, which also worked fine - but everything still with my main account. So I tried with one of my unprivileged test account on one of my test machines, and there it was.

The error reads "getaddrinfo: Beim Datenbankaufruf ist ein nicht behabbarer Fehler aufgetreten. (11003)" or in English "getaddrinfo: A non-recoverable error occurred during a database lookup. (11003)" ... whatever that was meant to mean. After I finally found the documentation on the error codes on MSDN I at least knew what I was dealing with. Or did I? Because the entry mentioned DNS I tried using the IP address of the machine the VNC server was running on instead of its hostname, only to get the next error message.

This time the error read "socket: Ein ungültiges Argument wurde angegeben. (10022)" or in English again "socket: An invalid argument was supplied. (10022)".

And this was were everything started to go wrong.

Considering the VNC viewer and the connection to the VNC server were working fine when used with my main account I was wondering what the difference between my main account and the test account I was using were. The most obvious difference would be that my main account is member of an AD group that gets added to the local administrators group through GPO. And through another GPO I would also get the right to use registry tools, which normal users are not allowed to use.

Thinking back to the EGVP problem where the blocked registry tools prevented the software from accessing the registry in order to extract the proxy information I tried enabled the registry tools for my test account but the error messages persisted.

Maybe the VNC viewer wants to write to some local files or the HKLM part of the registry and fails because the test account does not have the needed permissions? So I added the test account to the local administrators group of my test machine (and rebooted) but that did not change the result either. I was still getting the same errors.

I also read somewhere that the IPv6 loopback (pinging localhost did return ::1) could be an issue, so I disabled IPv6 completely on the test machine. I even enabled the localhost entry in the machine's host file. And I disabled UAC for fun and giggles. Nothing. Same old error.

I even uninstalled the anti-virus software, just in case there are some special rules in place for users from the IT department that I am not aware of. But this also did not change anything.

There are also no weird non-ascii characters in the path to the VNC viewer executable. And the path is also shorter than 256 characters.

Let's use ProcMon to investigate and look for any "ACCESS DENIED" entries. Since ProcMon requires admin privileges I used my main account's credentials to run it, filtered out the random Windows background clutter, started the VNC viewer and tried to open the connecti.... WAT? I was expecting to see the same error. But I did not. The connection worked. No error. Just the screen of the remote machine. What was going on?

I closed ProcMon and the VNC viewer, opened the VNC viewer and tried the VNC connection again. It failed. I closed the VNC viewer, started ProcMon with my main account's credentials again, started the VNC viewer and tried the connection. It worked.

Starting the VNC viewer first, then ProcMon with my main account's credentials and only then trying the connection also results in the error above. So ProcMon must be running first, before the VNC viewer is started.

What was going on here? This made no sense. Why would running ProcMon with my main account's credentials make the VNC viewer connection running under a completely different user suddenly work? Was ProcMon somehow leaking my session? Was the VNC viewer somehow latching onto the ProcMon session?

So I downloaded Process Explorer to check what happens when I start the connection. The VNC subprocess was running with the credentials of my test account, not with the credentials of my main account. Good.

I started ProcMon with the credentials of the local administrator account, just to see what would happen. Then I started the VNC viewer from the desktop of the test account and tried to open the connection. I did not expect what I got though...

The process could not find its own executable? Despite me just starting it from the very same location it claims it cannot find?! Checking Process Explorer I saw that the main VNC viewer process had spawn a child process, successfully, from the same location as the main process. So the child process was able to start (which is the process showing me the error messages) but something inside that process is failing ... using the privileges/credentials of the user running the ProcMon process. The local administrator account naturally does not have access privileges to the home/desktop directory of my test account (or any other AD account), which is why it fails to locate/access the file. But that still does not explain why it even tries to do that with the wrong credentials. Or how.

Before investigating further down this road, I wanted to first figure out what made the VNC viewer work under my main account. Differences in AD group and whatever rights and permissions came with those were still high on my list. The test account is in the AD group of another department. So I removed the account from that department's AD group and added it to the AD group of my main account's department. VNC viewer worked. Great ...

After some more testing I found the single one AD group that made the VNC viewer work. Add a user to that group, relog and enjoy all the VNC you want. I then proceeded to compare registry dumps of "HKCU\Software\Policies\Microsoft" and "HKCU\Software\Microsoft\Windows\CurrentVersion\Policies" but they were identical between the user being member of the group and not being member of it. GPOs were not the issue either ... I even dumped the whole HKCU branch and compared 40.000 lines of registry, still nothing.

Next I downloaded AccessChk to see if maybe my main account (for some reason) has more privileges on the test machine despite both the main account and the test account currently being local administrators. Using the parameter "-a *" for both accounts gave me the exact same list of Windows account rights. Another dead-end.


At this point I'm kind of out of ideas and still got two major questions:

1. Why is the VNC viewer working under my main account but not under my test account?
2. Why is the VNC viewer working under my test account when I have ProcMon running in parallel started with the credentials of my main account?

[1] https://www.reddit.com/r/sysadmin/comments/6t0jkk/weird_issue_with_vnc_client/
[2] https://twitter.com/BeingSysAdmin/status/895973601660350464
[3] https://twitter.com/BeingSysAdmin/status/896006027673579520