How do search engines work?

Adam Kessel recently commented in his blog about how MSN found his 'hidden' domain. Adam appears to think that by not spreading a URL of a webserver, he can hide the site from search engines.

As Google explains, however, hiding a site in such a way is plainly impossible.

That being said, of course MSN should honour robots.txt... if it really is MSN. Adam doesn't provide the IP address of the robot that crawled his 'hidden' webserver; but of course, if he runs whois <IP address of the bot requesting the page>, he can easily figure that out... and complain if required.