Originally posted by BrowneIssue
View Post
Because of the Robots Exclusion Standard (or Protocol, depending on who you talk to).
It stops spiders indexing stuff they're not supposed to. Effective use of it can be of great importance in SEO.
Comment