• Visitors can check out the Forum FAQ by clicking this link. You have to register before you can post: click the REGISTER link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. View our Forum Privacy Policy.
  • Want to receive the latest contracting news and advice straight to your inbox? Sign up to the ContractorUK newsletter here. Every sign up will also be entered into a draw to WIN £100 Amazon vouchers!

robots.txt

Collapse
X
  •  
  • Filter
  • Time
  • Show
Clear All
new posts

    robots.txt

    I'm not sure where it's coming from, but robots.txt keeps being downloaded onto my laptop. Currently at robots(14).txt...

    It contains:
    Code:
    # https://www.robotstxt.org/robotstxt.html
    User-agent: *
    Disallow:
    Any ideas what's going on?
    Down with racism. Long live miscegenation!

    #2
    Are you a robot?

    Comment


      #3
      Assume you have a website and some tool is creating it for you, otherwise dunno.

      https://developers.google.com/search...g/robots/intro

      Comment


        #4
        Originally posted by jamesbrown View Post
        Assume you have a website and some tool is creating it for you, otherwise dunno.

        https://developers.google.com/search...g/robots/intro
        I'm seeing some weird files being downloaded by Safari at the moment almost like something is misconfigured on a server and Safari is treating it using path b rather than path a.
        merely at clientco for the entertainment

        Comment


          #5
          It's Skynet:

          When the fun stops, STOP.

          Comment

          Working...
          X