robots.txt - Contractor UK Bulletin Board

Visitors can check out the Forum FAQ by clicking this link. You have to register before you can post: click the REGISTER link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. View our Forum Privacy Policy.
Want to receive the latest contracting news and advice straight to your inbox? Sign up to the ContractorUK newsletter here. Every sign up will also be entered into a draw to WIN £100 Amazon vouchers!

You are not logged in or you do not have permission to access this page. This could be due to one of several reasons:

You are not logged in. If you are already registered, fill in the form below to log in, or follow the "Sign Up" link to register a new account.
You may not have sufficient privileges to access this page. Are you trying to edit someone else's post, access administrative features or some other privileged system?
If you are trying to post, the administrator may have disabled your account, or it may be awaiting activation.

DoctorStrangelove replied

9 June 2023, 11:29
It's Skynet:
Leave a comment:
eek replied

9 June 2023, 11:19
Originally posted by jamesbrown View Post

Assume you have a website and some tool is creating it for you, otherwise dunno.

https://developers.google.com/search...g/robots/intro

I'm seeing some weird files being downloaded by Safari at the moment almost like something is misconfigured on a server and Safari is treating it using path b rather than path a.
Leave a comment:
jamesbrown replied

9 June 2023, 10:34
Assume you have a website and some tool is creating it for you, otherwise dunno.

https://developers.google.com/search...g/robots/intro
Leave a comment:
ladymuck replied

9 June 2023, 10:26
Are you a robot?
Leave a comment:
NotAllThere started a topic robots.txt

9 June 2023, 07:51
robots.txt
I'm not sure where it's coming from, but robots.txt keeps being downloaded onto my laptop. Currently at robots(14).txt...

It contains:

Code:

# https://www.robotstxt.org/robotstxt.html User-agent: * Disallow:

Any ideas what's going on?
Tags: None

Working...

X