• Want to receive the latest contracting news and advice straight to your inbox? Sign up to the ContractorUK newsletter here. Every sign up will also be entered into a draw to WIN £100 Amazon vouchers!

Web scraping

original PM

Double Godlike!
Joined
Apr 1, 2008
Messages
12,848
Likes received
2,921
Thanks received
732
Location
Cheshire
We are trying to use a service or some software to trawl the web to obtain competitor prices

Has anyone ever done anything like this?

Any suggestions/recommendations?

TIA!!
 
I've done a fair bit of ad hoc web scraping, usually involving a bit of Python knocked together in an hour or so. For example, I've got a script I run occasionally to archive Monday Links to AWS S3 then parse and extract to a database, so I can easily search to make sure I'm not posting the same thing twice. (I caught one I'd already posted about seven years ago recently.)

If I'd realised people were willing to pay for that kind of thing, I would have made it a plan B ages ago :ohwell
 
I've done a fair bit of ad hoc web scraping, usually involving a bit of Python knocked together in an hour or so. For example, I've got a script I run occasionally to archive Monday Links to AWS S3 then parse and extract to a database, so I can easily search to make sure I'm not posting the same thing twice. (I caught one I'd already posted about seven years ago recently.)

If I'd realised people were willing to pay for that kind of thing, I would have made it a plan B ages ago :ohwell

Didn't you do tpd too?
 
AtW should have been able to sell you the data, already scraped via his backlink scanning service, but last time I raised this as a potential additional income stream he said they didn't retain all the scraped data. Scrapes the whole internet and throws out the majority of the data. :facepalm:

Plenty of web scraping tools out there, and sure you can scrape a competitors site, but if they catch you and have said not to do such a thing (in their robots.txt and/or T&Cs page or copyright notices) then you risk being 'done'. I suppose the risk increases if you make the data public rather than capture it for internal use only, but if they are clued up they will be checking their web logs for obvious competitor activity. :eyes
 
I've done a fair bit of ad hoc web scraping, usually involving a bit of Python knocked together in an hour or so. For example, I've got a script I run occasionally to archive Monday Links to AWS S3 then parse and extract to a database, so I can easily search to make sure I'm not posting the same thing twice. (I caught one I'd already posted about seven years ago recently.)

If I'd realised people were willing to pay for that kind of thing, I would have made it a plan B ages ago :ohwell

I worked with a company that used a 3rd party company to scrape prices (wont say more as will identify the business).

Apparently, it's a constant battle to keep the scraping software working because the sites being scrapped are constantly changing things around to stop it. I was told the 3rd party employed people to continuously keep the software up to date, doesn't sound fun at all.
 
I worked with a company that used a 3rd party company to scrape prices (wont say more as will identify the business).

Apparently, it's a constant battle to keep the scraping software working because the sites being scrapped are constantly changing things around to stop it. I was told the 3rd party employed people to continuously keep the software up to date, doesn't sound fun at all.

Seems many companies are wasting resource scraping or preventing scraping when the better solution would be for the target to provide an API to sell the info. If it's getting 'stolen' anyway, they may as well make money from it. Also makes it more solid in court if the data is licensed via an appropriate channel and others are stealing the data to avoid paying the licence. :smokin
 

Advertisers

Back
Top