- Joined
- Apr 1, 2008
- Messages
- 12,848
- Likes received
- 2,921
- Thanks received
- 732
- Location
- Cheshire
We are trying to use a service or some software to trawl the web to obtain competitor prices
Has anyone ever done anything like this?
Any suggestions/recommendations?
TIA!!
Services out there that do this.
e.g. Datafiniti | Intelligent Web Data for Data-Driven Businesses (might be mainly US, but they'll add in stuff to your requirements)
What product line?
Hee hee.Thanks I'll give them a try - and I cannot tell you that because it would probably be too obvious where I work!

I've done a fair bit of ad hoc web scraping, usually involving a bit of Python knocked together in an hour or so. For example, I've got a script I run occasionally to archive Monday Links to AWS S3 then parse and extract to a database, so I can easily search to make sure I'm not posting the same thing twice. (I caught one I'd already posted about seven years ago recently.)
If I'd realised people were willing to pay for that kind of thing, I would have made it a plan B ages ago![]()


I've done a fair bit of ad hoc web scraping, usually involving a bit of Python knocked together in an hour or so. For example, I've got a script I run occasionally to archive Monday Links to AWS S3 then parse and extract to a database, so I can easily search to make sure I'm not posting the same thing twice. (I caught one I'd already posted about seven years ago recently.)
If I'd realised people were willing to pay for that kind of thing, I would have made it a plan B ages ago![]()
I worked with a company that used a 3rd party company to scrape prices (wont say more as will identify the business).
Apparently, it's a constant battle to keep the scraping software working because the sites being scrapped are constantly changing things around to stop it. I was told the 3rd party employed people to continuously keep the software up to date, doesn't sound fun at all.

Didn't you do tpd too?
