It is remarkably easy these days to make a pretty big bot. In a short time doing some EC2, nutch, hadoop, lucene work and you can develop something that absorbs vast amounts of data from the web.
Making it useful, or relevant, is the challenge.
-
Save the Humans!
Bots now outnumber humans on the Web | InfoWorldBot traffic has surpassed humans this year, now accounting for 59 percent of all site visits, according to a report released today by Distil Networks. By comparison, last year, bots accounted for 45 percent of all traffic to Distil's customers' websites.
The majority of the bot traffic was "good bots," said Distil CEO Rami Essaid, such as that from search engines and social media sites. These bots comply with "robot.txt" files that website administrators use to tell bots what to do, and bring value to the websites they visit.
He attributed the growth in bot traffic to more aggressive indexing by Bing, and by new search engines coming online.
"The 'good bots' almost doubled as a category this year," he said.
"Good bots" accounted for 36 percent of traffic this year, up from 21 percent last year. "Bad bots" were responsible for 23 percent of traffic this year, down slightly from 24 percent last year -- not because volumes were down, Essaid repeated, but because the number of "good bots" rose dramatically. Human traffic was just 41 percent, down from 55 percent last year.
I've always thought there was a certain cold robotic inhumanity about many of the posters on CUK.
Leave a comment: