SKA news - more than 1 trillion unique URLs found - this beats Google's figure

[Coalman]

Congrats - whats next?

[Spacecadet]

Congrats - whats next?

according to TPD rules:
1000001000001 being palindromic is a significant number

[HairyArsedBloke]

Just checked HAB Inc's entry and it has sheadloads of URL's dating back to when I was hosted elsewhere - I moved it something like two years ago.

Correct links are probably < 0.01 % in my case.

[ThomasSoerensen]

Yes.

http://www.majesticseo.com is our backlinks search engine that now has more unique URLs than announced by Google last year. I'd love for them to update on their current number so that we could catch up with it shortly...

Every day SKA run by SKA Troopers crawls around 350-400 mln URLs with total size of crawled and analyzed data being over 10 TB - every day.

ta for good wishes everyone - SKA is developing nicely - get ready for even bigger SKA news in the future.

I just entered my no-longer-existing site and got a very nice report.
Serious question, how do you get rid of no-longer-existing pages?
Do you re-build your index regularly?

[NetwkSupport]

i may be being stupid but whats the use of it?

[bogeyman]

i may be being stupid but whats the use of it?

Shsssssssss!

[AtW]

I just entered my no-longer-existing site and got a very nice report.
Serious question, how do you get rid of no-longer-existing pages?
Do you re-build your index regularly?

Why would we want to get rid of this data? It's perfect for deciding if you want to buy a particular domain that may have expired or no longer working.

We now rebuild index (does not mean all pages recrawled) every month, and we will mark backlinks as deleted if upon successful recrawl they are no longer present on original page.

[AtW]

Just checked HAB Inc's entry and it has sheadloads of URL's dating back to when I was hosted elsewhere - I moved it something like two years ago.

Correct links are probably < 0.01 % in my case.

There is a date on when links were found. There are also flags if they were deleted (marked when we recrawl that page successfully).

Our links are correct - you say it yourself, only you suggest that they are not up to date: we keep complete view of the web as we find it, the same as Google, some links are broken, some expired - all that is useful information for an SEO if they want to take advatage of it.

It is also useful data for our future full-text search engine.

[minestrone]

As I have said before about the search engine, search on Manchester United and abolutely nothing comes up about football.

[DiscoStu]

As I have said before about the search engine, search on Manchester United and abolutely nothing comes up about football.

It isn't really a search engine as yet