Reply to: Any _real_ experts on compression here?

[AtW]

Your holiday is not an issue, The issue here is whether you cah $%£k 72 virgins in a short period of time, can you do it or not?

[mcquiggd]

Nah, according to my client I have nearly a months 'enforced holiday' - have them delivered gift wrapped on the 17th, garnered with rose petals, and ill send them back in the middle of January, freshly deflowered...

[AtW]

I feel you OWE us something AtW....

I think 72 virgins (attractive females, of course) would be sufficient. Now thats quite a tough challenge in modurn britin if you want to stay on the right side of the law...

72 female virgins, would not that be too much for you mcquiggd?

[mcquiggd]

No, as you are a guest in my country, living off income support through my taxes, and getting preferential treatment over deserving British swamp donkeys who are literally wobbling towards the queue for 'social housing', I feel you OWE us something AtW....

I think 72 virgins (attractive females, of course) would be sufficient. Now thats quite a tough challenge in modurn britin if you want to stay on the right side of the law...

[AtW]

"binary alligned integer coding" - sounds vaguely interesting. care to expound?

Well, I will try my best here even tho I had 8 tequila (sp?) shots by now - but I am still not drunk. A good reference source would be this: http://www.cs.mu.oz.au/~alistair/abs...05inforet.html

To be honest their idea was not far away from mine, but I have to give them credit for publishing it first.

AtW, what percentage of revenue do we get for this consultancy?

You will gain my good graces, is that not enough for you peasant?

[mcquiggd]

AtW, what percentage of revenue do we get for this consultancy?

[scotspine]

"binary alligned integer coding" - sounds vaguely interesting. care to expound?

[AtW]

"An efficient indexing technique for full-text database systems"

My system evolved beyond that paper - compression methods that they use are not applicable, but recent Moffat's work on binary alligned integer coding is perfect -- I am using it to implement compression, however I wondered if there is anything better exists: all those guys tend to ignore modern search engine requirements of storing additional information about a "hit" - not just document in which it occured, but also position/type of the hit. Consequently their compression techniques do not take this into account, even though these hits actually account for majority of index size.

[threaded]

I am not sure how your SKA works, but on a guess have a look at his paper: "An efficient indexing technique for full-text database systems"

[AtW]

Noooooo, if you are indexing 65535 words your index will be smaller and quicker to traverse than if you are indexing 66000 words ...

I can't just throw out short words - only stop words like "of"/"to"/etc get this type of treatment, and suprisingly they do not account for THAT much space.

This does not answer the main question - how to compress best the index.

[threaded]

Noooooo, if you are indexing 65535 words your index will be smaller and quicker to traverse than if you are indexing 66000 words ...

[AtW]

You were not helping Franko.

threaded: there is no benefit in not indexing short words: they take as much space in binary index as long words. The issue is how to compress binary index best - provided decompression speed is fast of course.

[Francko]

"I am interested in hearing from those who know what "MOFFAT" (in context of compression) stands for"

This literally only means that you want to hear from somebody who knows what MOFFAT "stands for". Now I know. No, I don't know anything about it. Was just trying to be helpful, we can all make silly mistakes like failing a simple search. Obviously, not the russian God of all times. Sorry for myself about trying to help you (and besides, it took me a few minutes to find his personal home page, minutes that I should have used looking for porn rather than helping ungrateful pompous nerds). Pesciol ti.

[threaded]

Well, the way it works is that the domain is restricted, so if you can restrict the domain further then you can achieve better compression.

So if you don't index short words ...

[AtW]

"I am interested in hearing from those who know what "MOFFAT" (in context of compression) stands for."

Francko - the key words is "know": knowledge of what it means is not the same as parroting copy from Google's search, this is no more "knowing" than parrot who says words he remembered.