Big Data

[DimPrawn]

and under that is a structured database of tax payers. With details of how much tax they've paid every year, that they can then use to analyse the impact of the use of the big data solution. i.e. Are the actions taken effective or not?

Take away that tax payer database and you're left with a lot of analysis about data assets you don't have a handle on.

Oh, wait a minute......

I don't think you get it. Luckily a lot of people do ;-)

[FatLazyContractor]

I have around 100TB of Porn stashed on the cloud across various providers. Does that constitute Big data?

[DimPrawn]

I have around 100TB of Porn stashed on the cloud across various providers. Does that constitute Big data?

Yes, if it's analised carefully.

[GB9]

I don't think you get it. Luckily a lot of people do ;-)

I'm not sure they do.

Analysing unstructured data like tweets and Facebook posts maybe one area of Big Data but I've done that before using sql server as it had to be connected to the companies structured data e.g. there may be 'likes' about a certain product and this data needs joining. And the number of tweets per organisation is vastly smaller.

IBM's definition includes voracity too, so unstructured isn't the only thing.

Either way I don't believe that the vast majority of companies need big data solutions.

[DimPrawn]

I'm not sure they do.

Analysing unstructured data like tweets and Facebook posts maybe one area of Big Data but I've done that before using sql server as it had to be connected to the companies structured data e.g. there may be 'likes' about a certain product and this data needs joining. And the number of tweets per organisation is vastly smaller.

IBM's definition includes voracity too, so unstructured isn't the only thing.

Either way I don't believe that the vast majority of companies need big data solutions.

The last bit is true. Fred's Fish and Chip Shop doesn't need it for sure. Government, military, big global companies do.

E.g. analyse every cell phone conversation on the planet. Speech recognition in every known language, and then looks for trends in terms of military threats.

Look for patterns, predict likely probabilities. In real time.

[DimPrawn]

Analyse particle accelerator collisions to look for unusual interactions.

Big Data Vital to CERN Large Hadron Collider Project, Says CTO

For example, proton and anti-proton collisions at the recently closed Tevatron accelerator at Fermilab generated an estimated 100 MB of data per second since its opening in 1967 - and that data volume represents only 100 Hz of a 1.7 MHz collision due to the limited bandwidth the data was able to be captured.

[bobspud]

I don't think anyone here has done big data, if they had they would realise no DB on the planet could do it. It's unstructured unrelated txt and data to be analysed in vast quantity and then apply predictive analytics to cluster, predict and gain insights.

It's not my DB is 3TB in size, I'm doing big data.

Heard a story about a CIO in the US that needed to have a big data platform so he didn't feel left out at cocktail parties. So he got someone to put some stuff into a salesforce instance. They scaled it so big he ended up with 196 copies of every single piece of data in his DB just so it could be found again

When I look back at the number of truly tulipe database instances that I have bumped into over the last decade its pretty obvious that schema design and old school data analysis is almost extinct so taking a tulip schema and trying to pump whatever is inside it into a data pond is never going to end well...

[DimPrawn]

Anyway, it's the Predictive Analytics that gives the value.

Clustering such as K-means, dimensionality reduction (PCA anyone?) and regression that provides the value. Anyone doing PCA using SQL Server? What about Bayes inference of future trends?

Anyone?

[original PM]

unless you can identify the businesses key drivers and understand what is needed to be done to affect these drivers then all big data does is hide the answers you need amongst lots of other data.

Also you can come up with a huge amount of incorrect corollaries because data seems to have some sort of link but it does not.

[DimPrawn]

unless you can identify the businesses key drivers and understand what is needed to be done to affect these drivers then all big data does is hide the answers you need amongst lots of other data.

Also you can come up with a huge amount of incorrect corollaries because data seems to have some sort of link but it does not.

That's where you need a data scientist and statistician.

Big Data is such a terrible name, whoever coined it needs shooting. It's data science that's big not the data size...