Big Data

[MrMarkyMark]

Depending on what you read volume is just one element.

As the guys have alluded to in earlier posts, it can also include breadthe of data. I've heard several instances where clients have been told they can join anything and everything and the correct result will magically pop out. Usually something plops rather than pops.

And many conventional db will manage more data than most organisations have, although there are a few instances where volume does become an issue

In general I would regard the term Big Data as marketing hype with organisations being sold a dream that they could easily crack with tools other than hadoop etc.

+1 Good design is king

[DimPrawn]

The key difference with Big Data™ is that it tries to relate unstructured/semi structured data, not replace well organised relational data.

If your data is very good and structured, it can go into a traditional RDMS and then into a data warehouse.

If your data is a billion word documents, you can't stick them in a RDMS and get any insight into them.

A good example is analyzing all the company emails, tweets about the companies products, internet articles and blog posts to identify sentiment and trends.

[MarillionFan]

I run a predictive analytics team and as part of a launch of an internal tool last month the Sales EVP told our 2000 sales for that a new Big Data sales tool using Predictive Analytics would be used to drive our cross sell and upsell opportunities.

We actually did it in a SQL database and analysed the data in Excel to produce a model. Less than one million records.

The hard part was getting the bloody data in the first place as all the sources were cloud.

It's marketing bollocks.

[DimPrawn]

Atw is doing big data, probably without realising it.

Crawling the Internet for unstructured data and analysing to summarise, catagorise, give insight into its meaning and relationships.

[Intel]

It's marketing tulip force fed to CIOs so they have something to one-up their mates on the golf course on a weekend.

BI is a journey, most of my clients barely have any idea of how to collate, analyse and monetise their internal data assets before looking at external and BD assets.

Build the foundations, then the walls etc. Trying to start with the roof is pretty difficult....

On the upside it's keeping me in gigs knocking together Enterprise Data Warehouses after the teams implementing it realise they have no idea about the quality and accuracy of their data....

[DimPrawn]

It's marketing tulip force fed to CIOs so they have something to one-up their mates on the golf course on a weekend.

BI is a journey, most of my clients barely have any idea of how to collate, analyse and monetise their internal data assets before looking at external and BD assets.

Build the foundations, then the walls etc. Trying to start with the roof is pretty difficult....

On the upside it's keeping me in gigs knocking together Enterprise Data Warehouses after the teams implementing it realise they have no idea about the quality and accuracy of their data....

That's not big data. That's analysing structured data in a DW.

Big data is analysing something like every tweet in realtime and obtaining understanding of how the World sees your products.

BTW there's 9100 tweets per second to analyse, 24/7/365. Try that in your SQL server DB.

[Intel]

That's not big data. That's analysing structured data in a DW.

Big data is analysing something like every tweet in realtime and obtaining understanding of how the World sees your products.

BTW there's 9100 tweets per second to analyse, 24/7/365. Try that in your SQL server DB.

Yep I understand the difference between structured and big data. I was referring to the journey you have to go on in order to effectively utilise big data to drive revenue opportunities in a business. If you don't have the structured data and analysis capabilities in place first trying to use big data is like putting the cart before the horse. How do you measure the impact of your use of BD if you don't have a handle on your internal data assets?

[DimPrawn]

I don't think anyone here has done big data, if they had they would realise no DB on the planet could do it. It's unstructured unrelated txt and data to be analysed in vast quantity and then apply predictive analytics to cluster, predict and gain insights.

It's not my DB is 3TB in size, I'm doing big data.

[DimPrawn]

Yep I understand the difference between structured and big data. I was referring to the journey you have to go on in order to effectively utilise big data to drive revenue opportunities in a business. If you don't have the structured data and analysis capabilities in place first trying to use big data is like putting the cart before the horse. How do you measure the impact of your use of BD if you don't have a handle on your internal data assets?

Govt might want to analyse every tax return ever submitted, each year the fields and structure are different. Would be a big task for a RDBMS to handle the vagueness and the textual aspects.

SA100 for individuals paying Income Tax
SA800 for partnerships
SA900 for trusts and estates of deceased persons
CT600 for companies paying Corporation Tax
P35 for PAYE deductions by employers and National Insurance contributions
VAT100 for value added tax

Then drive insights on likely errors, tax dodgers, fraud etc.

Not that they are doing that, ahem,

[Intel]

Govt might want to analyse every tax return ever submitted, each year the fields and structure are different. Would be a big task for a RDBMS to handle the vagueness and the textual aspects.

SA100 for individuals paying Income Tax
SA800 for partnerships
SA900 for trusts and estates of deceased persons
CT600 for companies paying Corporation Tax
P35 for PAYE deductions by employers and National Insurance contributions
VAT100 for value added tax

Then drive insights on likely errors, tax dodgers, fraud etc.

Not that they are doing that, ahem,

and under that is a structured database of tax payers. With details of how much tax they've paid every year, that they can then use to analyse the impact of the use of the big data solution. i.e. Are the actions taken effective or not?

Take away that tax payer database and you're left with a lot of analysis about data assets you don't have a handle on.

Oh, wait a minute......