Drupal Question

[MarillionFan]

Have a project where the CMS is written in Drupal. The business wants to extract all the data from the CMS and move it into a central database & use internal reporting.
At present a 3rd party charges the client per report they build through the front end which is costing a fortune.

They're saying it would be expensive for them to write reports to extract the data and are offering to just give us the reports they output. Ideally I need to understand the schema and ideally get them to give me a copy of the data. They're saying there is no backend database.

Questions for the panel are:-

(1)Is it complex?
(2) Is there a schema or not
(3) I read it has a backend MySQL database, would this be correct and shall I just tell them to send a copy of that?

[MrMarkyMark]

Have a project where the CMS is written in Drupal. The business wants to extract all the data from the CMS and move it into a central database & use internal reporting.
At present a 3rd party charges the client per report they build through the front end which is costing a fortune.

They're saying it would be expensive for them to write reports to extract the data and are offering to just give us the reports they output. Ideally I need to understand the schema and ideally get them to give me a copy of the data. They're saying there is no backend database.

Questions for the panel are:-

(1)Is it complex?
(2) Is there a schema or not
(3) I read it has a backend MySQL database, would this be correct and shall I just tell them to send a copy of that?

Looks very painful to report on in the way you describe.
Of course there is a schema, however it is likely to be extremely complex and totally unsuitable to report on.

I guess you would be looking at a series of transforms to get the data into a shape fit to report on.
Then you run into a complete lack of business requirements....

https://www.drupal.org/forum/general...is-methodology

[MarillionFan]

Looks very painful to report on in the way you describe.
Of course there is a schema, however it is likely to be extremely complex and totally unsuitable to report on.

I guess you would be looking at a series of transforms to get the data into a shape fit to report on.
Then you run into a complete lack of business requirements....

https://www.drupal.org/forum/general...is-methodology

Thanks for that. I'll take a look. Having looked at the application I was able to generate a decent relationship model. Just not sure yet if that's what it's doing back end.

If they give me report extracts I need to work out how to reprocess that into something useful. Not ideal.

[MrMarkyMark]

Thanks for that. I'll take a look. Having looked at the application I was able to generate a decent relationship model. Just not sure yet if that's what it's doing back end.

If they give me report extracts I need to work out how to reprocess that into something useful. Not ideal.

No problem, glad to help rather than taking the P*ss for once

Yeh, but what you see isn't usually reliable when it comes to the back end model.

Having worked with such things such as Siebel and SAP, we have always transformed it somewhere else, either a data warehouse or SAP BW Cubes for reporting.
The schemas have around 10,000 tables possibly more, most of SAPs being in German and Seibel's named numerically

Anyway to manage expectations, generally not straightforward and certainly not easy.

[original PM]

No problem, glad to help rather than taking the P*ss for once

Yeh, but what you see isn't usually reliable when it comes to the back end model.

Having worked with such things such as Siebel and SAP, we have always transformed it somewhere else, either a data warehouse or SAP BW Cubes for reporting.
The schemas have around 10,000 tables possibly more, most of SAPs being in German and Seibel's named numerically

Anyway to manage expectations, generally not straightforward and certainly not easy.

Good to know - we are moving onto SAP in next year or so.

We will also be going down the data warehouse model - although we are more likely going to start with a data puddle - then grow to a pond then lake etc - generally only cherry pick the data needed to put into the warehouse rather than grabbing it all and then still not having a clue what it is.

May work - may not!

[MrMarkyMark]

Good to know - we are moving onto SAP in next year or so.

We will also be going down the data warehouse model - although we are more likely going to start with a data puddle - then grow to a pond then lake etc - generally only cherry pick the data needed to put into the warehouse rather than grabbing it all and then still not having a clue what it is.

May work - may not!

To design a good DW you need strong reporting requirements.

The writings of Ralf Kimball are a good place to start, with the back end design.
Some use BW for the reporting, some use other solutions, I believe IBM do one based on their DB2 database and a SAP R3 connector.

if you have massive volumes you can incorporate the hosting, of one or more components, on the SAP HANA in memory solution.

[MarillionFan]

To design a good DW you need strong reporting requirements.

The writings of Ralf Kimball are a good place to start, with the back end design.
Some use BW for the reporting, some use other solutions, I believe IBM do one based on their DB2 database and a SAP R3 connector.

if you have massive volumes you can incorporate the hosting, of one or more components, on the SAP HANA in memory solution.

You've used the dirty K word the team I'm working for says is no longer necessary.

With Big Data Platforms back end design is no longer needed. Chuck it into an Hadoop/MapR data lake, chuck in some Spark, Parquet, Apache drill and unlimited nodes and it does it all for you. Piece of piss really. Who needs back end design and structure anymore?

[eek]

You've used the dirty K word the team I'm working for says is no longer necessary.

With Big Data Platforms back end design is no longer needed. Chuck it into an Hadoop/MapR data lake, chuck in some Spark, Parquet, Apache drill and unlimited nodes and it does it all for you. Piece of piss really. Who needs back end design and structure anymore?

People that want to know they have 100% of the data they expect and not somewhere between 60-99%

[MrMarkyMark]

You've used the dirty K word the team I'm working for says is no longer necessary.

With Big Data Platforms back end design is no longer needed. Chuck it into an Hadoop/MapR data lake, chuck in some Spark, Parquet, Apache drill and unlimited nodes and it does it all for you. Piece of piss really. Who needs back end design and structure anymore?

Depends on what you are dealing with, if low integrity unstructured data then, sure, something like Hardoop is good, its also good for batch processing at great speed.

I saw and heard the same before when a large ISP I was working for bought a state of the art Netezza box, split node processing, the dogs boll*cks at the time etc..

The ISPs German guys spouted the same shizzle as your team, who obviously can't model data as they don't have the skills.
However, credit where credits due, your team do an absolutely fantastic line in the latest buzz words .

The unstructured data and structures went on.

The Netezza box shuddered to a halt shortly afterwards, what was a Ferrari turned into an Allegro at best.

Worked fine prior with all the well modeled stuff on it

[original PM]

To design a good DW you need strong reporting requirements.

The writings of Ralf Kimball are a good place to start, with the back end design.
Some use BW for the reporting, some use other solutions, I believe IBM do one based on their DB2 database and a SAP R3 connector.

if you have massive volumes you can incorporate the hosting, of one or more components, on the SAP HANA in memory solution.

And thats the key - we still get reporting requirements for one of the legacy systems we have had in place for ten years - to expect to be able to get exactly what you need defined upfront and for it not to change is naive i reckon.

In addition if you give people too much data and to many reports then they loose all value and if you simply give them a data warehouse with everything in then 1000's of hours will be spent creating reports nobody cares about.

Ideally you need to give people just enough to give them the info they need to do the job they need to do.....

Well that's my view anyway.