Reply to: Suity boomed CUK style

[Pondlife]

I've been involved in SAP implementations for years - since 1997 . Never yet seen one that was derailed because of data migration. Obviously, SY01's got no-one on his project with SAP migration experience, or he wouldn't have got away with it.

This

Btw, sometimes the most cost-effective and quickest method is to hire a couple of data entry clerks and get them to type the data in.

and that.


The source data is always a mess and invariably requires someone from the business who understands it to sort it out. Techy solutions will only get you so far.

[NotAllThere]

Data Migration is nothing to get excited about. That *is* usually done by Bobs from what I see.
How is it hard? Understand the data model on the left, understand the one on the right (you probably have guidance from an SME for this anyway), extract, transform and load. Just make sure you have several dry runs of the migration leading up to production migration. And sort out your reconciliation process. It's not rocket science is it.

I've been involved in SAP implementations for years - since 1997 . Never yet seen one that was derailed because of data migration. Obviously, SY01's got no-one on his project with SAP migration experience, or he wouldn't have got away with it.

Btw, sometimes the most cost-effective and quickest method is to hire a couple of data entry clerks and get them to type the data in.

[aussielong]

Quality bellendry there mate, well done. Data migration is not rocket science, it is dirty, ball aching and fecking tedious. Managing the migration is hard. Setting expectations, gathering and baselining requirements, pushing back on bulltulip requirements, wiping the techies noses, motivating, bargaining, chasing, harrasing and reporting progress.

That and we have no time to do requirements definition up front so I've turned the whole thing into agile sprints, which will be a first for me, delivering a data migration in an agile way.

Suity (internal dialogue): Man this migration is dirty, ball aching stuff. Man this data is hard to migrate.

Suity (wiping forehead and looking up at the *sun): I just gotta ... I just gotta map this field ... just gotta keep it together for one more day.

Suity: Hey Bob, why is production down?! Quick, lets have a standup to discuss.

Bob: Well I did what you asked for, I mapped the test case into the production scenario

Suity (scratches head): What do you mean you've used my User ID to load test data into production?

Door swings behind him as Suity hits the street again. On to the next data mapping gig. It's dirty work, but someones gotta do it.


* battered fluoro light on some trading estate on the outskirts of Slough

[suityou01]

He makes a molehill out of a molehill then shouts mountain.

6/10

[MarillionFan]

Jesus, you don't half make a mountain out of a molehill.

He makes a molehill out of a molehill then shouts mountain.

[suityou01]

Jesus, you don't half make a mountain out of a molehill.

2/10

[aussielong]

Quality bellendry there mate, well done. Data migration is not rocket science, it is dirty, ball aching and fecking tedious. Managing the migration is hard. Setting expectations, gathering and baselining requirements, pushing back on bulltulip requirements, wiping the techies noses, motivating, bargaining, chasing, harrasing and reporting progress.

That and we have no time to do requirements definition up front so I've turned the whole thing into agile sprints, which will be a first for me, delivering a data migration in an agile way.

Jesus, you don't half make a mountain out of a molehill.

[Freamon]

You get the business to guide you through the data model, not techies. Business people tend to stick around longer in organisations , perhaps because their knowledge is less transferrable.

Business people who understand their data? Sounds ideal, but you won't find any of them in a lot of organisations.

[suityou01]

Data Migration is nothing to get excited about. That *is* usually done by Bobs from what I see.
How is it hard? Understand the data model on the left, understand the one on the right (you probably have guidance from an SME for this anyway), extract, transform and load. Just make sure you have several dry runs of the migration leading up to production migration. And sort out your reconciliation process. It's not rocket science is it.

Quality bellendry there mate, well done. Data migration is not rocket science, it is dirty, ball aching and fecking tedious. Managing the migration is hard. Setting expectations, gathering and baselining requirements, pushing back on bulltulip requirements, wiping the techies noses, motivating, bargaining, chasing, harrasing and reporting progress.

That and we have no time to do requirements definition up front so I've turned the whole thing into agile sprints, which will be a first for me, delivering a data migration in an agile way.

[aussielong]

My favourite one was the promise to the head of marketing - Don't worry about the 5 years of crap data people have been entering the data migration will fix that. Someone promised as the justification of the migration that the migration would fix the missing first names, surnames and email addresses that sales people hadn't been bothered to find out over 10 years.

I provided two options:-

1) your current dataset with all the crap
2) a probably valid complete dataset (with only 8% of the original data). Needless to say neither option was acceptable outside of IT.

Reconciliation between source and target would be a nightmare if you tried cleaning up the data on the fly as you load it in to the target system.

I wouldn't do these projects. It is truly soul destroying work.

[eek]

I've just had clientco purchase me a DNB license with 250,000 downloads and 15000 credits for data cleanup.

Interestingly, I provided a sample of around 300 records to do some data matching to DNB. I checked the data, but didn't manipulate hugely. The results came back at a disapointing 60% match rate with a 5% error.

I then knocked up a simple Excel function using the Google Maps Web service. I then ran the address data through that, which returned a full address string, where I sent that off to DNB for matching with company info.

It came back with a 97% match rate, with a 3% error. The next phase is to merge contact data, and the first pass I have highlighted 30% (approximately 15000) records to remove. That Mickey Mouse fella must get an awful amount of spam.

In that case at least you have a chance of matching the data. If however the record is firstname Mark surname blank email blank you really haven't got a pray.

[MarillionFan]

My favourite one was the promise to the head of marketing - Don't worry about the 5 years of crap data people have been entering the data migration will fix that. Someone promised as the justification of the migration that the migration would fix the missing first names, surnames and email addresses that sales people hadn't been bothered to find out over 10 years.

I provided two options:-

1) your current dataset with all the crap
2) a probably valid complete dataset (with only 8% of the original data). Needless to say neither option was acceptable outside of IT.

I've just had clientco purchase me a DNB license with 250,000 downloads and 15000 credits for data cleanup.

Interestingly, I provided a sample of around 300 records to do some data matching to DNB. I checked the data, but didn't manipulate hugely. The results came back at a disapointing 60% match rate with a 5% error.

I then knocked up a simple Excel function using the Google Maps Web service. I then ran the address data through that, which returned a full address string, where I sent that off to DNB for matching with company info.

It came back with a 97% match rate, with a 3% error. The next phase is to merge contact data, and the first pass I have highlighted 30% (approximately 15000) records to remove. That Mickey Mouse fella must get an awful amount of spam.

[eek]

And if you're migrating secure data such as passwords or credit card numbers, you've got encryption etc to worry about too.

Address data can be its own special brand of fun.

My favourite one was the promise to the head of marketing - Don't worry about the 5 years of crap data people have been entering the data migration will fix that. Someone promised as the justification of the migration that the migration would fix the missing first names, surnames and email addresses that sales people hadn't been bothered to find out over 10 years.

I provided two options:-

1) your current dataset with all the crap
2) a probably valid complete dataset (with only 8% of the original data). Needless to say neither option was acceptable outside of IT.

[aussielong]

Rama Chemudupati, explains his specific experiences on a recent SAP data migration project.

SAP Data Migration

This is one of his "Key Concepts" that must be grasped to be successful:

Identifying and involving the right resources and clearly defining their roles and responsibilities can help you achieve significant success in a data migration project. Over- defining roles can mean blurring the responsibilities. Under-defining the roles can affect the migration timelines and deliverables. Involving business representatives in your migration process and ensuring their availability at various stages along the process also factor into this role definition.

[MarillionFan]

You get the business to guide you through the data model, not techies. Business people tend to stick around longer in organisations , perhaps because their knowledge is less transferrable.

WHS

And that's the Bob proof skill. Don't sit in IT. Sit in the business with IT skills, delivering to the business and explaining to the dullard IT Techies or Bobs how to do it.

Works for me. Kerchiiiiiing.