Suity boomed CUK style

[eek]

I've just had clientco purchase me a DNB license with 250,000 downloads and 15000 credits for data cleanup.

Interestingly, I provided a sample of around 300 records to do some data matching to DNB. I checked the data, but didn't manipulate hugely. The results came back at a disapointing 60% match rate with a 5% error.

I then knocked up a simple Excel function using the Google Maps Web service. I then ran the address data through that, which returned a full address string, where I sent that off to DNB for matching with company info.

It came back with a 97% match rate, with a 3% error. The next phase is to merge contact data, and the first pass I have highlighted 30% (approximately 15000) records to remove. That Mickey Mouse fella must get an awful amount of spam.

In that case at least you have a chance of matching the data. If however the record is firstname Mark surname blank email blank you really haven't got a pray.

[aussielong]

My favourite one was the promise to the head of marketing - Don't worry about the 5 years of crap data people have been entering the data migration will fix that. Someone promised as the justification of the migration that the migration would fix the missing first names, surnames and email addresses that sales people hadn't been bothered to find out over 10 years.

I provided two options:-

1) your current dataset with all the crap
2) a probably valid complete dataset (with only 8% of the original data). Needless to say neither option was acceptable outside of IT.

Reconciliation between source and target would be a nightmare if you tried cleaning up the data on the fly as you load it in to the target system.

I wouldn't do these projects. It is truly soul destroying work.

[suityou01]

Data Migration is nothing to get excited about. That *is* usually done by Bobs from what I see.
How is it hard? Understand the data model on the left, understand the one on the right (you probably have guidance from an SME for this anyway), extract, transform and load. Just make sure you have several dry runs of the migration leading up to production migration. And sort out your reconciliation process. It's not rocket science is it.

Quality bellendry there mate, well done. Data migration is not rocket science, it is dirty, ball aching and fecking tedious. Managing the migration is hard. Setting expectations, gathering and baselining requirements, pushing back on bulltulip requirements, wiping the techies noses, motivating, bargaining, chasing, harrasing and reporting progress.

That and we have no time to do requirements definition up front so I've turned the whole thing into agile sprints, which will be a first for me, delivering a data migration in an agile way.

[Freamon]

You get the business to guide you through the data model, not techies. Business people tend to stick around longer in organisations , perhaps because their knowledge is less transferrable.

Business people who understand their data? Sounds ideal, but you won't find any of them in a lot of organisations.

[aussielong]

Quality bellendry there mate, well done. Data migration is not rocket science, it is dirty, ball aching and fecking tedious. Managing the migration is hard. Setting expectations, gathering and baselining requirements, pushing back on bulltulip requirements, wiping the techies noses, motivating, bargaining, chasing, harrasing and reporting progress.

That and we have no time to do requirements definition up front so I've turned the whole thing into agile sprints, which will be a first for me, delivering a data migration in an agile way.

Jesus, you don't half make a mountain out of a molehill.

[suityou01]

Jesus, you don't half make a mountain out of a molehill.

2/10

[MarillionFan]

Jesus, you don't half make a mountain out of a molehill.

He makes a molehill out of a molehill then shouts mountain.

[suityou01]

He makes a molehill out of a molehill then shouts mountain.

6/10

[aussielong]

Quality bellendry there mate, well done. Data migration is not rocket science, it is dirty, ball aching and fecking tedious. Managing the migration is hard. Setting expectations, gathering and baselining requirements, pushing back on bulltulip requirements, wiping the techies noses, motivating, bargaining, chasing, harrasing and reporting progress.

That and we have no time to do requirements definition up front so I've turned the whole thing into agile sprints, which will be a first for me, delivering a data migration in an agile way.

Suity (internal dialogue): Man this migration is dirty, ball aching stuff. Man this data is hard to migrate.

Suity (wiping forehead and looking up at the *sun): I just gotta ... I just gotta map this field ... just gotta keep it together for one more day.

Suity: Hey Bob, why is production down?! Quick, lets have a standup to discuss.

Bob: Well I did what you asked for, I mapped the test case into the production scenario

Suity (scratches head): What do you mean you've used my User ID to load test data into production?

Door swings behind him as Suity hits the street again. On to the next data mapping gig. It's dirty work, but someones gotta do it.


* battered fluoro light on some trading estate on the outskirts of Slough

[NotAllThere]

Data Migration is nothing to get excited about. That *is* usually done by Bobs from what I see.
How is it hard? Understand the data model on the left, understand the one on the right (you probably have guidance from an SME for this anyway), extract, transform and load. Just make sure you have several dry runs of the migration leading up to production migration. And sort out your reconciliation process. It's not rocket science is it.

I've been involved in SAP implementations for years - since 1997 . Never yet seen one that was derailed because of data migration. Obviously, SY01's got no-one on his project with SAP migration experience, or he wouldn't have got away with it.

Btw, sometimes the most cost-effective and quickest method is to hire a couple of data entry clerks and get them to type the data in.