Six Sigma, Business Intelligence and Data

Six Sigma is a management strategy developed by Motorola in 1986.  Basically, it’s a set of tools designed to analyze and improve business processes by removing defects.  DMAIC is one of the project methodologies used in Six Sigma to improve existing business processes.   DMAIC has 5 steps: Define, Measure, Analyze, Improve and Control.   Thinking about data along these lines will greatly improve the quality of data in an organization.


A critical part of most IT projects is defining the data requirements properly.  The challenge is capturing all the nuances of the business in well defined entities and attributes so a complete picture of the business transaction is captured.   IT projects dealing directly with business processes define the initial data capture for the business.  The data needs to be complete enough to support subsequent business activity.

Customer Relations management tools are a good example.  You need to capture enough information about the customer to faciliate a good client customer relationship.


In Six Sigma, measure is the step to define what and how you’ll measure the process and what your KPIs are.   This same concept can be applied to data quality.  One approach is to measure Data Quality (i.e. whether the data is ‘reasonable’, and if the data is captured in a timely manner).   Data quality ensures the facts are correct but not necessarily business meaning.

Business Intelligence, on the other hand, is looking at the data captured during a business transaction and reporting back what that data means in a business context.   Business Intelligence is a good check that the data, as defined, represents the essence of the business transaction.


One of the key activities in Business Intelligence is analyzing all the data captured about a business and putting that data back into a business context.  Often, it’s not always possible to report as complete a picture of the business because of Data Shrinkage.  Simply put, there’s a data gap because not all data captured in the business processes are available for reporting.


This brings us to the Improve step in the DMAIC process.  Looking across the business, you’ll find many silos of data, multiple applications collecting data, inconsistent data definitions and gaps in data.  This all comes to light when the business asks a simple question like ‘What is my retention rate in the West region for medium sized customers who have been with us for at least 3 years?’   Chances are it will be difficult to find the region, size and longevity of a customer in one place.

Most likely, your Business Intelligcnce team will be asked to answer the question.   The Buiness Intelligence team is highly skilled technically with a good understanding of the business.  They have all the tools they need to cobble data together from multiple sources and answer the business question.  Unfortunately, it usually ends there.


Control is the last step in the DMAIC process.   For Six Sigma, this is the step to look for deviations in the process to prevent defects.  For data, this concept can also be used to look for defects in the data when doing Business Intelligence reporting. Inability to easily answer business questions with the data available should be a red flag indicating defects in the how the data is captured and managed.


DMAIC is a great framework for looking at data holistically for an organization.  It’s a way to think about data from when it is first captured to how it’s used to measure and understand the business.     Business Intelligence uses data to provide insight into the customers and business trends and help to drive marketing strategies and grow the business.  Too often it ends there.

Business Intelligence should also be the feedback loop to continually improve how data is captured about the business.   The feedback loop completes the data life-cycle from data collection to consumption to continuous improvement of data collection.  This feedback loop is a good way to continually improve data quality and drive business results.   The Business Intelligence team is the catalyst that drives data quality in your organization.

Posted in Data | Leave a comment

Data Shrinkage

The concept of shrinkage is considered when ordering raw materials to account for the loss of material during the manufacturing process.  Shrinkage is also used in accounting to represent the loss of finished product from the point of manufacture to final sale.  Data shrinkage is when data is lost in the process of executing a series of automated business processes.

Most of us are customers of some business.  Becoming a customer often starts with an application.  For example, to become a brokerage customer  you fill out a brokerage account application.   The application has all the data required or should I say may be required to open the brokerage account.   After filling out the form, the data is entered into a business application so the party you’re doing business with has the data they need to evaluate your application.  The first step in the process is done.

Some time in the future a business analyst, BA, is going do an analysis of brokerage customers.   Do you think all the data entered on the original form is available to the BA?  How about all the data entered into the business application to open the account?  Chances are the BA will only have access to a subset of the data about the customer due to data shrinkage.

Typically, what happens is that each process step has different requirements for data to move forward.   In the case of opening a brokerage account, the broker collects data about the new customer like annual income, assets, investment experience etc.   The broker uses this data to make sure he sells the appropriate investment product.   Once the application is accepted you proceed to the next step and open the account.

Once you open the account, you don’t necessary need to carry all the data from the application into the business application that is used to administer that account.  Initially, from the application you have a vivid the image of the customer and what you might expect in investment behavior from this new customer.  After just two steps, filling out the appliction and opening the account, you may have a much starker image of the customer based solely on account activity.  Why?

The first step of getting to know the customer has a lot of data about the customer.  The data may include notes from the broker or other ancillary data.  Once the broker has evaluated the new client, the broker enters  only the key data needed by the Open Account Business Application.  The Open Account Business Application doesn’t require all the suitability data so it’s not captured with the new account.

There are a number of reasons why the full set of data is not passed from process to process.  It can be as simple as a space restriction or to reduce re-keying in downstream systems because of a lack of system integration.  One solution for minimizing data shrinkage is to require that all data is entered into every business application.  This would be very inefficient.   A better approach is to take the time to understand your business processes and all the data collected.  Once you understand this, you can define the key data relationships so you can link the data collected in each process step to create a complete picture of your business.

Using data relationshps, based on values, enables you to leverage database technology.   Database technology helps you reduce the need to re-enter data while providing the capability to relate your data across business processes.   Relating the data from all the processes about the customer enables the BA to re-create the vivid image of the customer who filled out the application thus minimizing the affects of data shrinkage.

Posted in Data, Solution Architecture | 1 Comment

Perspective on Data

Having the proper perspective is one of the more challenging aspects of data management.   One way to approach data is to consider it in terms of information.  Data is the raw material needed to make an informed decision.   In other words, information is the result of applying process (i.e. analysis or thought) to the data.

Data consumers from different parts of an organization often require the same data, raw material, for their analysis.  But each has their own perspective and interpretation of the data. Quite often, these different perspectives raise questions of data quality when in fact you’re not actually comparing data but the intepretation of the data.

Below is a parable about Blind Men and the Elephant that illustrates how very different information can be when derived from different perspectives.  Always consider the consumer’s perspective when evaluating data quality.  Next time you’re in a discussion about data quality, data might literally be the elephant in the room!


Blind Men and the Elephant

poem by John Godfrey Saxe (1816–1887)

It was six men of Indostan
To learning much inclined,
Who went to see the Elephant(Though all of them were blind),
That each by observation
Might satisfy his mind

Blind Men and the Elephant - Elephant

The First approached the Elephant,
And happening to fall
Against his broad and sturdy side,
At once began to bawl:
“God bless me! but the Elephant Is very like a wall!”

Blind Men and the Elephant - Wall

The Second, feeling of the tusk,
Cried, “Ho! what have we here
So very round and smooth and sharp?
To me ’tis mighty clear
This wonder of an Elephant
Is very like a spear!”

Blind Men and the Elephant - Spear

The Third approached the animal,
And happening to take
The squirming trunk within his hands,
Thus boldly up and spake:
“I see,” quoth he, “the Elephant
Is very like a snake!

Blind Men and the Elephant - Snake

The Fourth reached out an eager hand,
And felt about the knee.
“What most this wondrous beast is like
Is mighty plain,” quoth he;
” ‘Tis clear enough the Elephant Is very like a tree!”

Blind Men and the Elephant - Tree

The Fifth, who chanced to touch the ear,
Said: “E’en the blindest man
Can tell what this resembles most;
Deny the fact who can
This marvel of an Elephant
Is very like a fan!”

Blind Men and the Elephant - Fan

The Sixth no sooner had begun
About the beast to grope,
Than, seizing on the swinging tail
That fell within his scope,
“I see,” quoth he, “the Elephant
Is very like a rope!”

Blind Men and the Elephant - Rope

And so these men of Indostan
Disputed loud and long,
Each in his own opinion
Exceeding stiff and strong,
Though each was partly in the right,
And all were in the wrong!

Blind Men and the Elephant - Wrong


So oft in theologic wars,
The disputants, I ween,
Rail on in utter ignorance
Of what each other mean,
And prate about an Elephant
Not one of them has seen!

copied from
Posted in Data | Leave a comment

‘Off the shelf’ Solution

A solution is a collection of business processes and data that allow the business to complete functions such as sell a product, process an insurance claim, or manage a payroll. Modeling the process and data required by the solution provides a logical representation of the portfolio of business applications needed to run and manage a business. This provides the necessary context when buying an ‘off the shelf’ solution.

Best case, you’ll have one Business Application for each Solution. In a large company, you’ll typically find many business applications supporting one logical business function. The more complex or specialized the function, the more likely you’ll see multiple Business Applications for one logical Solution.

For example, you’re less likely to find multiple payroll systems at a company than you are claim processing systems. In fact, a company like ADP exists because a payroll solution is so well understood. But even with a solution such as ADP, customizations like adding a company logo or specialized deductions for company specific purposes are still needed. To properly evaluate any vendor solution you need to have an inventory of all the applications currently providing the logical solution.

No Customizations

Beware when you hear, “We’re going to implement this solution ‘off the shelf’, no customizations!” Three months into the project you will be wondering why does the solution require so much customization.   Unexpected customization will delay the project and drive up costs.

Understanding what a solution is goes a long way towards explaining why ‘off the shelf’ solutions are customized. The solution you purchased is the instantiation of a vendor’s vision of the business processes and data. In order to sell the product to multiple companies, it has to be generic. You’ll soon realize that you need to modify this generic solution to accommodate how your company is actually organized and does business. Translation – you need to customize the product you just purchased ‘off the shelf’. Most likely, the customization of the software is what differentiates your company from your competitors.

Setting the right customization expectations

You can minimize customizations by comparing the logical model of your business processes and data to the default processes and data included in each vendor’s solution. Then choose the solution that most closely matches your company’s solution. Being aware of the gaps in the vendor’s solution allows you to set the right expectations for customization. Not knowing can be disastrous!

Posted in Cost Management, Solution Architecture | 1 Comment

Cloud Based Software Solution

Cloud computing is the latest buzz in the IT world.  How is it different from your own IT that is services based?  One word, provider.  A large company’s  IT department organizes using Service Management, typically based on ITIL to better manage services.  Within the context of each service is a provider.  For example, a large company may have a server team, a network team, and a web hosting team.  Each of these are competency centers within your company that provide the specific IT Services needed to deliver a home grown, web based software solution.

Along come IaaS, PaaS, and SaaS in the Cloud! is an example of Software as a Service or SaaS.    Bundled within SaaS is an Infrastructure, IaaS, and a Platform, PaaS.  Add your business’ content and you have your complete CRM solution from

From your business’ standpoint, you only need to be concerned with the content you add to your CRM solution.  You’re not concerned with any of the components needed to keep your software solution up and running.  Your solution’s availability is managed through service level agreements, SLAs, with  That brings us back to the provider question.  Does provide all the servers, network, web instances, and storage required for your CRM solution?  Or was your solution built on some other provider’s platform, PaaS, and/or infrastructure, IaaS?  Does it matter to you the consumer?

Thinking in terms of who is providing your IT Services helps you understand what you’re actually buying when you invest in Cloud-based solutions.

Posted in Cloud Computing, IT Services | Leave a comment