In my previous post, I announced my intention of writing a series of posts talking about my recent experience with data warehouse build out in an agile transformation of EDS for a Fortune 100 company. Let’s continue our discussion on this interesting topic.
So, what exactly it is? What do I mean by agile data warehouse build out?
First of all, let’s talk about Data Warehouse in general. A data warehouse initiative typically involves building out data structures in the enterprise data model, pulling data from multiple sources, doing some (data) transformation, and then ‘parking’ them into the target data structure. Sounds about right?
At a high level, at least, this is the work flow for a data warehouse build out. To get to next level of details, your data warehouse builds out includes:
- Identifying the data elements and sources for that data
- Coming up with logical data model that will support the business’s analytical needs
- Often, getting this logical model; or as technical teams would say LDM (Logical data model) approved through some governance body in the organization
- Getting the physical data structures, PDM (Physical data model) created
- Extracting and transforming data from various sources to load into the physical data structures
To summarize, a typical data warehouse build out contains following workflow (of events):
 |
| Typical workflow for a Data Warehouse build out [Please note that we will be modifying this as we discuss the topic further] |
Now, let’s talk about Agile data warehouse (ADW). By ADW, I mean:
- We build the data warehouse in increments
- Delivering Potential shippable increment (PSI) of the data warehouse at a regular frequency [and not have customers waiting to get the data warehouse as one big-bang delivery]
- Involving customers through the build process
- Understanding customers, and their needs – what are the business reasons for them to request this data warehouse
- Focus on creating customer value incrementally, and not the technologies or the implementation of it. Shift your focus from technology (and data elements, data structures, primary key, foreign key, etc.) to customers needs.
Often times data folks are so focused on their tables and primary keys that they start driving the build out from that vantage point only 🙁
Let the customers’ need be your guiding beacon!
[Next post: Know thy customer!]