Not long ago, selling technology in SA was as easy as selling ecstasy at a rave. The market was on cloud nine and large corporate clients were buying "one of everything".
Those heady but unsustainable days are now just a memory to IT vendors and customers. These days IT is more of a headache than a head rush and the biggest worry is IT's new deliverable - return on investment (ROI), which affects every sector of the IT industry. IT departments and vendors must justify every cent they ask for and this applies to the traditional, budgetary IT planning cycle as much as to the discretionary spend domain.
Like most technologies, data warehousing has had its share of successes and failures - mostly because businesses climbed on the bandwagon with no idea of what they hoped to achieve or of the kind or volume of data they had to handle. "To be successful, data warehousing should be implemented not as an end in itself, but as a means to an end," says Global Technology Business Intelligence CEO Johan Cloete.
"It must be designed in the broader context of a business intelligence [BI] strategy - but this fits bang in the discretionary spend area, so when you request funds for outlay here, you'd better have your business case and ROI arguments well prepared," Cloete says. "There is a common misunderstanding about what a data warehouse actually is," says Bruce Jones, manager of sales support for the SA subsidiary of Sas Institute. "It is a storage place for structured data in a format that makes it ready for analysis."
It sounds simple, so why, 10 years after data warehouses hit the scene, do most companies still own overfilled filing systems or archives that include vast amounts of valuable company information collected over many years? Few of these companies have learnt more about such business-critical issues as sales performance, customer preferences, manufacturing efficiencies, competitor behaviour and key marketplace trends.
"The challenge of any data warehousing or BI strategy is to create intelligence from data," says Jones. "An organisation must first understand its business issues, define the required intelligence and then establish the value to the business. Only then can it develop its BI corporate models."
In a typical large enterprise, there may be more than 100 different data sources running on numerous platforms and devices.
All this disparate data needs to be extracted from these sources, then transformed, so it can be transported into a repository where it can be stored meaningfully. "Once data is intelligently stored, it can be used to generate reports and analyses that lead to actionable goals," says Jones. "It does not matter whether the data comes from the Web, legacy systems, enterprise resource planning [ERP] systems, campaign management systems, call centres or cellular phones - it can easily be accessed and turned into knowledge."
However, it is not a trivial exercise; it requires extensive planning and precise execution. This is the reason many companies have not developed a data warehousing and BI strategy.
The high cost of implementing the BI systems needed to transform informal information collections into viable data repositories have counted against such projects, says Oracle SA BI specialist Itayi Mandonga.
Many companies have deployed separate and nonintegrated information systems that serve specific functions. These stove-pipe information systems generally keep their own batch of information, making it difficult to achieve a panoramic view of the enterprise or one view of the customer.
"You end up with a constantly shifting, disparate environment. Somewhere in there is usable information to help make decisions, managers meet quotas and engineers advance the company's capabilities. The challenge is to simplify the process of getting it and sharing it with people both inside and outside the organisation," says ICL's Systems Integration & Business Intelligence marketing & communications manager Tony Cross. The problem, he says, is that few companies have achieved truly integrated information systems in the mainframe world, the client/server world or the data warehousing ERP world. Processes are usually built around collecting data and transactions, then later worrying about analysing the information. "More than 75% of companies that have data warehousing systems do not access all available data, primarily because of a lack of integration."
Other hurdles are the lengthy implementation cycles required by many complex BI systems and the inability of these systems to satisfy the requirements of corporate decision makers.
Synergy Computing MD Andrew Connold believes the complexities involved and companies' focus on (rapid) ROI discourages companies from making the investment in data warehousing. "Data warehousing has a long-term and indirect payback, you can't look at it as a short-term solution," he says. But financial directors and managers are not crazy about the idea of recouping benefits in the long term. "They're looking for quick wins," he says.
There are ways around the problem, though. "Few companies realise there is a way in which a data warehousing/BI project can be launched on a limited budget and then, over time, be expanded and scaled to meet growing corporate requirements," says Mandonga.
"The answer is to first tackle the data warehousing system. Data warehousing techniques constitute the basis of all BI systems, so there is a compelling reason for users to select a single vendor for both functions and to launch their programme with just a data warehousing implementation before moving on to BI later," he says. "In this way, users will benefit from low initial implementation costs, yet have a clear vision of their integrated BI - data warehousing infrastructure of the future."
Cloete agrees: "We need to devise a new way of designing and delivering BI systems. In our experience, as a midsize to large corporate you're much more likely to achieve your goals with an enterprise-wide data warehouse." This, he says, may represent the most important opportunity for the business to roll out and entrench data quality as a corporate imperative.
This flies in the face of much conventional thinking, which says companies should rather implement multiple, function-specific datamarts. Each of these marts can provide line management with the functionality it needs, without holding up the delivery of value.
Connold argues companies can achieve early wins if the technologists address the pain points first through structured datamarts that can later be integrated and built into a data warehouse. "Here the data warehouse ROI is a payback-as-you-go, self-financing, concept, where investments are recouped each step of the way," he says. Whatever the route you take, it is clear the days of building data warehouses, which meant throwing money at an architecture over an extended period with no payback guarantees, are over. Now each aspect of delivery must be proved and the "goods" delivered as development continues.
What is clear is that the data warehouse will play a key role in the success of a BI project. "Effective data consolidation makes or breaks a BI project," says Mandonga. A successful BI system begins with consolidating data from transactional or operational systems to perform fast, accurate business analysis on high-quality data.
"Poorly integrated or low-quality data will deliver poor or worthless information analysis," he says, stressing the importance of effective data extraction, transformation and loading (ETL) from production sources into the data warehouse.
"Many BI systems fail because the process of ETL is poorly understood," he says. "Data consolidation is complex because of the enormous amounts of diverse data sets being generated by different transactional and legacy systems.
"Nonstandardised business terminology needs to be consolidated into a consistent, single view within the data warehouse long before a BI implementation is even considered," he says.
The traditional data warehousing needs include high performance, scaleability, manageability and security, but Mandonga says these can be extended to include integrated BI capabilities such as Olap (online application processing), data mining and ETL within the database itself.
" Organisations no longer have to move and maintain data in specialised servers for Olap, data mining and ETL. With no data movement and a single database engine for data warehousing and BI, companies have a faster path to information at a significantly lower cost," he says.
"But with the benefit of a solid and reliable data warehouse foundation, a BI system can quickly turn high-quality data into valuable information.
"Users will then have immediate access to that single source' of reliable data on which they can base critical business decisions," he says.
Taking BI a step further is analytical intelligence, which looks into the future as well as the past, says Jones.
"Prediction is achieved accurately through predictive modelling, forecasting and data mining, all analytical intelligence techniques," he says. By determining what a company should be doing, rather than what it has already done, managers can make meaningful decisions, for example, about changing business strategies or processes. This results in several benefits, including major cost savings.
"Data warehousing is a vital link in the intelligence value chain that is made up of BI planning, quality ETL, intelligence storage and analytical intelligence."
He cautions that a data warehouse with a purely Olap focus has limited value. "Olap is good, but it focuses on history," he says. "It may tell where companies have made mistakes or good decisions, but does not show how these can be prevented or repeated. It is backward-looking, limited to reporting on what has happened in the business in the past.
"Data warehousing is a strategic imperative that enables companies to respond quickly to shifting market conditions, but companies that focus on extracting intelligence from that data will achieve the greatest return on investments in customer relationship management, supplier relationship management and other mission-critical applications," he says.
