The ecosystem around data is a vast universe. It is so diverse that for any organization to make sense out of the available would require implementation of systems to manage, monitor, analyse, and interpret data. For enterprises today, data is a major fuel that propels all decision making within the organization. However, even with this criticality, we are seeing data been stored in isolated systems, making it difficult for the organization to analyse the data. Some of this data is stored in data warehouses or data hubs and some are lost in what is called data lakes.
What is data hub?
A data hub is a modern data storage system that helps organizations to consolidate and store enterprise-wide data. It also allows companies to push data into other systems such as business intelligence systems or AI engines for further analysis. Those enterprises who are looking to operate data in silos should understand that having data will completely streamline their data management process, and smoothen data flow across the enterprise.
There are multiple technologies such as data warehousing, data science, and data engineering that culminate into a data hub architecture. More than a technology, it can be considered as a methodology to ensure effectiveness in managing data and how the data can be stored, to help organizations process further.
How does Data Hub work?
Once its implemented, each user or delivery partner, or operator has to execute a usage agreement that gives them permission to transfer data securely to the data hub repository. This is to ensure the confidentiality of the data that users have access to. The transfer of data happens through a secure and recognized integration methodology.
The collected data is made centrally available and is standardized for uniformity. Subsequently, there will be a series of analytics run on the collected data to provide meaningful information, across departments, operating units, and other sectors. Finally, the data gets pushed back to respective systems for further consumption. This is explained in a simplified diagram as mentioned below
Diagram Source: Dataversity(1)
Why Data Hub?
A major reason why any organization needs a data hub is to connect all data touchpoints and make the data available at a central location – technically termed as data integration. At a fundamental level, it provides subscription capabilities. However, when you implement it effectively, there are numerous other factors, which makes it a go-to-framework for enterprises
-
Security
Most companies enforce security measures by defining access controls on who can access what kind of data. For instance, companies don’t want to give access to Finance and HR data to some set of employees or probably customer data should only be restricted to Sales and Finance teams. Having it will ensure that your organization hierarchy is well defined, data access points are well classified, and the controls are put in place.
-
Cost-Effective
Imagine you have multiple systems and you have somehow integrated these systems, but it is not seamless. There is already an investment you made in having these individual systems and you invested further in integrating these independent systems. However, since it was not full proof, there is still the challenge of not having visibility. Over a period, this investment becomes a huge operational expenditure. If you implement it, you get rid of unwanted integration touchpoints and have a single point-to-point integration, making the overall project more cost-effective.
-
Agile
Implementing a data hub makes the entire framework agile. It expedites the integration of other business systems and the flow of data becomes fast and seamless. In absence of It, there will also be a scenario where systems will try to fetch or call data from other systems. Then there is the creation of integration touchpoints and interfaces, adding weeks and weeks of implementation time. Having it ensures all data is available at a central location through a set of APIs, access policies, and a well-defined subscription process.
Types of data hub
In this section, we shall look into the various types and what are the different types of end touchpoints
- Master Data Hub: In this type, the endpoints are usually operational systems. The data is authored either in the hub or at the endpoint
- Application Data Hub: Here again the data endpoint is an operational system. The difference is in the data authoring because, in this type, data is authored in the hub and not at the endpoint.
- Integration Data Hub: In this type, data authoring happens at the endpoints. These endpoints can be of various types such as operational systems, analytical tools or engines, or any external entity.
- Reference Data Hub: In this type, the data is created and stored either in the hub or at the end, depending upon the business scenario. Here also, the endpoints are similar to the integration data hubs, such as operational systems, analytical tools or engines, or any external entity.
- Analytical Data Hub: Analytical data hubs store or create data on endpoints only, which are operational systems.
Data Hub vs Data Lake
If we look at the data warehouses, data lakes, and data hubs, people say that they are interchangeable. However, they are different in some ways and they usually complement each other. Let us look at a comparison between the data hub and the data lake.
| Data Hub | Data Lake | |
|---|---|---|
| Primary utilization is around operational processes. | Data lake is primarily used for analytics, machine learning and reporting. | |
| Usually, it is a structured data set. | Data like can be structured and unstructured. | |
| Stringent governance process to enforce rules. | There is no strict governance to enforce rules for accessing data lakes. | |
| Quality of data managed in data hub is extremely high. | Quality of data stored and managed in a data lake is of medium or low quality. | |
| Provides real-time integration with bi-directional flow of data from/to other systems. | The flow of data is completely unidirectional, which is usually ETL or ELT in batches. |
Over and above the aforementioned differences, data hub is primarily considered as a driver of enterprise business processes, while data lakes are majorly focused on processes around machine learning.
The benefits of a data hub
By now we have got an understanding of what it is and how it functions. We also know the significance of having this platform across an organization. Here are some important benefits of implementing a data hub across an enterprise.
A fundamental benefit of having it is to enable sharing of data. This is done by connecting data creators or sources and data users or consumers. These touchpoints are also known as endpoints and they interact with the Data Hub by pushing data into it or retrieving data. The hub is a junction, which gives visibility of the data flow.
Another benefit is that it establishes seamless and real-time connectivity of different business systems. This ensures that a major challenge around data exchange is addressed, particularly if data needs to be exchanged in a faster response time.
To summarize, the benefits can be put into four buckets
- Consolidation of data stored in silos into a unified system
- Flexible and high-performance system to manage workflow
- Better visibility and ease of access to data across the organization
- A unified system with a unified interface
Examples for Data Hub Technologies
As mentioned earlier, a data hub is not just a technology but more of a platform and an approach adopted by organizations to centralize the view of data across the board. However, we do see many products that are sold in the market. Here are few examples that are sold as technology products in the market.
- Google Ads
- Cloudera, Enterprise
- Cumulocity IoT
Additionally, we also see SAP as another example. The below diagram gives an idea about the structure of the data hub and the interaction of SAP’s data hub with other business systems and technologies.
Source: SAP(2)
Final Thoughts
Today, since organizations have multiple operating units, spread across different geographical locations, it is important for the management to centralize the data that will help them to extract as and when required, to make an informed decision. Having a data hub is more of a platform than just a technology framework.
