Empower your Digital Integration Hub with Fast Data and Low Code features

One of the key factors that allows companies to be competitive is the adoption of a data‑driven approach. This means that strategic decisions should be led and managed by data. Being able to make timely decisions can allow you to jump on new business opportunities as they arise. Therefore, it is fundamental to rely on data that reflects the situation in real time. This is where Fast Data comes in, a subset of Big Data that collects only essential data, i.e., data that should always be available and updated 24/7 to drive business decisions.

The context of traditional architectures

Traditional IT infrastructures are built on a request‑based paradigm. This means that the back‑end waits for an API request to start fetching and aggregating data from the different sources in which data can be stored. In this way, each request goes directly to the company’s Systems of Record (SoR, i.e. management and business applications, e‑commerce, CRM, or any other type of system that stores data), which have to manage a high workload.

However, request‑based approaches suffer from several constraints:

• High scalability and maintenance costs;
• High infrastructural complexity;
• Difficulty keeping legacy SoRs available 24/7, and reaching them via API calls;
• Lack of communication between different Systems of Record and consequently the risk of siloed data;
• Low performances and lack of agility;
• Problems adding data from new SoRs due to already existing aggregation logic;
• Inability to handle heavy workloads on legacy machines.

Digital Integration Hub, a new perspective

In response to these constraints, there is a new “event‑based” architectural paradigm that improves data integration: this is what Gartner calls Digital Integration Hub (DIH). This architecture involves the embedding of a data management layer, which decouples communication between the Systems of Record and external channels.

This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from [https://www.gartner.com/en/documents/3987367].

Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

This architectural paradigm has three major differences compared to the previous one:

• API requests insist on a data replica, decoupled from SoR, that is available in near real‑time. Responses will therefore be much faster, always up‑to‑date, and available 24/7, even if some SoRs have to be disabled for maintenance or due to a problem;
• Thanks to the Command Queue Responsibility Segregation (CQRS) model, all data is always up‑to‑date. When a record is changed, the Projections are affected, and, as a consequence, the Projections update the linked Single Views;
• Data is already aggregated and structured in Single Views, according to the needs of the specific channel from which the request arrives: the response to each request is thus available and ready to use.

Benefits of the Digital Integration Hub

The advantages of adopting a Digital Integration Hub are many, and include:

• Improved User Experience for end‑users, as systems and data requirements, are more responsive;
• API services are available 24/7, thanks to the scalability and reliability of the new systems;
• Decoupling of API services from SoRs, leading to SoRs experiencing lower workloads because they no longer have to directly handle requests from external clients;
• Support the modernization of legacy systems, which, once decoupled from service delivery, can be easily upgraded and, where possible, even replaced by cloud-native technologies;
• Support integration and aggregation of data from different SoRs, also applying transformation logic, if needed;
• Advanced and near real‑time monitoring of requests status, useful both for debugging purposes and to provide insight for business analysis.

How Mia‑Platform Fast Data works

Mia‑Platform Fast Data is the key to building an event‑based architecture. Thanks to it, you will be able to create the data management layer that decouples Systems of Records from external channels, creating your own Digital Integration Hub.

Mia‑Platform Fast Data is built mixing the Data Mesh and Data Fabric paradigms, two new approaches to enterprise data architecture. Our solution is based on three principles:

1. Data sources are the “source of true” of data;
2. Data is ingested, land on the Data Mesh Platform and is validated as raw;
3. Raw data is curated for different usage and is exposed to consumers.

Data is collected by the SoR using several ingestion methods – which depend on the type of data and the speed at which you want to process it – and then uploaded to an Apache Kafka queue. From there, data is extracted and uploaded to MongoDB Projections. Projections are replicas of the individual entities (for example, a table in a relational database) where the original data is located. This means that for each entity within a System of Records there is only one Projection created within MongoDB, which is a replica (projection) of the original entity.
The data in the projections will, in turn, be aggregated to create Single Views, a set of data that is grouped and organized to meet the needs of the channels that require it. Finally, data is exposed and made available externally via APIs. The whole process of reading data on the corresponding SoR, aggregating it, and exposing it to the different channels takes place in a very short time, thus offering near real time data.

Since Mia‑Platform Fast Data provides always available and up‑to‑date data, companies that decide to adopt it will be able to become data‑driven and access new business opportunities. Mia‑Platform Fast Data is continuously improving thanks to the introduction of low Code components, speeding up the configuration phase. Low code/no‑code approach speeds up many tasks and reduces the risk of mistakes being made in the process: for this reason, Gartner has listed low code/no code capabilities as one of the 4 cloud-native trends of 2022.

Fast Data Low Code

In order to simplify the work of developers and improve the Developer Experience, starting from Mia‑Platform v9.0, we have progressively and drastically reduced the time and effort required for configuration. In fact, it will no longer be necessary to manually write the code to search for the Single View that needs to be updated and to define how the data should be aggregated. It will only be required to write JSON configurations that specify in a declarative way how you want to implement Fast Data. In case you need more elaborate or customized configurations, there will always be the opportunity to write custom code.

In addition to this major improvement, some sections have been made completely No code, such as the creation of Systems of Record and the creation of Projections within them. In particular, now you can directly import a DDL Schema file, thus avoiding the manual creation of every single Projection. The process is very simple: first, you’ll have to download the DLL file from the database, a feature common to many DBMS. Once the file is exported, you can upload it in just a few clicks. During the upload, it is possible to check the imported data and make any necessary changes; the creation process will then be automatically carried out.

Conclusion

Mia‑Platform Fast Data is the tool that enables the construction of your Digital Integration Hub, offering the best possible Developer Experience to your development team. In fact, we believe that developers should focus primarily on developing functionality related to the company’s core business, and should reduce the time spent dealing with configuration and maintenance tasks. Thanks to the low code/no code features, not only the configuration operations will be faster, but also the probability of mistakes being made during the operations will be limited.

At Mia‑Platform, we are constantly working to bring the development team back to the center. Our continuous developments help improve the Developer Experience, so that your developers create services that are more and more competitive, scalable and adaptable to market needs.

Don’t waste time setting up your platform, just push the code!