In the design and planning of a data management system with flexible storage capabilities, it is important to reference modern technological advantages in mixed data architectures. As applications, software, and data are an interconnected system that drive the interactions between businesses and information, the information needs to be kept in a secure and portable manner. Engineering a cloud-based solution with data pipeline capabilities to manage the data flow, offering mixed storage for submitted data to documents, and combining them into a management portal, requires a data lake solution.
Through a few steps, Financial Technology (FinTech) processors can submit account information, digitally load trade confirmations for the accounts, and provide these in a structured format from structured and unstructured data through a bundling process. If the financial institution needs to provide the data to the account holder, or with the account holders consent, to another financial institution, the data will be encapsulated in an accessible form. The financial services team, investment advisors, fund managers, and account holders will be able to access a universal application with Role-based Access Control (RBAC) and Attribute-based Access Control (ABAC) that allows them to complete tasks and view the required financial information.
Operationalizing the data
In the primary access layer, the users will access a Microsoft Power Pages application that hosts the Entra Id authentication portal, providing a low-code and secure development method, connected to a secure and manageable Identity Platform (IdP). In this platform account holders will be able to review their finances and provide information, financial services team will enter information and results, investment advisors will oversee the results and work with the fund managers to direct the assets of individuals. These main roles will require explicit permissions that have RBAC/ABAC details attached, limiting the view of the account holders from seeing other data while also restricting teams, traders, and managers from accessing data from account holders in which they do not interact with. These pages are designed to be Payment Card Industry Data Security Standard (PCI DSS) compliant, providing the most secure browsing experience available for the entire FinTech application.
When information is submitted through the application, it is sent through Azure Lakeflow Connect (Databricks) to Azure Lakeflow jobs to be process by a decision engine. Once the data is processed and updated it will be parsed and either structured into an Azure SQL Database (plaintext) or stored in Azure Blob storage (documents). The storage of this data is secured in a PCI DSS compliant manner that will protect the information, similarly to the user interface layer.
As the information is retrieved from the data lake for review, Microsoft Power BI embedded in Microsoft Power Pages can provide actionable insights and analysis, scoped to the services that the FinTech provider offers, these will be built based on the different user stories of the organizational hierarchy and end-user account holders. The financials can also be searched based on account holder with Azure AI search, which will be configured to be PCI DSS compliant, as it provides these services in the PCI DSS environment, but requires limitations to access based on the processing of Personally Identifiable Information (PII), to be fully compliant.
As the system is built on Platform as a Service (PaaS) software, through an Enterprise platform, an account with administration access will be required to submit Microsoft Azure support requests. These tickets will be prioritized based on the cloud contract and can be channeled through the contacts provided when establishing the Memorandum of Understanding (MoU). In the event of cybersecurity threats and outages, these contacts will provide assistance and information around remediation available from Microsoft Azure as outlined in the Service Level Agreement (SLA).
Securing the lakehouse
Throughout the Azure cloud system behind this data lakehouse, PII will be passed through HTTPS/TLS handshakes, between securely networked PaaS systems, managed and auditing by Microsoft and its partners. In this configuration, data will be encrypted in transit and at rest, with limited access to the source, only for maintenance and operations where the qualified engineers can oversee IT upgrades and monitor security of the data at rest.
In the data lakehouse, all information, documents, PII, and financial records will be processed and categorized into separate tables based on their use-case. Some records may require Primary Key (PK) and Foreign Key (FK) lookups that relate the information, including references to metadata of documents for similar functionality. Through the simplification of this PaaS and highly secure storage system with internal-only network endpoints, the data entering the system will be safeguarded in the system. Information that is required to be viewed will be limited based on RBAC/ABAC, and if required in scenarios where partial data is needed, masked for obfuscation.
The security features provided by Microsoft Azure as a PaaS will provide the confidentiality required to house this data from the onset. In the addition of Microsoft Defender and Microsoft Purview, these technologies themselves can be monitored at the endpoints where the data is being accessed, from devices, to in transit exchanges, and monitored for misuse. The Azure security framework added onto the PaaS solutions provides the integrity layer, and auditing capabilities, without direct access to the data being interchanged. In an accessibility spectrum, the Microsoft SLA provides an uptime range of 99.99% to 95%, in most scenarios almost nearly always up, baring extraneous circumstances.
Scaling the lakehouse
As the FinTech lakehouse scales, adding or removing services, the low-code configurations in Microsoft Power Pages and Microsoft Lakeflow can be reconfigured quickly and easily to meet the changing needs of the FinTech team and account holders. The benefit of the PaaS configuration allows for a continuous delivery of configurations with a secure backing of technologies, that are also flexible to change. The skills required are technical and functional but will not outweigh the benefit of upgrading the platform, as Microsoft offers a series of learning tools and certifications, in addition to these platforms being highly utilized, industry wide.
In a maintenance fashion, the FinTech provider will need to have engineers that are able to oversee the automated upgrades and regression test the application components, work through database version changes, and oversee the security monitoring. The maintenance team should set up error alerts for these scenarios in Azure Monitor and focus on addressing the larger segments down to the lower rates to keep the system running at capacity.
In most software configurations, data will fluctuate, including the formats and values in which the information is persisted. The fabric of this connected data will need to be audited and reconnected in the data lakehouse configuration, when these situations arise.
Engineering resilience, security, and future-ready software architecture. Follow on X, GitHub, and LinkedIn, and visit https://joealongi.dev/.
Engineering data lakehouse infrastructure for Financial Technology (FinTech)
In the design and planning of a data management system with flexible storage capabilities, it is important to reference modern…