In today's data-driven world, larger organisations are increasingly turning to data platforms to solve the challenges of multiple integrations, siloed data and untapped potential to transform data into competitive advantage. 

Many larger organisations are already well underway or have data platforms in place. There are a number of products on the market, but some of the components used in a unified data platform include Kafka, Data Lake and Databricks.

Although it is becoming increasingly easier to set up, it is still expensive to build a data platform with subsequent customisations in the IT landscape. Data platforms typically serve as both a central integration platform and the basis for the company's analysis activities, including machine learning/AI-based models.

When companies build their data platforms, there are a wide range of compliance issues that need to be considered. This is central to the design, data utilisation and governance of the platform. The list of considerations is long. Here are a few selections;

• Business Intelligence often wants to retain as much data as possible as long as possible to strengthen current and future analysis models. This presents personal data challenges that must be considered from the start if, for example, 'infinite logs' are used. There may be a need to delete individual fields in transactions to ensure correct anonymisation, which will require detailed analyses both during implementation and when adding new data sources. 
• If Kafka is used then data is stored in 'topics' instead of tables. They function in much the same way as logs. For personal data, it must be considered whether old values (e.g. contacts) should be retained for as long as new values or whether old values should be deleted independently.
• When the company's market data is gathered in one place, there are competition law challenges that are uncovered, e.g. in relation to the company's pricing 
• Is the collected data and structure appropriate to fulfil ESG-related requirements or will there be a need to build new data structures
• How to manage the retention period both in terms of access to data and the final deletion of data, which must take into account all the different legal requirements as well as the company's own needs
• Some customers require their data to be kept completely separate from other customers' data for their own compliance purposes. How is this incorporated into the design of the data platform?
• How to ensure adequate processing of data subject requests, i.e. can the selected personal data be technically identified and deleted with the chosen solution
• How to ensure a legal basis for processing personal data when many applications use data from a wide range of tables with very different data sources. Do you really need to build a compliance onboarding when new applications are connected to the data platform?
• Have all the information security aspecs been considered allowing for the necessary compliance towards new legislation and customer requirements
• Does increased data sharing make it possible to monitor employees and others - intentionally or unintentionally?

The GDPR principle of Privacy by Design will have a significant impact and must be considered from the start. Especially in the development of governance, there will be a need to draw on compliance people who can assist with the work. This also applies when adding data to the platform.

The use of data platforms by businesses is here to stay, as they offer significant advantages in terms of data processing, storage and analysis. They are technically very complex, requiring in-depth compliance analyses to ensure the company's data crown jewels are handled correctly now and adapted as regulatory requirements change. Compliance functions should therefore get involved in these projects as early as possible. This includes ensuring that;

• Applicable standards in areas such as privacy management, competition law and contractual requirements are complied with and monitored
• You have the right to upload personal data before data is utilised
• The privacy policy covers and explains the processing performed on the data platform and the application that pulls data from the platform
• Data is classified correctly
• Roles and responsibilities for data are clearly defined and anchored
• Personal data is deleted based on deletion policies and ad hoc - both for master data and transactional data so the risk of re-identification is minimised
• There is an appropriate opt-out solution, if the data platform is used for tracking individuals 
• Access rights are tightly controlled ensuring that access to the large amounts of data on the platform is limited to what is necessary
• There are contracts with partners, data brokers, marketing agents and data processors that ensure personal data protection in the necessary global context
• If AI is built into the data platform, future regulatory requirements should be incorporated in a timely manner
• Controls are comprehensive and performed with the right frequency

If you want to know more about how compliance with data platforms can be realised, please contact us.