To What Extent are Big Data Technologies Influencing the Financial Sector?
Close-up Photo of Monitor. Source: Pexels.
What is “Big Data”? Essentially, it is a data set that is too complex to be interpreted by normal data-processing software and is currently transforming the finance sector with enhanced decision-making and improved operational efficiency. Moreover, as it can easily process data and utilize numerous advanced ML algorithms, it is able to provide an insight into consumer behaviour as well. Regarding regulation, there are numerous bodies that use big data to detect criminal activities, monitor real-time risks and, of course, ensure transparency. However, it is imperative to recognise that there are challenges such as data privacy, biases within algorithms, and lags behind technological evolutions and all will be discussed further in the article.
Having defined big data, a crucial concept must be discussed and that is the 5 Vs: volume, velocity, variety, veracity and value. Volume is the amount of data that is generated per second; velocity being how fast the data is being generated and processed; variety refers to different types of data generated; veracity looks at the integrity and quality of the data; and the value looks at what can be derived from the data. These five dimensions are what makes up the big data and represent a critical aspect of data management and analysis. The specific applications of big data in the context of finance are fraud detection, algorithmic trading, scenario testing as well as risk modelling. Regulatory oversight is vital in finance as it allows for consumer protection and market stability, which in turn reduces damage to investors. The traditional approach of regulation is often time intensive, but with big data, a real-time and more predictive approach is taken. Lastly, policy-making in the context of the traditional era usually works from the top, with executives, down to the level of workers. The traditional era refers to how institutions were run before democratic management styles had become common. But, with big data we see this to be a more evidence-based approach with interactions amongst numerous regulators and greater set of discussions with data in changing policy.
How can big data affect regulatory oversight? We know how the traditional approaches are manually intensive, but with real-time monitoring and predictive analytics we are able to detect signs of fraud in earlier stages, as well as insider trading and other practices that are not mentioned in the Security Exchange Commission Ethics Document. Predictive analysis also makes use of machine-learning algorithms that analyse different types of training data sets as an attempt to ‘predict’ future outcomes with optimisers like Adam, which help to minimise error by adjusting learning rates, being used. They are able to also identify any anomalous values in the market. For example, the SEC had used a MIDAS system to monitor irregular trades. It collects data from proprietary feeds made available by each equity exchange as well as on options and futures. This was a much more data-driven approach and allowed for the efficient analysis of book data. Regulatory oversight also encompasses risk assessment, which is slightly different in the data that it analyses, such as news sentiment and social media. This is the type of data that is used in scenario analysis and stress testing, as previously discussed. For example, there was a 2025 Bank Capital Stress test which tested how strong the UK banking system is during global recessions and while experiencing high interest rates. This shows the impact big data can have on banking however there are downsides like the emphasis on quantitative figures and a lack of knowledge with regards to the sourcing of data. Another important consideration is RegTech, in the context of compliance automation. Compliance automation means the use of technology to constantly help organizations adhere to regulatory requirements set by a body. It can reduce operational costs in a company however concerns must be shown as there are risks in cybersecurity/data leaks as well as the underlying factor of ethics when an automated decision is taken.
Regarding policy application, we often observe that policies are data-driven rather than sentiment-driven and the same trend would apply here with thousands of policy-makers using big data for financial stability, particularly liquidity and capital adequacy regulation which ensures companies have enough resources to meet their obligations without a loss to profit. This shows an evidence-driven approach however it is imperative to expect challenges like data fixation to arise. Also, advanced machine learning tools, such as Gated Recurrent Units, have improved the accuracy of liquidity risk predictions, showing the benefits of big data driving changes in policy making. Algorithmic biases will always be an issue as it can affect the fairness of how data is interpreted and where the data is sourced. This is why there should be a certain level of power that AI should get because, as it gets powerful enough to determine policy, there should be neutral algorithms that do not amplify existing patterns of discrimination. Another problem concerns regulatory lag, as a result of which policy and legislation is delivered at a slower rate than the rate of technological innovation, resulting in a delay between the emergence of financial innovations and the establishment of certain regulatory frameworks that would lead to a systemic risk. Furthermore, big data is key to inter-organisational data sharing, such as between the International Monetary Fund and the Basel Committee, which allows greater global policy harmonisation. An example would be the data governance which promotes cross-border data flows and regulatory coordination. Yet, it is important to consider the trade-off between such benefits and broader national security and individual liberty concerns.
What are the ethical considerations of such policy change? Data privacy and security are always considered when data is used, and it is essential to protect consumer information that is held in large, complex data sets. There are risks such as cyber-attacks and data leaks that we have seen in today’s world. This would introduce legal issues as consumer data is protected in the GDPR. This is a policy that decides how data should processed and how to maintain confidentiality/integrity of the data. There are other aspects that should be considered, such as algorithmic transparency. For instance, black-box AI, where the user is unable to see the internal logic of the AI and therefore would be lacking ethical oversight as well as introduction of new concerns. There will always be public distrust which would cause regulations lags, therefore an innovation delay. To a certain extent, one must note that human oversight is needed and that there should not be an overdependence on technology alone. In a financial context, it may reduce stability in the markets as people would use the same models and that would cause an amplification of shocks in the market.
One pertinent case study is that of FCA Project Innovate. This was a project launched with the goal of supporting innovation in the financial services offered by the industry. It attempted to help to create policies in the interest of the consumers and aid them in avoiding red tape. It made use of the Sandbox that was a testing environment where financial firms could access complex datasets and test products; it would simulate real-word data like demographics and ESG metrics. This is a way that doesn’t blatantly violate privacy and ethics with more of an experimental oversight alongside data stewardship. This synthetic data framework is what makes it unique and complies with the GDPR and also gives foresight in the risks that big-data poses as well as ensuring that there are no discriminatory biases in the algorithms being used. Research indicates that regulators can also reduce information asymmetry, which is when one party has more information than another, and improves AI accountability reducing exploitation.
In conclusion, big data is quickly altering the state of regulatory oversight and policy-making. While considering privacy and ethical concerns, we should still progress away from traditional, manually intensive processing to more efficient, detailed and data-driven approaches. Challenges are yet to be overcome, with algorithmic bias and regulatory lag still prominent, but the invention of a sandbox will be able to tackle this as shown by FCA Project Innovate. It is imperative to consider that big data has vast amounts of potential to transform the financial industry, but that this should be done in a way that it is transparent, in the interest of everyone and able to retain financial stability. In the future, AI will become an even greater force in policy-making but should always be tempered by human oversight. Ultimately, big data stands as a catalyst capable of transforming financial regulation and policy-making into a more evidence-based process whilst also reminding us that in striving for innovation, we should remain aligned with ethics.
