It is the year 2020. The Governor of the Bank of England is informed that a major U.S. investment bank, Garland Brothers, is collapsing.

He reaches for his iPad and clicks on an app called Big Brother: “Show me the credit exposure of the top five banks in the United Kingdom to Garland Brothers.” The iPad responds instantly with a bar chart showing the exposure of each bank in billions of dollars.

“Break it down by currency and legal entity.” Instantly, the chart updates with the additional detail. “Hmm. OK, I’d like you to run a scenario for me. Let’s assume the worst and write down all these exposures to zero. Now let’s overlay our adverse market scenario on their trading portfolios.” He studies a chart of the depleted Tier 1 ratios of the five banks. He mumbles to himself, “OK, I think we can get through this.”

During the last financial crisis, many financial institutions and regulators were found to be flying blind, with little idea of the size or even the real location of their risk exposures. And even now many financial institutions still lack reliable and comprehensive data about the risks they face. So the preceding story may seem to belong to science fiction.

However, advances in big data technology mean that this vision is within the realm of possibility. (See Exhibit 1.) Much work remains in order to get there. But, in our view, all the obstacles are surmountable and banks should prepare: Big Brother will soon be watching you.

Source: 2014 Oliver Wyman Risk Journal

Storage Cost No Longer an Obstacle

Big data experts like to boast by telling you how many filing cabinets worth of paper can now be stored electronically on their mega servers. For example, if you covered the entire floor space of One World Trade Center tower twice over with filing cabinets of double-sided paper, it still wouldn’t be enough data to fill a 30 terabyte database. The more impressive statistic is that 30 terabytes of storage now costs around $800 a year.

Of course, the world’s financial institutions probably produce enough data to populate 100 World Trade Center towers every hour. Yet the cost of storing all of this data would still be only about $150 million a year. This is a drop in the ocean of the annual IT spending of a large bank. In short, if you want to take the brute force approach of literally “storing everything,” you now can.

In-Memory Data

Another major development in big data is in-memory data storage. This is the data that an application has at its fingertips without the need to search through databases. This type of data supports superfast analytics and live queries: that is, it supports the type of real‑time response illustrated in our opening story.

While this kind of data storage is more expensive than traditional database storage, most data need not be stored in‑memory. Clever ways of aggregating and compressing data without losing too much vital information can give users access to everything they need in real time.

All the obstacles are surmountable and banks should prepare: Big Brother will soon be watching you.

Access All Areas

The obstacles to accessing the data that regulators might need are now political rather than technical. Client confidentiality and legal restrictions to sharing data are likely to present bigger obstacles to accessing and storing the data than any technical problems. However, regulators are in a privileged position when it comes to overcoming these obstacles, especially following the financial crisis. Provided the data is handled with the appropriate level of security, regulators should be able to gain access to all areas.

What Are the Possibilities for Regulators?

Microprudential Regulation

We opened with an example of using big data analytics in microprudential regulation. Regulators should now be able to get all the information they need to understand what is going on inside individual banks. The key challenge here is in understanding which information is important.

Information overload must be avoided. Risk reports can contain so much information that it is difficult to know what to focus on. This is a problem that is familiar to the industry and one that it is addressing. The typical solution is to start with high-level aggregate reports for senior stakeholders and provide more detailed and granular reports to junior stakeholders.

In the big data vision of our opening story, however, senior executives can also drill into the detail if they want to—for example, by running custom queries on the fly. Or perhaps big data technology might allow the user to double-click on a cell in a table to see the information that underlies it. This is surely a regulator or senior executive’s dream.

Macroprudential Regulation

In the aftermath of the financial crisis, regulators decided to increase their focus on macroprudential regulation, which is designed to address threats to the broader financial system, rather than threats faced by individual institutions. One idea, for example, is for regulators to be on the lookout for bubbles in the real estate market which, if not kept in check, might bring down the entire financial system when they burst.

Big data can serve well the purposes of macroprudential regulation. To gain a complete picture, macroprudential monitoring requires access to information spread across the entire system. For example, if a bank launches a new product targeting low credit-quality borrowers, regulators might want to know how the bank’s competitors are reacting. If they are joining in with this new easy credit boom, this could create a systemic threat. The ability to see how banks are behaving in the aggregate would allow regulators to spot systemic risks.

Conduct Risk

Perhaps the most intrusive regulatory application of big data analytics would be in the area of conduct risk. This new branch of regulation aims to control the way financial institutions interact with their customers. Is a bank treating its customers fairly? Is an insurer being transparent to a customer about the risks they are taking when they buy a financial product?

Most financial institutions already record telephone conversations, and regulators often request access to e-mail trails in the event of a suspected regulatory breach. But regulators might begin to take a more proactive approach, seeking data before they have reason to suspect anything has gone wrong.

The more data regulators are able to get from across the entire industry, the better they can establish “normal” patterns of behavior. This can then help them to spot unusual activity, such as a rogue trader hiding transactions or patterns of communication that suggest insider trading or the manipulation of market benchmarks. By using big data applications to detect unusual patterns early, they may be able to prevent breaches rather than responding to them after they have occurred.

In our view, financial regulation will inevitably be shaped by the big data revolution. Financial institutions should prepare now for its ramifications.

This piece  first appeared in the 2014 edition of the Oliver Wyman Risk Journal.