Market Regulators Tap Into Big Data
Big data has already become a competitive differentiator in the private sector. But government regulators in the US and the UK are showing they can apply big data lessons in the public realm too, and have launched big data programs of their own.
The Security and Exchange Commission (SEC) went live in early 2013 with Market Information Data Analytics System (MIDAS), a computer system that tracks the complete order books for all the equities traded on each of the 13 national exchanges. Every day, MIDAS ingests more than 400 GB of data from the exchange tickers, including posted orders and quotes; modifications and cancellation of orders; trade execution against orders; and off-exchange trade executions.
MIDAS time-stamps market data at the micro-second level, which allows the SEC to go back and analyze the behavior of equities over periods ranging from six months to a year. This fine-grained analysis gives the SEC a powerful tool for the purpose of understanding mini-flash crashes, reconstructing market events, and developing a better understanding of long-term trends, the SEC says on its website.
MIDAS gets its data from the consolidated tapes from the national exchanges, as well as from the separate proprietary feeds made individually available by each equity exchange, the SEC says. “These individual exchange feeds are typically used by only the most sophisticated of market participants such as market makers and high-frequency traders,” the SEC says. “Most institutional investors, retail investors, and academics, generally do not consume this data–it is extremely voluminous, challenging to process correctly, and requires specialized data expertise.”
The UNIX-based system, which reportedly costs the SEC $2.5 million per year to operate, can analyze 100 billion records at any given time, the SEC says. And with an archive of market data that’s 1PB in size, MIDAS gives the SEC tremendous power to see how current market events shape up against the historical record.
Meanwhile, across the pond in the UK, the government’s Financial Conduct Authority (FCA) is ramping up a big data initiative of its own. But whereas the SEC is using big data analytics to tease out malfeasance that may lie hidden in the market data, the FCA is stepping in to insure that financial services firms aren’t abusing their newfound big data powers.
Specifically, the FCA wants to know analyze how insurance firms are using big data. In its annual business plan, the FCA says:
“We will conduct a market study to investigate how insurance firms use Big Data, such as web analytics and behavioural data tools [including the increasing use of social media] as well as other unconventional data sources. We will identify potential risks and benefits for consumers, including whether the use of Big Data creates barriers to access products or services. We will also examine the regulatory regime to ensure that it does not unduly constrain beneficial innovation in this area.”
In the insurance business, big data has been hailed as a boon by some, and a boondoggle by others. Supporters tout big data’s ability to provide fine-grained empirical data from the field. For example, in the car insurance business, companies like Progressive are able to base insurance premiums on the exact driving characteristics of individual drivers, as opposed to using actuarial tables.
But big data has its dark side too. In the health insurance business, some insurers caught hot water for using big data to create profiles of their members, including whether or not they were obese.