Follow Datanami:
September 17, 2015

Peering In On the Pols with Big Open Data

It’s no secret that our government is using big data technologies to track and monitor the activities of regular citizens, ostensibly to protect our freedom. But data analytics can also be used to keep track of what our government is doing—or at least what the politicians and other lawmakers are up to.

The advent of open data initiatives at all levels of government is giving citizens, companies, and other interested parties access to vast amounts of data generated by various agencies and governmental bodies. This tidal wave of open data will help people generate new insights in a variety of areas, from medical science to social demographic research.

Another area of research enabled by open government data is how the politicians themselves are behaving (at least while they’re on the clock). Political reporters, lobbyists, and other insiders have long been accustomed to getting the scoop on how the sausage-making took place, but now big data is helping to shine a light on the mysterious goings-on in the 50 state capitols and Washington D.C.

One firm at the apex of this movement is FiscalNote. The Washington D.C. company leverages big data analytic technologies and techniques to track all the legislators and the legislation they bring in all 50 state houses, as well as both houses of the United States Congress and the District of Columbia. What used to be an overwhelming job for small governmental affairs groups at corporations and non-profit organizations is made considerably easier by its Web-based service, called Prophecy.

John Zoshak worked in the government affairs shops of pharmaceutical companies before joining FiscalNote recently as its senior policy manager. fiscalnote_logo“Keeping on top of all 50 states is next to impossible,” Zoshak says. “Sure we have services that push legislation alerts and let us know if the bill is moving. But what is missing are the insights you can get from data about those things.”

That’s the hole that FiscalNote is filling for its clients. The Prophecy service starts with the basics: What new bills are proposed, what politicians are sponsoring and co-sponsoring them, what committees the bills will go through, and when (or if) it comes to the floor for a final vote.

FiscalNote expands upon this base by digging into the histories of politicians’ careers, their policy bent, and their political networks to provide additional context to the bill-making process. The service, which uses graph analytic technologies, natural language processing, and other machine learning algorithms, generates predictions and forecasts about the likelihood of specific bills successfully becoming law.  The company claims a 94 percent accuracy rating for its predictions.

“We have data going back a decade in each state that will tell us if the sponsor is good at getting bills passed,” Zoshak says.  “That’s a good use case right there–just verifying that the person who is pushing the bill is good at pushing bills–or the opposite, which is actually more common.”

These insights can be very useful to a large swath of companies and organizations, whose fortunes may rise or fall based on the laws that politicians pass, or prevent from passing. “It’s a broad use case,” Zoshak tells Datanami. “We work with everybody from tech companies to healthcare to education and healthcare non-profits. We kind of run the gamut, from Fortune 500 to tiny education non-profits.”

FiscalNote uses a Postgres database running on Amazon Web Services to keep track of and analyze legislation and legislators. There are 1.6 million pieces of federal legislation in its system, along with millions of other documents, such as summaries and comments, says Vlad Eidelman, the company’s vice president of research. The state repositories varies a bit, state by state.


FiscalNote tracks the effectiveness of every state and federal politician across multiple areas.

All told, FiscalNote has upwards of 50 million documents in its system. That may not sound like a lot, but when you consider that it’s largely unstructured text and that it has a direct influences the laws that every man, woman, and child in this country must follow, then the scope of the undertaking begins to show. “In terms of the content of the actual documents, it’s very heavy,” Eidelman says. “There’s a lot of data in there. It’s very influential.”

In addition to using NLP algorithms to parse the text of bills, the company uses a collection of rule-based approaches to further deduce meaning. This is especially important when trying to understand very dense documents written in legalese, where something as elementary as the placement of a comma or the use of a certain definitive in a sentence can have a big impact on the ultimate meaning. Instead of

It’s quite common for lawmakers to pass laws, and then leave the actual writing of the regulations to somebody else. FiscalNote also deploys its suite of algorithms against all the comments that various parties make as part of this regulation-writing process, and can give clients (it currently has more than 100 customers) insight into how they’re trending. Previously it just tracked federal regulations, but today it launched Atlantis, which tracks regulations at the state levels.

“The benefit of doing it with models based on machine learning is we can do it in real time,” Eidelman says. “As soon as the law comes out of the state, we predict what will happen at every stage and that’s update based on every action that is added on the bill. So we have a prediction as soon it’s introduced, then as soon as soon as it’s assigned to committee… Our whole motto is the more data we can provide you, the more decisions you can make.”politiciams

The algorithms are critical for helping government affairs shops make sense of what’s going, especially when politicians or voters behave anomalously. Every politician will know who they can count on to help them pass bills, both in their party and the opposing party. But those connections aren’t always obvious to those who don’t track the connections so closely.

“It’s a legislator social network,” Zoshak says. “It’s interesting to see the people who work in that space say ‘Oh yeah Bob and Linda work together all the time.’ Then that bares out through the algorithms. It’s cool because we don’t know that like they know it, but we know it through the data.”

The algorithms can also point to voting trends that may not readily appear to governmental affairs shops using basic tracking systems. “We try to graph everybody in ideology, from conservative to liberal, compared to everyone else in the chamber,” Eidelman says. “And sometimes you think of someone as very conservative or liberal. And then by their voting or sponsoring behavior, that actually might not be the case.”

We’re a nation of laws, but all too often, our visibility into the people and the process making the laws is not what it could or should be. There’s a reason why the 19th century American poet John Godfrey Saxe compared the lawmaking process to making sausages: It’s perhaps better not to see how they’re made.

FiscalNote’s platform provides better visibility into the process. “If you’re a lobbyist and you have all this backroom knowledge, it’s to your benefit to have an objective source of knowledge coming in and augment any anecdotal information you may have,” Eidelman says. “At the end of the day it’s not just about having more data–it’s about having data that’s actually useful and provides a useful signal to our users.”

Related Items:

Startup Launches Legislative Prediction Engine

How Data Analytics is Shifting Politics

Big Data Analytics Give Electoral Edge