People to Watch 2017
Senior Director of Engineering
When Joe Witt created what would become Apache NiFi back in 2006, few people outside of America’s spy agencies understood how important managing and manipulating large streams data would become. After the NSA donated NiFi to the open source community in 2014, Joe would continue to develop NiFi, which now forms the basis of the Hortonworks Data Flow product.
Datanami: Hi Joe. Congratulations on being named a Datanami Person to Watch in 2017! To get things started we wanted to ask what role the NSA had in shaping what NiFi would ultimately become.
Joe Witt: The NSA years for what became NiFi were very important on a couple of levels. The most obvious is the incredible scale, complexity, security, and compliance requirements that it imposes and how that impacts which features and ‘ilities’ were built and in what order. Things like organizational and mission agility were vital. But another interesting aspect is the incredibly talented software developers and operations personnel that exist there who contributed code, design ideas, and provided constant feedback on what needed to be better. Frankly, that trained many of us for what open source life would be like.
Lastly, we’re very fortunate that NiFi was created in an organization that has such an impressive and talented Technology Transfer department. The NSA technology transfer program does a lot of great work getting the great work of the agency available to others.
Datanami: Moving forward, are there specific data flow challenges that hold a special interest to you?
I think right now people are still really enamored with ‘scale’ in terms of how fast or how much data they can have and if they end up flowing a whole lot more than they did before or storing a lot more then they see this as a measure of success. But, what I am far more interested in is how to effectively on-board new flows faster at enterprise wide scale and how to measure effectiveness of existing and new flows and systems involved in a truly end-to-end sense where at one end we mean the most far out edges of an enterprise and on the other we mean every consuming system be that in the datacenter, the cloud, or in delivering data to a business partner. So for me the most exciting aspects of the flow game are how to provide absolutely enterprise wide command and control and how to do what is called ‘data provenance.’
Datanami: What trends do you see as being most important for big data as we look to 2017 and beyond?
I hinted at this a bit in the previous question but I think the most important gear shift that is occurring and must occur is moving beyond the excitement of much of the ‘vanity metrics’ that are talked about in big data and into measures of effectiveness and architectural agility. What development and operations teams want is to be able to get projects done faster and what top-level executives want is to be able to measure the effectiveness of their technology investments. Increasingly we see folks demanding and needing the ability to track every piece of data and where it comes from and goes to and how it impacts the bottom line. Data Provenance, the ability to understand the origin and attribution of data, has been studied for a long time in the academic arena and now it is heading toward being a key requirement of all systems. The concept of data provenance drives directly to the heart of solving critical enterprise wide debugging and operations challenges, proving compliance, and measuring IT ROI. I think this is awesome and will be doing all I can to help push that along.
Datanami: Outside of the professional sphere, are there any fun facts about you that your colleagues may be surprised to learn?
I married the girl I fell in love with in fifth grade. Sure it took her about eight years to let me take her on our first date, but those years taught me a lot about perseverance and vision both of which are traits that have directly impacted my professional life in very positive ways. Surround yourself with great people and work together toward a beautiful vision and wonderful things happen.
More about Joe Witt:
Joe Witt, Senior Director of Engineering at Hortonworks, Inc. Joe created a project at the National Security Agency in 2006 which was later released and is now known as Apache NiFi. After the NSA, he went on to found Onyara to commercialize an offering around Apache NiFi, which was acquired by Hortonworks, Inc. Joe now works at Hortonworks with an elite engineering team building out Hortonworks Dataflow and actively working with the Apache NiFi community. Joe is also an Apache Software Foundation member and advisor to DataTribe.