Women in Big Data: Does Gender Matter?

(Brian A Jackson/Shutterstock)

I’ve recently taken on a position as an executive in a technical role at a big data startup headquartered in the heart of Silicon Valley.  I see the growth of next-gen big data platforms as the most exciting and disruptive story of the 21st century.  The entire computational landscape is going through a profound change; legacy technologies are in decline and a new vanguard of distributed data platforms are emerging as the next-gen infrastructure for enterprise and research applications.  It’s a great time to be a technologist.

But it’s hard not to notice that in the midst of all of this new growth and innovation, women have played a decidedly small role.  This isn’t a new phenomenon; the gender gap in technology companies and in computer science degree programs has been well documented.  But since a lack of gender diversity is a headwind for any industry,[1] [2] it bears further exploration into the root causes.

Are Women Just Bad at Science and Math?

Why is it that women aren’t as numerous as men in technology and computer science?  Ten years ago, Larry Summers famously questioned whether the relatively few women in top positions in science might be attributed to “issues of intrinsic aptitude.”  Despite the public outcry that Mr. Summers understated the role of discrimination and socialization in the process of attrition, it’s hard to fault someone who merely asks the question from a perspective of curiosity.  I’ve even asked myself whether it’s possible that there is an innate difference in the abilities of women and men to reason analytically.  Do we have evidence that could prove or disprove this theory?

Personally, I have come to the conclusion that women are equally capable as men in science, technology, and math.  I believe this largely because it resonates with my own personal experiences collaborating with women and men in the field.  But there is also scientific evidence that, in my mind, makes it clear that proficiency in science and math is a result of interest and motivation — not innate talent.

An OECD study found the math aptitude of Icelandic girls exceed that of Icelandic boys (Image source: HuffPost)

For example, in 2005 the Organization of Economic Cooperation and Development administered a standardized test in 41 countries and found that among 15-year-old Icelandic students, boys trailed far behind girls in math aptitude.  A subsequent analysis by Icelandic researchers found that the discrepancy owed to differences in motivation and career ambition; boys planned to become fisherman and saw math as irrelevant whereas girls knew that a non-fishing career would require an education and admittance to college.  Similar studies have identified other non-Western cultures where there are equal numbers of female and male computer science students, such as in Malaysia.[3]  In each study, the researchers explain these numbers by pointing to reasons of motivation and how the discipline is perceived by students of either gender.

The Motivation Gap

If proficiency in science and math is correlated with motivation, then clearly many women are not, by and large, motivated to go into such fields.  There must be a psychological barrier that causes women to, in effect, opt out of careers in science and math.

It starts in school.  A number of compelling studies by researchers at Carnegie Mellon University[4] and others describe the psychological barriers that cause female students to opt out of computer science degrees.  In short, the rationale for opting out of computer science as a chosen major owes to perceptions that: 1) it’s too hard, 2) it’s boring, or 3) I won’t fit in with the other CS students (who are predominantly male and sometimes can be a little antisocial).

Perceiving computer science as an unpleasant discipline and technology as an uninteresting career path in turn causes women to suffer a lack of motivation in their math and computer science courses starting in secondary school and college.  In fact—and this goes contrary to the popular wisdom surrounding the story of Silicon Valley’s gender gap–these perceptual barriers to entry are much more important in explaining the attrition of women from science and technology careers than gender discrimination, unconscious bias among hiring managers, or a proverbial “glass ceiling”.[5]

Is This Really a Problem?

If we concede that women are fleeing from the tech industry of their own volition and not so much because of gender discrimination, is it such a bad thing that women are underrepresented in tech and big data?  Well, yes, it’s a problem for big data and other technology companies that need to hire exceptional talents when nearly 50% of the population is choosing to steer clear of our entire industry.  And it’s worse for the country because the tech industry happens to be a key driver of the growth of US GDP.

A Path Forward

The good news is that perceptions can change.  In my previous role as a professor of computer science, I worked with my colleagues to change perceptions around our field in order to increase the ratio of women enrolling in our CS courses with tremendous success (with women now comprising more than 50% of the pre-registered students in every single CS class for the upcoming fall 2016 semester).

The Grace Hopper Celebration of Women in Computing (ghc.anitaborg.org) is designed to bring the research and career interests of women in computing to the forefront.

The changes we implemented were straightforward.  Namely, we streamlined the students into different classes so true beginners would not be intimidated by those with more experience; we gave students options for assignments with choices ranging from applications from computational biology to sports analytics so the material was more engaging; and we made an institutional commitment to take students at all levels to the Grace Hopper Celebration of Women in Computing in order to provide a networking opportunity to connect with other women in the field.  Each of these three strategies addressed one of each of the problematic perceptions of computer science and in total they had a dramatic impact in just two years.

So I’m optimistic.  I fully expect the ratio of women entering big data and other tech subfields to grow in the next generation.  And in the meantime, I can offer young women technologists a few key tips in order to thrive in this fast-paced and exciting field.

Keys to success for women in big data:

• Work hard. Big data is an incredibly competitive field for both companies and employees.  Being smart is a prerequisite for entry into a big data company—smarts are just table stakes.  The way to differentiate yourself is through hard work, consistency, and discipline.  (This is true for men too, by the way.)
• Speak up. Your opinion counts.  The value in having a diverse team is that you come to a problem with different viewpoints, ensuring you examine the issue from all angles.  A good manager wants to work with people who know things he or she doesn’t already know – not people who just mimic back what they heard from their peers.
• Be yourself. Don’t put too much time or energy into trying to “be one of the guys” in order to fit in.  It’s a waste, and if the effort isn’t sincere, you will come off as phony.
• Look for mentors and allies. Everyone who has done anything important can point to the set of people who helped them learn the ropes and then grow into their fullest potential.  Align yourself with people who invest time and energy into you, and keep in mind that these people may very well be men.

In time, as perceptions about computer science change, we will start to see the profile of the big data community shift.  When the next generation sees that industry participants need not all be male uber nerds, but can instead represent a broader segment of the population, the result will be a much more diverse group of industry entrants, including a healthy number of women.

About the author:  Crystal Valentine is VP of Technology Strategy at MapR Technologies with a background in big data research and practice.  Before joining MapR, she was a professor of computer science at Amherst College.  She is the author of various academic publications in the areas of algorithms, high-performance computing, and computational biology and holds a patent for Extreme Virtual Memory.  As a former consultant at Ab Initio Software working with Fortune 500 companies to design and implement high-throughput, mission-critical applications and as a tech expert consulting for equity investors focused on technology, she has developed significant business experience in the enterprise computing industry.  Dr. Valentine received her doctorate in Computer Science from Brown University and was a Fulbright Scholar to Italy.

[1] Lois Joy, Nancy M. Carter, Harvey M. Wagner, and Sriram Narayanan.  The Bottom Line: Corporate Performance and Women’s Representation on Boards.  Catalyst, 2007.

[2] McKinsey & Company, Women Matter series, 2007, 2008, 2010, 2011, 2012.

[3] Mazliza Othman and Rodziah Latih. 2006. Women in computer science: no shortage here!  Commun. ACM 49, March 2006 3: 111-114.

[4] Jane Margolis, Allan Fisher and Faye Miller.  The Anatomy of Interest: Women in Undergraduate Computer Science.  Women’s Studies Quarterly, Spring/Summer 2000: 104.

[5] Stephen J. Ceci, Donna K. Ginther, Shulamit Kahn, and Wendy M. Williams.  Women in Academic Science: A Changing Landscape.  Psychological Science in the Public Interest, December 2014 15: 75-141.