We Need a New Study

or, Expedience vs. Accuracy

I’m at Open Source Bridge this week — one of my favorite conferences — and less then 2 weeks age I was at AdaCamp SF, which was also great. Both events involve a lot of people I respect who are dedicated to increasing diversity in the contributor pools of open source projects, which I love. Many of them keep referencing a stat that I think is outdated and mostly irrelevant, which I don’t love. Yes, I’m talking about FLOSS-POLS. That 1.5% number (percentage of FLOSS contributors who are women) is based on research that doesn’t represent the open source community today, so whether the current number is closer to 1.5 or 10.5 (and yes, I would guess that overall, especially in the developer segments, it’s probably still on the lower end) doesn’t matter; we shouldn’t be using a number that is outdated when making a case for why change is needed.

The FLOSS-POLS study was conducted from 2004-2006 over a 24-month period. The survey execution was completed within the first year, followed by interviews and analysis leading up to publication that took more than a year. So we are now talking about results based on a survey that is almost ten years old. Things change in a decade. Would we reference stats on mobile browser usage from 2004 to frame a discussion on why responsive design is important? Only in that 2004 would be a point on a graph showing the change in mobile browser usage stats over time. We don’t have that when it comes to assessing the percentage of women in open source projects because we need a new study.

“But, but —” you may sputter, “everyone knows that the mobile browser stats from 2004 would be irrelevant because the iPhone didn’t come out until 2007 and that was a game-changer. We haven’t had a game changer yet when it comes to women in open tech and culture.” True, that example may have been selected as an outlier where change happens so rapidly that stats from year to year are dramatically different. The number of women showing up on project leadership teams isn’t growing by those same leaps and bounds, but things *are* changing, and without tracking those changes, how can we really speak intelligently about the state of things? Looking at the numbers over recent years in specific programs like OPW, it seems obvious to me that we are in a pivotal time, and we should be collecting data like our lives depend on it so that we can look back at this time with accuracy, and so that future generations can learn from it.

Another problem I have with the FLOSS-POLS study is the methodology as outlined in Deliverable D 16: “Gender: Integrated Report of Findings”

This report primarily relies on anthropological research carried out amongst F/LOSS participants in France and other parts of Europe in 2004 and 2005

Yes, I know there are a lot of open source project contributors in France and Europe. Yes, I know Americans are often ethnocentric and think we rule the world (sorry about that, by the way, many of us know we can really suck and are embarrassed by that). Yes, I know that was the participant pool they chose for a specific reason:

The FLOSSPOLS project aims to support Europe-wide goals, and clearly needs to be conducted at a European level.

from Project Outline

To be fair, the report states that while the offline research was mostly in France:

parts of it were also conducted in other European countries (England, Wales, Germany and the Netherlands) as well as other countries (North America and India)

That’s all well and good, but this study gets referenced as if it’s the Bible of open source gender stats, and it just isn’t, given the geographic focus when many FLOSS projects are decidedly more distributed than their coverage. Also, note: “North America” isn’t a country.

Another issue is numbers, specifically this number: 1541 participants. That number just seems laughable to me. Hell, WordPress and Drupal alone could get more contributor survey responses than that — especially if we extend our definition of contribution to include more than just core development — and we’re just a couple of CMS projects that started in 2003 and 2001, respectively. Oh, look, two really big FLOSS communities that were barely off the ground in 2004. Here’s where that mobile browser comparison starts to not look like such an outlier.

In 2004, WordPress had a handful of contributors. In 2013, WordPress has hundreds of core contributors with each release, and the project as a whole has thousands more when you roll in the developers of plugins and themes at wordpress.org, the mobile apps, the docs and support teams, the event organizers/speakers/volunteers, the teachers, the translators, and all the other people who contribute to WordPress. So I take back what I said before about WordPress and Drupal together being able to get 1541 contributor survey responses… I think WordPress could get that many all on its own.

So, yeah. I don’t think this study is up to date, or particularly relevant to discussing the current state of things. Relevant to discussing trends? Yes! It’s now historical data. Historical data is the foundation of analyzing progress. But we need current data to know what the state of things really is.

I have a lot of other issues with the FLOSS-POLS study being taken as current-day truth, but I’m thinking that a) you don’t want to spend another hour reading this and I don’t want to spend another hour writing it, and b) you get my gist.

So how do we get new information to plot on the timeline of progress?

We need a new study.

Before moving onto what I think we should do about this, I’d like to take a moment to address a point raised in a session the other day by Kronda Adair in her session Expanding Your Empathy, and that we talked about a bit afterward. She was recounting how she’d tweeted that women still earn .50 on the dollar compared to men, and how someone replied asking what her source for that stat was, that according to CAP stats it was more like 96.7 cents on the dollar. Here’s the set of tweets from her slide:
screenshot of tweetsIn the session, Kronda posited that the reply was derailing the conversation, that the important nugget in her original tweet was that there was inequality, and that the respondent shouldn’t try to cloud that message by focusing on specific numbers. She also (rightly) pointed out that while .50 may not be the average in our industry anymore, that some women somewhere were earning at that ratio, some were earning more, and some were earning less, so unless you were going to be very specific about the group being referenced, saying that pay is still unequal was still an important message. I do agree with that. I disagree that the numbers don’t matter.

When you use numbers to make a point, they just have to be accurate, or you’re setting yourself up for your entire statement to be distrusted. If I can’t trust the numbers you’re quoting, can I trust your overarching statement? Oh, you weren’t quoting numbers, you were just repeating something you’d heard without researching it to verify? Okay, but now everything else you’re saying will be judged by that oversight. Just saying, “Women still make less than men 50 years after the equal pay act,” would have made the point. Not as dramatically, but that’s the power of stats, and that’s why it’s important they be accurate.

Just as I keep hearing people talking about how they would like to stop being distracted at work by having to discuss the issue of being a woman in technology and just get to focus on the actual technology, inaccuracy in talking about these issues is a distraction that pulls focus from the central problem, that there is imbalance and bias and discrimination and all those things that we want to correct moving forward. So let’s remember that what we say (not what we think) is what people hear and what they will use to judge our veracity, and try to be accurate in the words — and especially the numbers — we use.

Note: I loved Kronda’s talk and thought it was one of the best so far. Would love to see her give it at WCSF or the community summit if/when we do one again. I also already shared my thoughts on the importance of accuracy with her directly. I hate it when people deconstruct what someone else said without talking to the person directly.

Now, back to FLOSS-POLS and how I think we need a new study.

I’ve been thinking about this for a long time.  The draft post from when I first started trying to sum up my thoughts on this is from November. Much of the following is taken from that draft.

If we’re going to try to grow the numbers of women in open source projects and want to know our level of success/failure in that regard, we need to have a baseline number against which we can measure. We need good stats. We need reliable data. And we can’t count on a university funding a study once a decade. We need to take responsibility for measuring our own progress.

“Wait,” you’re thinking. “Isn’t that what the Ada Initiative is there for now? Won’t they take care of all that stuff for us?” Dudes — and I say that in an entirely non-gender-specific way — be realistic. Look at the Ada Initiative mission, look at all the projects they have going, look at the fact that it’s two passionate people trying to raise enough money to do awesome things, and think about what it takes to run rigorous academic studies, conduct data analysis, and to do it every year.

The Ada Initiative’s opening action was in fact a survey, the Ada Initiative Census of open technology and culture in March 2011. They surveyed a bit fewer than 3000 people about attitudes and their experience within their open source communities. The oft-cited FLOSS-POLS study surveyed just over 1500 (mostly in Europe). But we all know that if you tallied the contributor lists from all the open source projects, or even just the ones that, say, have participated in Google Summer of Code, we would have a community of many more thousands of people, probably tens of thousands of people, especially once we loop in the project contributors who focus on support forums, design, documentation, QA, etc. So how can we get the most useful data?

I dunno.

I say that semi-jokingly, but the truth is, we should all be asking what data is needed, and what the best way to acquire it is. Many people don’t want to ask contributors to identify themselves by gender (or race, or age, or sexuality, or education level, or socioeconomic status, or any of the other demographic slices in which one group dominates others) because we think it’s the work that matters, not the profile. And that’s true. But if we want to increase diversity — and pretty much everyone agrees that we ought to — then the profiles clearly *do* matter, or we wouldn’t go on about how we need to work for change. You see the conundrum?

My proposal: We form a diversity coalition of F/LOSS projects, with a rep from any open source project that is willing to be involved (hopefully, most of them). We consult with the Ada Initiative, and we consult with some lovely academic researchers who love open source (or who just love one or more of our projects, whatever) who would be willing to put in some time to help formulate a repeatable study plan and ongoing data collection mechanism. We do whatever we need to do to get contributors to our own projects to pitch in and agree to an annual diversity census and/or private, opt-in demographic  information being stored with their contributor profiles. We round up appropriate  researchers (volunteer? paid? either would work) to collect and analyze the data. We release the results to the F/LOSS community each year, so that each project has an accurate baseline against which to measure progress in their own project (if they are so inclined) and in relation to other projects.

This is kind of a giant proposal, I know. It would require working together with a lot of people, and it would require a lot of work in general. I’m basically at the point where I’ve stepped back from core UX to focus on increasing diversity and participation in the WordPress community, so I’m already prepared to say yes to this on behalf of WordPress (unless that statement gets me fired, in which case I’m prepared to say yes on behalf of just me). The question is, do other projects want this information as badly as I do?

Tangent: And could we pool some money to buy floss.org and make it the home of a F/LOSS coalition in general? We should be collaborating on more than diversity. Just think of how awesome it would be if we collaborated on improving the tools we all use, contributing to upstream projects, and ways of making our projects better? It could be like an online cross between OS Bridge and CLS all year long!

WordPress was founded almost 10 years ago when Mike Little agreed with Matt Mullenweg about a need for something, and said so in a comment on a blog post. Could this post be the start of a coalition of open source projects? That would be so ridiculously great I don’t even have a word for it. If anyone from another project is interest in talking more about this idea, leave a comment and I’ll follow up with you. If you’re at Open Source Bridge this week, let’s talk about it in an unconference session on Friday. Maybe we can make this happen!

Footnote: I would like to end this ardent cry for better stats with something my wonderful friend Andrea Middleton wrote to me a very long time ago when trying explaining to me how she, a poet, had fallen in love with a statistician. It sums up most of my feelings about stats perfectly, since I care more about people and interactions than I do about numbers. She said, “Statistics are like poetry, beautiful but useless.”