Measuring progress

Use metrics to drive D&I priorities and accountability. Metrics can gauge success and create accountability for programs at your company — like a modified Rooney rule that mandates the inclusion of at least two candidates from underrepresented groups in the hiring pool. When shared responsibly, they can also collectively hold the industry accountable.

As a baseline, we recommend regularly reviewing demographic data and running employee engagement surveys. This includes data on different processes, like hiring, promotion, and managing attrition, cut by demographics, and comprehensive surveys periodically with lightweight pulse surveys more frequently. This quantitative and qualitative data is critical to understanding organizational health and areas for improvement. For example, it can help identify things like problematic dropoffs in recruiting pipelines, or key predictors for dissatisfaction and turnover, and correspondingly, good areas for intervention and improvement.

Measuring D&I will elevate it to the same status as other business priorities. This prioritization will make sure it’s approached with the same sort of rigor. For this purpose, at a minimum, metrics and survey findings should be transparent internally, to help companies track progress. Merely measuring and tracking performance isn’t enough, though; it has to be accompanied by a system for responding to results, and this system must be consistent and transparent.

Companies must review their data policies. Given today’s political climate, be thoughtful about whether to start or continue collecting or storing certain data. They could be used to discriminate against your employees (or clients or users or suppliers) should state or federal anti-discrimination laws change. You may wish to form a working group with the design, engineering, product, and other leads to discuss these concerns and decide on the best approach. If you decide to collect sensitive data, make sure your team and any third party provider have security and privacy baked in — including clean, immediate data deletion when needed.

Some data should be shared externally. Sharing data can help establish benchmarks and standards for diversity metrics, allowing companies to understand relative performance. Publishing data publicly also provides accountability for companies and their D&I initiatives. Companies may want to share their data for PR purposes or brand-building.

Avoid vanity metrics. The wrong metrics can present a misleading view of system health, and optimizing against them can produce distorted outcomes. Vanity metrics are dangerous, because they provide good optics but lack significance. For example, some companies include non-engineers, like administrative assistants, in their definitions of their engineering teams, artificially inflating their diversity numbers with groups that are usually more diverse. Internally, this practice can erode trust with employees who see a lack of diversity on their teams, and can delay identifying and addressing problems, while externally, it can lend a company artificial weight as an industry leader.

Design surveys and collection processes carefully. While surveys are invaluable for understanding sentiment, they only function well when they are thoughtfully designed and executed, and analysis is done correctly. Poorly designed surveys can not only produce incorrect results but, even worse, be offensive or damaging. Employee distrust of company-administered surveys, particularly around sensitive questions, can skew results due to misrepresentation or lack of response. Running surveys too often can create survey fatigue, lowering participation rates and similarly skewing the data.

Some companies may not track metrics or run surveys at all out of concern that they’re not instructive or actionable or that they may reflect unfavorably. For seed-stage companies in particular, the numbers may be too small to draw statistically significant conclusions, and it may not be possible to cut survey data by demographics because there simply aren’t enough people. The challenge is for these companies to recognize the value of diversity and inclusion early, and include measurement of it to assess progress and identify areas in need of improvement.

What are our recommendations

Use existing metrics definitions and surveys

Good survey design is an art and a science. There is a wealth of research and guidance on how to design and run surveys effectively, but in cases where it’s possible to reuse existing surveys or survey questions, we advise doing so.

Leverage existing metrics definitions and surveys, particularly on third-party survey platforms. They can usually aggregate results across organizations, which automatically incorporates many of the best practices we recommend. Also, others may have already tested their reliability and validity, and uniformity in surveys allows their results to be compared across organizations.

Be transparent with metrics and survey results

Transparency brings accountability. In addition, internal transparency builds trust and confidence, while external transparency allows a company to lead by example. This benefit holds true even though not all metrics need have the same level of transparency, and there are likely to be tiers of access to the information.

Metrics should be consistent across the industry

We recommend that companies prioritize metrics that provide a broad, useful overview of the diversity landscape from top to bottom.

  • Employees overall, by function, seniority, and tenure, cut by demographics

  • Employee status (full-time/part-time/contractor), cut by demographics

  • Management and leadership, cut by demographics

    • Employees reporting to female managers

    • Employees reporting to managers from underrepresented groups

  • Salary, cut by demographics

    • Raises and bonuses, cut by demographics
  • Equity, for all-time and 12 months trailing, cut by demographics

    • Employee equity pool, for all-time and 12 months trailing, cut by gender and race

    • Investor equity pool, cut by gender and race

    • Vesting rates, cut by gender and race

  • Board of Directors, cut by demographics

  • Candidate pools and hiring funnels, by role, cut by demographics

  • Voluntary and involuntary attrition rates, cut by demographics

  • Promotion rates, cut by demographics

  • Complaints (formal and informal), cut by demographics

    • Complaint resolution status

Use inclusive demographic breakdowns

We recommend using detailed, fine-grained breakdowns for metrics and surveys. We recognize that some of these get very narrowly specific, which can pose problems for small organizations. One possibility is to roll the data up into broader categories as appropriate. It is certainly advisable to do so to protect individuals if their anonymity would be compromised otherwise. A common rule of thumb is to require that there be at least five people in a group before survey data is available for review, for example.

  • Race/ethnicity, with affordance for multiracial identity

    • African-American/Black

    • East Asian (including Chinese, Japanese, Korean, Mongolian, Tibetan, and Taiwanese)

    • Hispanic/Latinx

    • Middle Eastern

    • Native American/Alaska Native/First Nations

    • Pacific Islander

    • South Asian (including Bangladeshi, Bhutanese, Indian, Nepali, Pakistani, and Sri Lankan)

    • Southeast Asian (including Burmese, Cambodian, Filipino, Hmong, Indonesian, Laotian, Malaysian, Mien, Singaporean, Thai, and Vietnamese)

    • White

    • Prefer not to answer

  • Gender

  • Disability status

  • Sexual orientation

  • Family status

    • Children in the home part-time or full-time

    • Responsibility for the care of other people

  • Immigration status

  • Religious background

  • Veteran status

  • English proficiency

    • Languages spoken
  • Age and tenure at organization

  • Educational attainment

    • Highest degree

    • Highest degree of parents

  • College attended: public/private/any

  • Criminal background

Make sure you understand how and where the survey answers are stored, that IP addresses are not affiliated with answers to sensitive questions, and that information can be rapidly and cleanly deleted if necessary. Some demographic breakdowns, including disability, transgender status, criminal background, religion, sexual orientation, immigration status, or substance abuse history, can be instructive to study, but they are sensitive and present disclosure concerns. Employees may be reluctant to share them, and employers may be cautious about having them on file, especially in an uncertain political climate. Working with a third party survey company that obscures identifying information (including by not reporting on groups smaller than five employees), can help, but many do not delete information cleanly and could pose a risk.

Many thanks to Culture Amp and Paradigm for their guidance on recommended demographic breakdowns. These are largely drawn from their collaborative effort on an Employee Inclusion Survey.

Set employee, leadership, board, and investor demographic diversity goals

The first step to achieving goals is setting them. Within companies, we recommend setting demographic targets not only across the employee base, but also for leadership, the board, and investors. We know it can be challenging to set goals without some guidance, and you should be thinking about building a company that reflects the demographics of the country, not just the geographic locale where you do business.

You should also consider reflecting the demographics of your userbase, if it is even more diverse, but don’t rely on them for setting metrics. If you have a relatively homogenous userbase, they would be better served by a diverse company, as diversity drives innovation and better outcomes, including a bigger userbase.

Gender and race/ethnicity are among the most visible classes and the ones with some population data available for comparison and setting goals. Of course, gender and race are not the only relevant demographic classifications, but they are a good starting point  —  and a lack of representation of women and people of color is a strong warning sign of a problem, even if a company is diverse or inclusive in other ways.

Companies should also think carefully about setting timeframes for goals until we have better benchmarks than the numbers currently provided by large tech companies. Not every company, department, or team will have the same targets, nor boards or investor pools. It is instructive to consider a number of factors in defining long-term targets and the shorter-term, intermediate goals to get there. These include the population demographics of the country/countries, state/states, and city/cities in which a company is located or from which a company hires; of the available talent pool, by role; and of the current and anticipated or desired customer base.

For example, there are significant differences in racial and ethnic breakdowns depending on the populations in question and where they live. Census data can provide highly informative insights to compare the population of the country and your region against that of your organization. If you operate in an area that’s 40 percent Latinx and two percent of your employees are Latinx, or ten percent are Latinx but they all work in support services, for example, that suggests a problem.

Furthermore, immigration plays a non-trivial role in driving current tech company demographics. A company might list employees in Spain as “Hispanic,” which is technically accurate, but not really in the spirit of identifying diversity and inclusion issues in tech.

Be careful when conceptualizing goals. Different roles may have strong constraints given the available talent pool, but some constraints are artificial, like self-imposed guidance on only hiring from a limited set of schools or from specific majors, when those criteria are often irrelevant.

Repeat surveys regularly, at the right intervals

Optimal survey intervals depend mostly on whether a company has had an opportunity to make changes based on the previous survey and to measure whether employees feel that specific improvements were made. For example, if a company implements new communication mechanisms as a result of survey feedback, it may be advisable to wait six months after implementing the mechanisms to re-survey.

But short and lightweight pulse surveys, e.g., about 10 questions, can generally be done more frequently. However, many startups report survey fatigue setting in quickly  —  survey response rate can be a useful yardstick for measuring survey fatigue. We recommend a survey every six months. The CEO should promote the survey with several reminder messages and/or set aside an all-hands time for staff to complete it.

Publish metrics once or twice a year [50+ employees]

Publish diversity reports at least yearly. For high-growth companies that are hiring rapidly, at least twice yearly. These reports should include details on your diversity and inclusion program, ideally with some detail on what is working and why. If you are still figuring out your strategy and program and have fewer than 200 employees, publishing diversity numbers could cause you to become less experimental and less willing to run comprehensive analysis. At more than 200 employees, if you are not publishing a diversity report, it may be a symptom of deep problems that you should investigate.

Resources

We share these helpful references as starting points and encourage you to continue exploring.

Existing diversity data reports:

Beginning in 2014, many companies now publish diversity data reports of some form. Open Diversity Data is a good resource to find these reports, and below are direct links to some as well: