There’s a lot of debate on whether a machine can do a better job than a human being, especially on research projects like ethnographies that require significant experience and expertise to execute. The short answer of course is yes, and the evidence is increasingly becoming hard to ignore. It’s one of the reasons why I didn’t have a lot of “traditional” anthropologists as friends in the early years of starting our company. Back then at least, they found our technology and vision a bit hard to believe. They couldn’t wrap their heads around the idea that a machine could help them avoid the typical mistakes that they, as humans, make time and time again. No matter the experience, the human being, simply through the act of living and breathing and holding a set of values and beliefs, cannot find a way to let their judgment of a situation or a series of data points cloud the resulting outcome. An issue that is only worsened by the lack of scale in the process of typical research projects driven by human analysis.
Consider this short case study as further evidence of my point. When a team of experts that included a Harvard Economist, three Computer Scientists, and a Bail Expert from the University of Chicago, pitted a machine against highly seasoned judges in New York City on a series of large samples (550,000+ Defendants) of bail hearings, they found that the machine was significantly more accurate, despite the judges’ rich experience and ability to look the alleged ‘perp’ in the eye and make decisions through an empathetic lens. How accurate? The algorithm would have reduced crime rates (among those released on bail) by almost 25% and reduced jail rates by up to 41%. The algorithm was also able to counteract racial biases, which as you likely already know, is in and of itself one of the greatest challenges facing the criminal justice system in the United States.
When we started our company four and a half years ago, we began with a focus on building tools that could not just aid in the ethnography process, but most importantly, intervene at critical points in the process to minimize human error. We wanted to create a level of purity that theoretically we knew big data and AI could enable — we just hadn’t seen it in action in the social sciences yet. Specifically, we wanted to build the tools in a manner that forced our researchers to follow the natural breadcrumbs left by the consumer in any culture being studied. We didn’t want researchers to go searching for these breadcrumbs (which is what typically happens in the traditional process). And we certainly didn’t want them to make assumptions along the way, filling gaps with their own knowledge and experience. Through a series of experiments and learning over the course of more than two years, we not only discovered that this was possible, we also realized that the machine could take care of more than 80% of the research process. Data gathering, modeling, and even interpretation, all done by the machine to essentially lay out a map for our researchers and users to follow. Following this map, rather than needing to create it, means that very little is left to human judgment and the parts that do require researcher expertise and experience come much later in the research process — where the researcher is relied upon to follow the consumer’s breadcrumbs and develop a narrative out of the findings. Since the interpretation of each of these breadcrumbs is already provided by the machine, the researcher is relied upon to make sense of the various layers of interpretation, rather than coming up with the interpretations and then having to build a narrative out of it. This is a significant achievement in the world of cultural anthropology and ethnography, and I’m proud to say that we are yet to come across another company/technology that is attempting to do the same.
The first version of our technology was born in 2017. We’re already on version 2 now with significant advancements in the machine’s ability to interpret and predict the future of trends, topics, and cultures. We’ve been able to achieve all this because our technology has allowed us to shift the tasks that humans were needed for, further down in the process — which has meant better nuance and marginal error. So, coming back to the point I began this article with — comparing man to machine — I don’t think we’re at the stage yet where the machine can do everything a human anthropologist can. However, by leveraging technology in the right places, we’ve been able to change the role of the human anthropologist (in the ethnographic process) for the better, with greater involvement in the latter half of the research process where judgement errors are minimized and the benefits of AI and Big Data are maximized.
The results speak for themselves. Collectively, over the last 4+ years, our clients have launched more than 100 brands and products using our machine-led ethnographies, generating more than 5 Billion in new revenue for their businesses. Yet, it still feels like we’re only just getting started.