Shoshana Zuboff’s 2019 book, The Age of Surveillance Capitalism is a powerful critique of the interplay between capitalism and Big Data. In about 700 pages pages, she makes a convincing argument that the very existence of human autonomy is under threat. In short, Dr. Zuboff is concerned that large companies have perfected a profit loop in which they collect immense amounts of data on consumer behavior, devise models that accurately predict behavior using that data, and then use those models to manipulate consumer behavior to drive profit. It is an intriguing and important read.
It defines several ways of thinking about the data collection and machine learning technologies at the core of this new surveillance capitalism. I think the book falls short in articulating a path forward, especially for people who work in the data industry.
If you’re a data scientist, analyst, or work in any of the myriad jobs behind this phenomenon, you might have concerns about how your 9-to-5 contributes to this mess. If so, you’ve probably asked yourself what to do – how to respond as an ethical, responsible, perhaps moral human being. I’ve put down some thoughts below, chime in if you’d like!
We need to name these things!
My partner always emphasizes that to address an issue, we need to be able to name it as precisely as possible. The devil hides in the details. To those of us with academic training particularly, we have the obligation to identify these names and make them accessible to our people – to make it plain as my grandmothers would say.
In that spirit, there are a few terms that I pulled from The Age of Surveillance Capitalism that helped me view commercial Big Data in a new light.
Rendition
Rendition is the activity that we (or our ETL flows) engage in when we reduce the features and signals about some human activity or human to some set of observable features and use those features to completely describe the person or activity. We render their engagement with our landing page to the act of clicking, the number of milliseconds from rendering of the page to first clicks, the time spent scrolling, etc.
We render emotion using the straw of a signal that we get from our browser API. She talks about rendition of emotions, rendition of personal experience, rendition of social relations, and rendition of the self: all operations that reduce very rich and complex human activities to numbers.
Behavioral Surplus
Behavioral surplus refers to data collected which exceeds the amount required for the application or platform to complete the task at hand. You have an app that provides secure communication. A user happens to talk about a certain brand of sneakers a lot when they’re using your app. Capturing information about their shoe habits doesn’t have much to do with secure instant messaging, but you capture these interactions and share them with a data aggregator who in turn shares them with a shoe retailer. The behavioral surplus is the “sneaker interest”. Your app user who was counting on privacy and security knows that their trust was an illusion when ads for the exact shoes show up everywhere they look online. Maybe that user is undocumented. Maybe ICE shows up at the Foot Locker they shop at.
Instrumentarianism
Instrumentarianism refers to a form of behavioral control, based on Behaviorist models of activity modification, enabled by large cross platform logging of online behavior, combined with predictive machine learning, combined with the capacity to conduct online behavioral experiments. The Instrumentarian regime knows enough about a given population, so that with some certainty C, when an action A is presented to this group, they will perform some behavior B. The regime can thus control this group with a given certainty. Because of the economies of scale, the Instrumentarian need only guarantee a small percentage of response to this stimuli to reap enormous profit.
Sanctuary
The right to sanctuary is the human right to some place (real or virtual) that can be walled off from inspection. Surveillance capitalism, through always on devices, potentially removes all sense of sanctuary. Every conceivable space is monitored through “smart” bathroom scales, “smart” televisions, “smart” smoke alarms, robot vacuum cleaners. In this theory, any device is loaded with surplus sensing that reports on and records personal and social interaction far beyond the purpose of the device. That is, your Echo records you even though you might have just purchased it to play your favorite music. For the surveillance capitalist, the device more than pays for itself by providing precious behavioral insights that might have otherwise been hidden in “sanctuary”.
Holes in Zuboff’s analysis
As I was reading, I noticed three glaring oversights in Zuboff’s analysis.
Instrumentarianism is explicitly violent
Zuboff makes the claim that Instrumentarianism does not require violence for control. But I still leave her book with a concern about the potential for violence that Instrumentarianism and Surveillance Capitalism empowers. To quote from the announcement for the upcoming Data for Black Lives Conference
Our work is grounded in a clear call to action – to Abolish Big Data means to dismantle the structures that concentrate the power of data into the hands of a few. These structures are political, financial, and they are increasingly violent.
increasingly violent. We just have to reflect on the way in whic digital manipulation and capitalist interest have coalesced in state supported violence in support of India’s Citizenship Amendment Act The 2019 El Paso shooting or the complex interplay of anti-Black policing with surveillance technologies and corporate interests in tools like Amazon Rekognition illustrate the complex ways in which violence has and could play out in surveillance capitalism.
Impact upon the marginalized is profound
Zuboff does not discuss the ways in which surveillance capitalism is especially exploitative of marginalized populations. Absent was a discussion of how companies like Palantir and private prisons exploit the incarcerated, including asylum seekers. Absent was the discussion of how Instrumentarianism is particularly pernicious and violent with respect to marginalized populations including the homeless, the undocumented, communities of color, the poor, and LGBTQ communities. There are important voices that are startlingly absent in Zuboff’s analysis.
Data sharing can be empowering
Zuboff doesn’t like the idea of the “hive mind” — mass collection of patterns of activity. When people of a group commit to a collective model, it can be a source of empowerment. Many of the policing abuses were identified because everyday people were able to pool knowledge and data – shared spreadsheets documenting the high incidence of stop and frisk, shared cell phone videos of police shootings if innocent citizens. Class action lawsuits against workplace discrimination and the willingness of people to participate in studies to identify housing discrimination are two activities that come to mind in which a “hive mind” (the group discovery of adverse behavior on the grand scale) is able to benefit the individual and the community. I think that she is arguing for transparency – what are the power relationships that the technology is reinforcing – as opposed to the technology itself.
A plan for action
Articulating the answer to “where do we go from here” is where the book falls short. Zuboff claims (convincingly) that nothing less than human agency is at stake. As far as I could tell, Zuboff’s only suggestion is to trust that the EU’s General Data Protection Regulation (GDPR) would come to our rescue. I think that’s too precious a right to trust just the GDPR.
So what to do?
Here are a few thoughts. Got more? Please chime in!
Unite and act
Ruha Benjamin in a recent talk cited an example of how students in a high school refused have their education mediated by a Facebook instructional platform. They demanded face to face instruction with human teachers and they refused to have all of their learning activity be rendered into the Facebook cloud. They lead a successful campaign for real instruction from real teachers and won. According to Benjamin, the most successful campaigns against surveillance capitalism require collective effort – the technology is too ubiquitous for the single individual to have much impact. A protest staged today at a dozen colleges against the use of facial recognition on campus emphasizes the power of collective action.
So you’re never too old or young or too math or technology challenged to act and make a difference. I think the most powerful “next step” is to think through what to do with your family, friends, neighbors, classmates, community. Start locally.
Educate yourself
There are a host of recent books, articles, posts, and classes that provide a more complete view and history of how to deal with surveillance capitalism. A sampling includes:
- Ruha Benjamin’s Race After Technology: Abolitionist Tools for the New Jim Code
- Cathy O’Neil’s Weapons of Math Destruction
- J. Nathan Matias’s Course Designing Field Experiments at Scale (the ethics of online experiments)
- The Detroit Community Technology Project along with other community organizations launched the Our Data Bodies Project which created a fascinating report Reclaiming Our Data that lays out concrete steps to reclaim personal and private data.
- Data For Black Lives organizes conferences and has outlined an action plan.
- Christo Wilson has done empirical studies on online data collection and experimentation platforms, and here is a presentation he gave at Automattic
Find or create community
There are professional and community organizations that are thinking through ways to rethink data science and other technologies ethically:
- the CONTESTED DATA workshop is looking at the fundamental premises of the right to collect data
- Data for Black Lives 2020 will interrogate how data collection and use is impacting the lives of Black people.
- The ACM Conference on Fairness, Accountability, and Transparency
- AI for Social Good Workshop
Advocate to center ethics in computing education
Computing is becoming more common in primary school, but in depth education on ethics and computing is still rare. It should be a central component of computing instruction. I’ve come across just a handful of classes on ethical design of data platforms. The topics usually appear in advanced classes or as the final lecture in course on machine learning or data science. Given the immense impact that even the simplest social app can have, we should advocate ethics having a central role in computer science and related fields. Talk to your friends in academia, the high schools and grade schools in your neighborhood. Or just…
Teach
A lot of the technology at the core of surveillance capitalism is available to everyday people in spreadsheet packages and cloud environments. The same tools and algorithms at the core of the surveillance regime can be “flipped” to identify and counter manipulative pricing, discriminatory and racist patterns. The groundbreaking work of Rediet Abebe demonstrates the potential of the good that can happen when the tools of data science are used by every day people to improve their lives.
Advocate for transparency and ethical deployment of software systems
Even inside Google, employees were able to advocate for model cards that explain to cloud service users how machine learning models are trained, how they might be biased. Google employees also raised questions about many of the company’s instrumentarianist practices. Certainly, advocating for transparency is one move that can insure that users are provided basic protections. There is still little in the way of openness and ethical training that is provided to users of online experimentation platforms like Optimizely
Build an inclusive workplace
People continue to argue that if we include the perspectives of the marginalized in the development of these platforms, we may be less prone to rush to deploy them in profoundly abusive ways. The presence of marginalized voices in the large surveillance companies remains at unrepresentative levels. Beyond hiring practices, we’ve yet to see wide-scale development of community governance structures implemented. As a data professional advocate for equity and inclusion.
Understand the humanity of your customers
It can be easy sometimes for the data professionals to think of “customers” as a probability distribution, a score, or a click. To use Zuboff’s terminology, our view of our social relationship with the people that use our systems have been subject to a kind of rendition as well. The data that you’re looking at makes it hard to associate a human being with those numbers, much less connect with one. Further, even if you are directly connected with a front end product (for example, making suggestions of videos to watch), you’re more often that not playing a game of aggregates – you’re looking at an abstraction of the actions of millions of people.
Final words
Surveillance Capitalism is a worthwhile read despite it’s flaws. We live in a critical time for data science and ultimately it will be up to all of us to determine what direction it takes.
Charles write here 