Every morning, millions of workers open an app to find their work assignments. The delivery rider in Rajasthan, the Uber driver in Kolkata, the Blinkit courier in Delhi, all work within the same established narrative: the platform is neutral, the algorithm decides the work, and human bias has no place in the system because data is everything.
That narrative is misleading, and always has been. In India, where society has long been shaped by entrenched caste, class and gender hierarchies, it is particularly useful for those in power to sustain this illusion of neutrality. The real question is not whether algorithms are biased — they clearly are — but whether that bias enters accidentally or sits at the very core of these systems.
The answer is not as complicated as it is often made out to be. When a delivery worker’s rating falls below a certain cut-off, they can be removed from the app almost instantly, losing their livelihood overnight. There is no manager to hear their side, no review process, and often no explanation from the platform itself. The company does not disclose why the decision was made because doing so would acknowledge that a human judgement exists somewhere in the chain. Silence becomes strategic. It was “the algorithm’s decision”, and therefore beyond contestation, at least for now.
This is precisely what happened in Hyderabad in 2024, where a delivery worker was removed from a platform without explanation. If gig workers were recognised as employees rather than independent “partners”, the financial model on which these platforms depend would begin to crack. Keeping both the work and the worker invisible is what allows the system to function.
While the Blinkit rider logs in merely to survive the urban economy, data annotators in labour hubs spend hours manually labelling images of street violence, disasters and explicit content. Their labour trains the very generative AI systems that the industry proudly describes as autonomous cognitive technologies. The narrative of artificial intelligence as a seamless, self-generating revolution is itself built upon the erasure of labour. Millions of human hands feed the machine, yet they are carefully edited out of the story. What remains visible is the output; what disappears is the human being behind it.
Whenever algorithms are criticised for bias, the standard defence is familiar: the training data was not representative enough, and better datasets will solve the problem. But the issue runs far deeper than flawed sampling. Algorithms absorb the prejudices embedded within the societies that produce the data in the first place. They then convert those prejudices into mathematics, reproduce them at scale, and stabilise them as systems of automated decision-making.
India offers a particularly sharp example. Algorithms learn from a social history in which privileged communities disproportionately occupied technical and professional roles because institutions were structured to keep them there. The bias is therefore already embedded in the training data before the algorithm even begins learning. There may be no explicit column labelled caste or class, but the consequences remain visible everywhere, hidden in plain sight.
Technological artefacts carry politics. Behind every digital tool or automated system are human choices. Usually, a relatively small group of people decides what counts as “efficiency”, “merit” or “productivity”, and those assumptions are then encoded into software. Their own social locations and institutional biases inevitably shape the systems they build.
Take the NREGA job-card system. The architects of the system assumed that the male member of the household would naturally be its head. As a result, digital forms were designed to register women under their husbands’ names. The consequence was immediate: many women lost the ability to collect wages independently. Nobody needed to openly argue that women deserved less financial autonomy because the discrimination had already been built into the architecture itself. The system simply reproduced the assumptions encoded within it. Bias became invisible because it appeared as the default.
Perhaps the most widely discussed illustration remains Amazon’s abandoned automated recruitment engine. In 2018, the company scrapped the system after discovering that it systematically downgraded applications from women’s colleges and penalised CVs containing the word “women’s”. The reason was straightforward. The algorithm had been trained on a decade of hiring data dominated by male engineers and, therefore, concluded that being male was structurally associated with success in technical roles.
When Indian IT firms replicate similar recruitment pipelines, they inherit the same logic within an Indian context. The surnames, educational institutions and social markers associated with “retention” or “performance” are not neutral indicators. They are residues of exclusion accumulated over decades.
The same concealment operates within automated performance-review systems. Research by J-PAL South Asia documents how patriarchal expectations and domestic burdens continue to shape women’s participation and retention in tech-mediated workplaces. The performance algorithm learns from these unequal conditions and then reproduces them as objective institutional conclusions. Subjectivity is effectively laundered into the language of data.
In India, policymakers frequently point to expanding digital networks as evidence of empowerment. If a delivery worker from a marginalised background owns a smartphone, policymakers are quick to present it as evidence of empowerment and social progress.
But does technology genuinely empower that worker? Can they negotiate wages, challenge unfair decisions, secure meaningful rights, or even understand the forces governing their working lives?
For many gig workers, the reality is harsher. The smartphone functions less as a tool of liberation and more as a mechanism of surveillance and discipline. Our legal systems were built for a world in which identifiable human beings made decisions. On digital platforms, however, that decision-maker has effectively disappeared. What remains is the worker confronting an opaque system, and the balance of power is overwhelmingly tilted against the human being.
For nearly two decades, India’s technology policy has been shaped by the language of democratisation: the promise that the internet would flatten hierarchies, platforms would distribute opportunity, and algorithms would remain blind to prejudice. But that vision was never likely to survive contact with reality.
The people building these technologies are themselves products of institutions that reproduce the very hierarchies technology was supposed to dismantle. The same structural logic shaped who built the systems, what values were embedded into them, and what forms of behaviour the systems eventually learned to reward.
Yet technology cannot be understood only as a tool of exploitation or erasure. Dalit communities have used social media to document caste-based violence, frequently ignored by mainstream media. Gig workers have organised themselves through WhatsApp networks with a speed and reach that traditional trade unions often struggled to achieve.
But these forms of resistance emerged despite the architecture of the platforms, not because of it. Even when platforms claim not to “see” caste or gender, those very hierarchies may already shape the architecture of the system itself. Algorithms may not explicitly speak the language of class, but they may nevertheless embody the logic of class power.
Unless debates around AI ethics, automated recruitment and digital governance confront these structural mechanics directly, they will remain incapable of addressing the real problem.
Technology is not neutral, and it never was. The real question, the one that ultimately determines everything, is this: who was the algorithm engineered to protect, and who was it engineered to make disappear?
(Pratishruti Bandyopadhyay is a researcher and a graduate of IIT-Jodhpur. Her work examines the intersections of technology, labour, language and society)
