Existing Tools for Content Moderator Wellbeing
The online sphere is rife with harms, spanning from innocuous spam to violent and egregious content. The most harmful content includes child sexual abuse and exploitation imagery, suicide and self-injury content, and terrorism and violent extremism.
Human moderators are tasked with removing such harmful content daily, as hundreds of thousands of pieces of content (audio-visual, text-based, and even live-streamed) appear across various platforms.
Given the proliferation of harmful online content and the potential psychological risks posed to human moderators, tooling systems have been implemented to support content moderators in mitigating overexposure to egregious materials.
Audio Moderation Tool Advances for Moderator Wellbeing
Typically, companies will partner with tooling companies that specialize in developing automated modifications to content, which reduce the risk of harm to moderators.
Automated Content Tools Reducing Harm
For audio-visual content, this may include automated gray scaling or blurring of imagery, reducing image or video size, and automatically muting audio through specialized audio filtering technology.
Text and Live Moderation – Tooling for Effective Protection
Text-based tooling may include features such as profanity filtering or automated detection of harassment and abuse that require less human intervention.
In the context of live streaming, moderation becomes even more challenging. AI Automation and machine learning, as a first course of response, have aided, with human moderators acting as a second review mechanism.
Risk Scoring and Accurate Tooling Solutions
More advanced tooling available to moderators may include risk or toxicity scoring. This is achieved through AI tools that can consider context and score harmful content in real-time, enabling teams to identify the most harmful content and take immediate action, as well as quickly detect unknown violations of platforms’ community guidelines or terms of service.
Shortcomings in Current Moderator Tooling
While tooling and automation through artificial intelligence have indeed reduced the risk of harm to moderators over the years, there remain reasons that tooling falls short of meeting the needs of this population of employees.
Productivity Versus Moderator Protection
The industry hears time and time again that moderators are under extreme pressure to remove violating content as quickly and accurately as possible. This results in moderators opting out of protective tooling options, such as image blurring and audio muting, because it slows down their productivity.
While opting in can reduce the shock of sudden exposure to egregious content, which may trigger a fight, flight, freeze, or fawn response, many moderators still choose to opt out to maintain productivity.
Accuracy and Content Context Challenges
In our experience, moderators have cited that these tools are impractical as they hinder the ability to make accurate decisions. In many cases, moderators will switch off the tool so they can view the materials as a whole, understanding context and nuance, ensuring accuracy in cases of violating content.
For example, if a moderator is reviewing a piece of content tagged as child sexual abuse and the tooling system has automatically blurred the visual, the moderator may still need to unblur the visual to verify the age of the child, particularly if the abuse is being perpetrated against an older adolescent.
Desensitization and Tool Adoption
Moderators who are tenured and have been exposed to egregious content for long periods may be susceptible to desensitization.
In psychology, desensitization is a process whereby individuals experience a diminished emotional response to aversive, negative, or even positive stimuli after repeated exposure. This may also be known as habituation.
In these cases, moderators who experience a diminished emotional response to egregious content may opt out of protective tooling options because they feel they no longer need them.
If moderators underutilize these protective tooling options due to reasons related to productivity, accuracy, and desensitization, this raises the question of what types of precision-based moderation technologies would better suit their needs to safeguard their psychological health and wellbeing.
While it is imperative that protective tooling measures, such as grayscale, audio muting, profanity filtering, and automation, remain in place for moderators, the reasons these are underutilized present a strong case for considering different tooling safeguards.
Preventative Mental Health Tools for Content Moderators
We have already seen some companies tackling this through technology solutions, such as integrating Tetris-like games into the wellness suite of moderators’ systems or sending out wellness break reminders throughout the day to support overall mental wellbeing.
However, these tools rely on moderators’ individual ability to notice when their sympathetic nervous systems are being activated, inducing an FFF(F) response.
Organizational Factors Affecting Tool Effectiveness
These tools also rely on the organizational structure to allow moderators to pause during work and engage in these activities, which can be challenging when productivity and quality metrics are key to maintaining a safe platform.
Holistic Approaches to Moderator Wellbeing
Rather than placing the onus on individual moderators, we advocate for a holistic approach that considers all aspects of the moderation role and their impact on the psychological health of moderators.
This not only includes exposure to egregious content but also systemic challenges such as team dynamics, leadership compassion, psychological safety in the workplace, workforce management, poor and ineffective tooling, and more.
The future of tooling should be preventative by nature (like existing tooling) but should focus heavily on ensuring moderators are provided agency and autonomy to utilize the tools effectively.
Cross-Industry Partnerships for Accurate Tooling Solutions
We strongly believe in the need for cross-industry and professional partnerships to enhance tooling and provide wellbeing solutions for Trust & Safety teams.
Key stakeholders need to be involved to ensure that these tools meet the needs of moderators, ensure their engagement with these tools, and address existing gaps.
Collaborative Development for Preventive Mental Health Tools
Mental health professionals, researchers, product teams, technology experts, and Content Moderators should come together to tackle this issue. What might this look like?
- Data science that underpins why moderators are under-utilizing existing tools and their specific needs for future tooling
- Product development that deeply considers point 1, which is led by a “safety by design” mindset
- Exploration of existing technology solutions that can accurately measure acute stress in real-time without infringing upon data privacy
- Development and implementation of new tooling features and a stepped-care approach to intervention.
Zevo Accreditation Program – Building Moderator Resilience
Advance your skills with the Zevo Accreditation Program (ZAP), a CPD-approved certification for mental health professionals in Trust and Safety.
Learn best-practice, evidence-based interventions to support Content Moderators and tackle risks like vicarious trauma and burnout.
Reframing the Future of Preventative Mental Health Tooling
Future preventative tooling must support Content Moderator’s wellbeing through practical, collaborative design. With the right structures in place, tools can strike a balance between accuracy, psychological safety, and ease of use.
Visit Zevo Health and our Trust & Safety Wellbeing Solutions to strengthen support for moderation teams.
