Content moderation has become increasingly important for online platforms to protect their users from potential abuses. The evolving regulatory landscape has also put growing responsibilities on the way user-generated content should be moderated online. Notably, the upcoming Digital Services Act (DSA), which affects almost every online service provider active in the EU, will bring unprecedented obligations to online services in a wide range of sectors, as well as considerable penalties for those who fail to meet the new requirements (up to 6% of annual global turnover).
Similar regulations are under development in multiple jurisdictions around the world (Australia, Canada, UK, and South Korea – to name a few). Thus, designing and implementing a strategy for content moderation is vital not only for contributing to online trust & safety and ensuring the retention and satisfaction of the platforms’ users, but also for a company’s ability to do business in the markets where regulations are being developed. A company’s success will largely be determined by the degree to which it has managed to ingrain the new content moderation requirements into its business model.
To understand the challenges in achieving efficient and effective content moderation, Tremau interviewed content moderators and managers working in the Trust & Safety departments across more than 30 companies, ranging from mega platforms to early-stage start-ups. Notwithstanding the different types of content that moderators are exposed to given the diversity of online platforms, we have identified a set of common important practices adopted by companies and clear areas for improvement. Three major sections identified include: detection of harmful or illegal content, moderation process and controls, and crisis management.
A major challenge in content moderation is the tremendous volume of content produced in real-time. In order to accurately identify the very small proportion of potentially problematic content from the rest, companies often use a mixture of re-active moderation (answering user reports) and pro-active moderation (automated detection tools). Based on pre-determined rules or machine learning detection models, AI-empowered automated detection usually selects content that is potentially illegal, such as terrorist content or counterfeit products, or content that clearly violates a company’s terms of service. Many companies also employ automated tools as a preliminary filter, and based on the confidence threshold in the detection, a human moderator is introduced in the process to verify results.
Despite the improved efficiency brought by automated detection, the overwhelming majority of our interviewees have pointed out that the room for improvement is still large. One frequently mentioned drawback is the difficulty in treating nuanced cases, which makes a human moderator’s job indispensable. Moreover, no AI tool can be a perfect substitute for human intervention in this job given the continuously evolving and highly diverse culture and requirements. Thus, automated content moderation tools should not be built upon the principle of replacing human moderators, but of working with them.
A common issue with content moderation systems is that companies typically have to continuously fill the gap between their existing workflows and the evolving regulatory obligations – often by frequently “patching” their moderation systems. Thus, a much-needed capability is to build content moderator-centric systems according to the company’s evolving regulatory obligations, allowing better coordination among different teams and a more effective and efficient moderation strategy.
Violations of content policies are often categorized into pre-defined groups such as violence, foul language, and extremis. However, moderators can often find themselves reviewing much more nuanced, complex or context-sensitive cases. A key practice adopted by various companies is to establish multi-level moderation teams & processes. In this structure, frontline moderators are responsible for making a “Yes or No” decision for the most clear-cut cases, and send more complicated cases to higher level moderators who have more experience as well as access to more information. In rare situations of very difficult cases, senior Trust & Safety managers or other departments concerned discuss and make the final solution.
Another practice to support moderation decision-making for frontline workers is to use a decision tree during the moderation process, a practice that has been widely adopted by customer support departments and other call centers. By decomposing a complex moderation question into an array of smaller and easier options, a decision tree allows moderators to judge cases in a more structured and standardized manner, which can boost the efficiency and quality of the overall process.
Accuracy and consistency of content moderation are also key concerns. Companies develop both ex-ante and ex-post control measures to improve the quality of content moderation. Intensive training before starting as a moderator is commonly seen across companies, and regular training sessions also take place in many companies to keep moderators tuned in with the latest regulatory or terms of service updates.
Considering the constantly evolving regulations, at both national and international levels, companies often draft extensive and detailed guidelines for moderators to refer to before reaching a decision. Reviewing the accuracy of past moderation decisions on a regular basis is also widely adopted by companies. Often a random sample of the total cases treated by a moderator in any given period will be pulled from stored data and sent for examination, or some cases may be given to multiple moderators to examine their consistency; the calculated accuracy rate is often a key component of the moderators’ KPI.
Another key challenge during the moderation process is that content moderators’ tasks involve much more than simply judging whether a post should be removed or not. For example, crisis management is also part of their job when, for example, they encounter urgent cases, such as a livestream of self-harm or terrorist attack such as the livestreaming of Buffalo shooting. Such cases demand immediate outreach to law enforcement departments or other appropriate local authorities and should be considered as the digital “first aid” of our time.
Content moderators also need to provide some degree of customer support, as users may file complaints against certain moderation decisions – hence moderators must also be enabled to easily retrieve all relevant information of past cases or of users to better communicate with them.
Although content moderation is essential for almost every online platform that hosts regular interactions among users, most companies usually do not have enough resources to build or, often more challenging, to maintain and keep up-to-date, efficient and effective internal moderation systems. On this note, Tremau’s conversations with content moderators enabled us to identify a number of recommendations to create an efficient and consistent content moderation processes.
For example, given the multi-faceted nature of content moderation, the most efficient approach to enhancing content moderation processes is to integrate related functions and controls into a more moderator-centric centralized system, which enables the moderators to avoid constantly shifting between tools, ensuring a smoother workflow, important efficiency gains and more accurate KPIs and quality control.
A centralized system also allows data to be reconciled in a unified platform, thereby giving moderators the complete context needed to make decisions and enabling automated transparency reporting. It also facilitates a risk-based approach via prioritization, which allows moderators to treat cases more effectively and enables the implementation of convenient contact channels with authorities and other stakeholders in case of emergencies. Such rapid reaction mechanisms are still not mature enough in many companies.
With access to more efficient processes as well as analytics, it then becomes possible to also better protect moderators’ wellness against traumatizing content.
To meet the challenges of protecting their users & complying with regulations that are continuously evolving, a number of online platforms will need to enhance their content moderation processes and controls. The measures discussed above streamline the moderation processes to be more efficient, and – with appropriate structuring of data – can automate transparency reporting, which is increasingly in demand across voluntary codes and regulations.
With regulations such as the Terrorist Content Online Regulation, which sets a 1-hour limit for online services to remove Terrorist and Violent Extremist Content (TVEC) from their platforms, there also needs to be further investments into reliable mechanisms to prioritize content in moderation queues. Thus, “Compliance by Design” will become a necessary focus for building effective and future-proof content moderation systems. Successfully building these capabilities will soon become a key differentiator, and even a critical factor, for survival.
Tremau’s solution provides a single trust & safety content moderation platform that prioritizes compliance as a service and integrates workflow automation and other AI tools. The platform ensures that providers of online services can respect all DSA requirements while improving their key trust & safety performance metrics, protecting their brands, increasing handling capacity, as well as reducing their administrative and reporting burden.
We would like to thank all the content moderators & managers who took the time to talk to us and contributed to our findings.
Tremau Policy Research Team
The growing regulatory spotlight on content moderation, shorter deadlines for content removal, growth of detection of potentially illegal or harmful content to be reviewed, and pressing needs to protect both the safety and freedom of expression of users, has increased the urgency to enhance existing online moderation practices. With these practices becoming widespread, it is important to ensure that this process is effective, efficient, of high quality, and that it keeps the best interests of all stakeholders at heart.
To achieve this, let us look at three key points in the process that can be optimized going forward:
Receiving continuous alerts from users can be overwhelming for human moderators, especially over extended periods of time. At this junction, it is crucial to prioritize and manage alerts – rather than follow, for example, a “first-in-first-out” or other sub-optimal approach. A solution for this is to ensure that user reports are labeled according to the level of harm they could cause (following a risk-based approach) and based on statistical analysis of the available metadata. This is important for user safety – especially in cases of emergency – as it allows cases that are time sensitive to be dealt with quickly. It can also be beneficial for moderator safety as they are warned that they will be viewing more harmful or less harmful content. A lesser considered point when discussing management of user reports is the moderators’ experience of the process itself. An optimized moderator screen can save decision making time and increase overall process efficiency by more than 20%.
Another pain point in content moderation is managing the process across a variety of platforms, people, and teams. As regulations demand increasing responsiveness and complaint handling from online services, it is important to ensure that you have the right mechanisms in place for end-to-end moderation and complaint handling that is helping build user trust and protect your brand. For instance, a moderation case cannot close immediately once it has been handled after a very first notice. This is because, under the Digital Services Act (DSA), a user can still contest – for at least 6 months – the handling of the case and even take the complaint to an out-of-court dispute settler. Content moderation teams will thus need to account for the possibility of the case continuing beyond the initial handling. This includes making sure that the complaints are uniquely identifiable to streamline this process and that all relevant information is easily available to ensure process quality.
The third point to consider is growing transparency reporting requirements. Over recent years, calls for transparency reports from online services have come from civil society and governments alike. This has led to a variety of different frameworks from private actors in the ecosystem and resulted in transparency reporting becoming a key part of digital legislation, as seen in the DSA. Transparency is critical to ensure the safe and fair moderation of online platforms. To produce comprehensive transparency reports, it is crucial to keep a clear and consistent account of all requests for removal or restriction of content. To do this, the tools used by the moderators need to be effective at managing large volumes of notices as well as streamlining storage and labelling of data.
Optimizing your content moderation processes will allow you to be more efficient with your costs as well as more effective in protecting your users, moderators, and brand. To achieve this, it is important to introduce new processes, incorporate automation and intelligence to improve speed and quality, and build moderator-centric tools. More importantly, it is critical to prioritize quality assurance to ensure that the right balance between safety and freedom of expression online is met.
With regards to regulation, the DSA states that following a user report the company is liable for the effect given to it. Poor content moderation can raise reputation, regulatory, and other business risks which can also lead to loss of users and market share, as well as significant fines (up to 6% of global annual turnover). Thus, adopting a content moderation system that meets technical compliance requirements from the get-go, as well as prioritizes human safety and quality, is crucial.
The Tremau tool is a single end-to-end content moderation platform designed to help you streamline your processes, automating them whenever possible, managing and prioritizing different reported content (whatever the source of detections), as well as continuously producing audit trails for transparency reporting – enabling you to cut costs and collaborate more effectively. The end-to-end process on a single platform allows all team members to see the progression of cases and ensure better communication, faster treatment, higher consistency and quality, and fewer bottlenecks in internal handling – while ensuring the privacy of its users.
The tool is also created to ensure smooth experiences for moderators. This is done through limiting the number of clicks and screen changes as well as including API connections to external stakeholders to ensure rapid contact. Finally, the tool collects and analyzes data throughout the end-to-end moderation process to ensure that nothing falls through the cracks and absolute transparency can be maintained. Such improvements enable platforms to increase reaction times towards removing or restricting content, thus ultimately protecting users and society. Moreover, it keeps the well-being and retention of moderators at their core by taking steps towards ensuring that their exposure to harmful content is limited and their tasks are streamlined.
To learn more about how Tremau can help you, contact us at info@tremau.com.
Tremau Policy Research Team
Online content moderation has been an increasingly important and debated topic, with new regulations, such as the EU’s Digital Services Act (DSA), expected to further reinforce this trend. Regulations will create more legally-binding obligations for online platforms with respect to content moderation, in order to improve users’ online well-being and the better functioning of the online world.
However, while millions of posts appear every day on social platforms, only a few hundred thousand people work in the current content moderation industry. Despite plans from platforms to recruit more moderators, the amount of work managed by each moderator remains very large: they often have to review thousands of posts every day, leaving them with a very narrow (and stressful) window to decide whether or not an online post should be removed, raising possible issues regarding the accuracy, consistency and potential fairness of a company’s content moderation and its impact on free speech.
In addition to the very limited time to make moderation decisions, the quality of moderation can also be affected by AI tools deployed by platforms, the highly contextual nature of many online posts, and the large quantity of online content falling in the grey zone between harmful and safe. Potential biases of content moderators further exacerbate the issue. For example, some moderators might be too lenient or too strict with respect to company guidelines, and can also be impacted by how long they have been working in the day, others may be accurate on some categories of instances but lack the expertise or training on some others, while other moderators might be biased specifically towards some categories of content (e.g., culturally, politically, etc).
Ensuring the quality of content moderation is a challenge that has important implications for the proper functioning of social media and freedom of expression online. Quality assurance (QA) for content moderation is essential to ensure that the right balance between safety and freedom of expression is met in a fair and effective manner. Poor content moderation can also raise reputation, regulatory, and other business risks for online platforms, including a possible loss of users. QA becomes even more challenging and important as companies outsource content moderation to external providers – whose quality also needs to be continuously monitored. In this context, online platforms are looking for ways to monitor and improve the quality of their moderation processes. Quality can be measured using metrics such as accuracy, consistency and fairness (e.g. similar cases get similar decisions). Consistency is critical both over time for each moderator and across moderators.
The typical quality assurance process for online content moderation is based on performing regular (for example weekly) controlled evaluations: for example, after carefully labelling a number of content items (e.g., users’ posts), managers provide them to multiple moderators, which allows to compute a score for each of them based to how they perform relative to each other as well as relative to the desired labels the company selected for these items.
However, this common QA practice does not leverage all data available, and as the evaluations are done only once a while, one cannot detect potential QA issues real time – for example because a moderator may drift even temporarily. An important challenge related to quality and consistency evaluation is the ability to use many, if not all past decisions from all moderators, in order not to be limited by a small number of weekly test instances. Very importantly, this help get rid of additional evaluation processes entirely, while improving the reliability of the evaluation and ensuring continuous monitoring.
In our study, we discuss some approaches for managing content moderation quality real time, without the need to perform regular (and costly!) tests or requiring multiple moderators to handle the same cases. We develop a new method for comparing content moderators’ performances even when there is no overlap across moderators in the content they manage (i.e., each instance is only handled by a single moderator), using the data of the moderators’ previous decisions. To this purpose, we also discuss how to adapt crowd labelling algorithms for performing QA in content moderation – an approach that we believe can be promising to further explore.
To find out more about building an accurate and efficient content moderation system, contact us at info@tremau.com.
To download Improving Quality and Consistency in Single Label Content Moderation, please fill out the form below.
Tremau Policy Research Team
Content moderators have become indispensable for online platforms’ everyday operations. However, major platforms outsourcing their content moderation to contractors all around the world face an increasingly pressing challenge: Employee turnover at these sites is high, as most moderators cannot continue for more than 2 years on average.
Poor mental health is one of the major reasons behind moderators leaving their positions, as their jobs require them to review large volumes of texts, pictures, and videos containing highly disturbing content around violence, extremism, drugs, child sexual abuse materials (CSAM), self-harm, and many more. Long-term exposure to such harmful content has triggered serious mental health issues among moderators, including depression and anxiety. With deteriorated mental health conditions, more severe issues like PTSD and addictions to drug and alcohol have also been noted to emerge.
Disturbing content does not only cost content moderators their mental health, it also has a financial impact on platforms. For example, the San Mateo Superior Court required Facebook to pay millions to content moderators who had developed PTSD on the job. Moreover, as Non-Disclosure Agreements (NDA) have become common practices, content moderators often find themselves unable to talk to trusted friends or family members about their work experience. This leads to a lack of support for moderators, misunderstanding of their precarious conditions, and growing unwillingness to voice their difficulties.
The intensity of the job is another major problem. While there are millions of posts appearing on various social platforms every day, there are only about 100,000 people working in the current content moderation industry. Despite mega-platforms’ promises to recruit more moderators in recent years, the amount of work distributed to each moderator continues to be enormous: they have to review thousands of posts each day, which leaves them only with a very tight window to decide whether a post should be removed or not – creating new issues around the accuracy and consistency of a company’s content moderation and impact on freedom of expression.
Indeed, ensuring the quality of content moderation is a challenge with important implications about the well-functioning of social media, freedom of expression, and fairness. Besides the very limited time frame for making moderation decisions, the quality of moderation can also be affected by individual biases, the AI tools deployed by platforms and the highly contextual nature of many posts, not to say the large amount of online content in the grey area between harmful and harmless. Apart from problems brought by online content, the complex constellation of laws, policies, platforms’ terms and conditions, and internal instructions also add difficulties for moderators to respond quickly and accurately.
The tech industry has already acknowledged these challenges. Several solutions exist to address these problems, but they still have considerable limitations. AI has been widely implemented in content moderation for both removing anything that is explicitly illegal, and for detecting suspicious content for human moderators to investigate. However, one salient drawback of AI is that it can only work on those “straightforward” cases covering broad categories, such as “nudity” or “blood”: for anything more nuanced, the current AI tools have proven to be prone to mistakes. For example, Thomas Jefferson’s words in the Declaration of Independence once got taken down automatically as “hate speech” because the phrase “Indian Savages” was flagged as inappropriate by the AI tool.
Another problem with current AI tools in content moderation is that most AI only works on text and visual-based content, while AI tools tailored for audio-based content moderation or in more interactive settings, such as live chat and live stream, are still in development. Furthermore, it has been established that AI tools often reflect the inherited biases of their creators, and for tools empowered by “black boxes”, their opaque decision-making processes may even create new problems in transparency auditing and quality assurance.
Providing mental health care for moderators is another important practice across companies. Wellness coaches and counselors are commonly seen in content moderation sites, as well as occasional employee support programs, but many moderators consider them inadequate and call for professional intervention from clinical psychiatrists and psychologists. “Wellness break” included in daily working hours is another expected buffer against deteriorating mental health, but it is also criticized for being too short compared to hours of exposure of traumatizing content.
There is still a lot that needs to be done to protect those who protect us from the worst aspects of the Internet. Possible improvements should be pursued in both technological and organizational solutions. Both the industry and academia have been working on improving the accuracy and efficiency of AI in automated detection and removal of harmful content. Apart from training smarter AI for more efficient automation, AI may also contribute in preventing or reducing exposure to disturbing content by interactively blurring them for human moderators.
Technology can also play a role in developing tools specialized in assisting content moderators in their work routines, in order to promote better task distribution across moderators, facilitate smoother internal communications for more complicated moderation decisions, and achieve more streamlined quality assurance of content moderation.
Companies should also assume more responsibilities in proactive protection of their workers’ mental wellness. For example, the tech industry can learn a lot from previous experiences from other high-risk jobs, such as the police, journalists, and child exploitation investigators. A first critical practice in these fields is to clearly inform employees and those who want to join the inherent risk of reviewing harmful content. Companies should also invest in building regular, long-term resilience training programs and hosting high-quality clinical mental health care teams in-house.
More importantly, there are strict maximum exposure times especially for those working in environments containing hazardous substances. Similar standards of maxim exposure time should apply to content moderation. Finally, across the nascent content moderation industry, building meaningful interpersonal networks among moderators can be valuable, fostering mutual support among “insiders” and eventually bringing the interests of content moderators to future agenda, who are crucial stakeholders of regulating the digital space.
With every third internet user being under the age of 18, online child sexual abuse has become a global public safety issue — producing a generation of victims. The WeProtect Global Alliance estimates that a staggering 54% of those who regularly used the internet as a child (now aged 18-20) were the victims of at least one online sexual harm. The stigma that still surrounds child sexual exploitation and abuse makes it likely that what we know is only the tip of the iceberg, and that our statistics underestimate the prevalence of the issue.
Though highly alarming, sexual exploitation and abuse are just one form of illegal or harmful content or conduct impacting young people online. Cyberbullying, impersonation, trolling, harassment, exposure to hate speech, encouraging self-harm, identity theft and phishing aimed at children are also on the rise. Consequences range from cautionary tales to harrowing tragedies. For example, Italy ordered TikTok to block anyone whose age could not be confirmed, following the death of a 10-year-old who attempted a dangerous challenge. We are also just learning that young people, regardless of gender, are susceptible to eating disorder trends that can be amplified by social media.
Read the full version of this article on the World Economic Forum.
By: Jacqueline Beauchere, Theos Evgeniou, Louis-Victor de Franssu
In a reversal of its long-held practice of “privacy first”, Apple announced in August 2021 that it would launch a new feature to scan images and videos on its devices in order to detect stored child sexual abuse material (CSAM). The policy shift epitomises the major changes happening today both in regulations and in businesses aiming at ensuring a responsible use of technology and a safe digital space. Yet, Apple’s new policy raised so many concerns from security and privacy experts that the company has delayed its plan.
The concept of a digital safe space is not limited to the proliferation of CSAM. Intermediary service providers, i.e. any firm that connects people, such as social media, marketplaces or online platforms for disseminating user generated content, face a growing number of abuses of their services. These include the spread of hate speech, terrorist content, illegal goods and services, spam and disinformation.
In fact, every year intermediary service providers around the world detect and remove billions of pieces of content from their platforms because the content is either illegal or contrary to their terms of service.
This affects small as well as giant platforms. Thousands of small online platforms have become home to a massive amount of illegal content posted by their users every month. Facebook identified more than 500 million pieces of such content in 2020 (1.3 billion, including spam) and spends hundreds of millions of dollars on content moderation. This content is so extreme and violent that people moderating it are reported to often suffer mental health issues.
Of course, the issue of illegal or harmful content did not appear with the rise of digital services. But the scale and speed at which such content can spread and be amplified by malicious actors who have become increasingly sophisticated, is worrying.
This has raised alarms for governments around the world which are designing new regulatory frameworks to mitigate some of these risks, with important implications not only for the future of society but also for the businesses they intend to regulate. However, achieving a safe digital space has and will continue to prove significantly challenging for regulators and companies alike.
Democratic governments attempting to regulate the online space must grapple with contradictory objectives. They need to balance between, on the one hand, keeping the internet safe by mandating platforms to prevent the spread of illegal content and, on the other, ensuring that fundamental human rights, including freedom of speech, are protected online.
With more than 95 million photos uploaded daily on Instagram, to name one platform giant, the sheer volume and potential for virality of content posted online makes ensuring judicial review prior to content removal nigh on impossible. Governments must therefore rely on setting out obligations for the private sector to moderate illegal content based on specific regulatory principles. But the more stringent the rules, the higher the risk of over-content removal and the more lenient the regulation, the higher the risks of illegal or harmful content spreading.
A related challenge for legislators is defining what effectively constitutes illegal content in a way that is broad enough to cover the targeted harms and specific enough to avoid the risks of censorship creep. Impractically broad definitions present serious risks for freedom of expression. Many worry that this difficulty could lead to political censorship in less democratic countries that would attempt to define rules without the proper safeguards.
Moreover, such regulatory definitions could leave substantial grey zones, requiring companies to decide on whether to remove content based solely on their discretion. This ambiguity combined with pressure on platforms to act as soon as such content is detected increases the risks of over-censorship, with important repercussions on freedom of expression online.
Another difficulty faced by regulators is how to implement effective obligations while ensuring competition within markets. This means finding the right balance between imposing minimum requirements for all related services without creating barriers to either innovation or market entry.
In an attempt to find fit-for-purpose solutions to these dilemmas, democratic governments and some of the largest digital services initially launched a series of self- and co-regulatory initiatives, like the Facebook white paper on regulation, or the EU Code of Conduct. Yet, outcomes were not always deemed sufficient by regulators which instead have started to develop new frameworks obliging online platforms to address detected illegal content or else face severe penalties.
In general, these new regulatory approaches can be divided into two broad categories: content-specific and systemic. The first consists of designing legislation to target a single specific type of online harm such as copyright infringements, terrorist content or CSAM and focuses on the effective and timely removal of that content. Examples of such regulations include the European Union’s Terrorist Content Online Regulation, the French law on disinformation, the German Network Enforcement Act (NetzDG) as well as the Directive on Copyright in the Digital Single Market.
In contrast, the systemic approach aims at providing a cross-harm legal framework whereby online companies must demonstrate that their policies, processes and systems are designed and implemented to counter the spread of illegal content on their platforms and mitigate potential abuses of their services while protecting the rights of their users. This is the direction proposed in the recent Online Safety Bill in the United Kingdom and the Digital Services Act (DSA) in the European Union.
In the case of the DSA for example, first presented by the Commission in December 2020, the legislators do not modify the existing liability regime, nor do they define illegal content online. Instead, the Commission sets new harmonised responsibilities and due diligence obligations for intermediary service providers: They must have in place processes and procedures to be able to either remove or disable content from their platforms when they find out that it is illegal. These regulations have implications for all intermediary service providers that go beyond potential large financial penalties.
Firms will need to move from the culture of “move fast and break things” to a more reasonable “move fast and be responsible” as they comply with complex cross-jurisdictional demands while maintaining customers’ trust. A shift towards a risk-based approach – already the path some regulators take, as the EU proposal on regulating AI indicates – requires organisational changes and the development of new risk management frameworks. Affected businesses need to understand the operational implications of the new regulatory obligations, assess their ability to comply and implement the appropriate risk mitigators.
Lessons from other sectors, such as finance, can prove useful. Much like in those sectors, online platforms will need to develop new policies and procedures, and then implement technical solutions. They will also need to create new roles and responsibilities, ultimately leading to organisational and cultural changes within their businesses.
First, companies, regardless of their size, will need to put processes in place to address the illegal content that they have been made aware of from a number of different sources, such as national competent authorities, the platform’s users or its internal moderation systems. They will also need to develop content moderation management processes and tools to ensure transparency, fairness, safety and compliance across different jurisdictions. These will unavoidably add cost and operational complexity for all online platforms.
For example, the European Commission estimates the annual cost of implementing and operating such tools, which includes content moderation management or transparency reporting workflows, can reach tens of millions annually for the larger players.
Second, new transparency requirements for online advertising call for online platforms to develop dedicated processes and tools to provide information to their users concerning the advertiser and their target audience. Additionally, providers of online marketplaces will also be required to enact Know your business customer policies and collect identification information from users operating on their platform. This obligation is largely inspired by similar requirements in the financial industry, adopted to limit the risks of money laundering.
And third, very large online platforms (VLOPs) will be subject to further requirements, including the obligation to conduct annual risk assessments on significant systemic risks stemming from the use of their services. These assessments will need to include risks related, for example, to the dissemination of illegal content through their services and the intentional manipulation of their platforms. While the EU Commission does not provide, at this stage, any advice on the risk assessment methodology, the DSA contains an initial list of potential risk-mitigation measures.
The development of an effective risk management framework will also require the set-up of a well-balanced enterprise organisation and risk culture, aligning compliance objectives with regulatory obligations, business and growth models, and reputation risk management. In fact, through the DSA, the European Commission will require that an organisation’s chief compliance officer has sufficient financial, technological and human resources as well as the adequate level of seniority to carry out the expected tasks. While these obligations target solely VLOPs, online platforms desiring to scale and expand their business across multiple jurisdictions within the EU will benefit from early adoption of such organisational structures.
Yet organisational changes will not be sufficient by themselves. As they grow, online platforms will need to move away from a Facebook culture to one of compliance where the firm’s systemic risks are understood and where employees are empowered to do the right thing.
Almost two decades after the first social media platforms arrived on the internet, revolutionising the ways human beings interact, communicate and do business, we have come to a bit of an impasse. The talk about regulating these businesses has amplified globally, especially given the potential impact social media can have on our political and socioeconomic systems. These platforms can become home to different communities but also targets of illegal content postings and coordinated attacks. The upcoming regulations under development across multiple jurisdictions will not change this but will force the digital industry to adapt to a new paradigm and to find innovative solutions to tackle harmful and illegal online content.
This is an adaptation of an article published in WEF Agenda.
François Candelon, Managing Director, the Boston Consulting Group; Louis-Victor de Franssu, CEO, Tremau; and Theodoros Evgeniou, INSEAD Professor of Decision Sciences and Technology Management | October 25, 2021
François Candelon is a Managing Director and Senior Partner at the Boston Consulting Group. He is also the Global Director of the BCG Henderson Institute.
Louis-Victor de Franssu (INSEAD MBA ‘18D) is a co-founder and CEO of Tremau.
Theodoros Evgeniou is a Professor of Decision Sciences and Technology Management at INSEAD.
INSEAD Knowledge is now on LinkedIn. Join the conversation today.
In recent years, a number of international organisations, regulators, governments, academics, and as well businesses have worked on developing principles of Artificial Intelligence (AI).
Alongside the development of these principles, there is an on-going discussion on how to regulate AI in order to best align risk management with optimising potential value creation of these technologies. Risk managing AI systems will likely become a regulatory and social expectations requirement, for all sectors and for both business and government.
However emphasis on how to implement the proposed AI principles and upcoming regulations in practice is more recent, and appropriate tools to achieve this still need to be identified and developed. For example, implementing so-called Responsible AI requires the development of new processes, frameworks and tools, among others. We review the current state and identify possible gaps.
Boza, Pal and Evgeniou, Theodoros, Implementing Ai Principles: Frameworks, Processes, and Tools (February 10, 2021). INSEAD Working Paper No. 2021/04/DSC/TOM, Available at SSRN.
By: Pal Boza and Theodoros Evgeniou
Artificial intelligence and machine learning (AI/ML) algorithms are increasingly developed in health care for diagnosis and treatment of a variety of medical conditions. However, despite the technical prowess of such systems, their adoption has been challenging, and whether and how much they will actually improve health care remains to be seen. A central reason for this is that the effectiveness of AI/ML-based medical devices depends largely on the behavioral characteristics of its users, who, for example, are often vulnerable to well-documented biases or algorithmic aversion.
Many stakeholders increasingly identify the so-called black-box nature of predictive algorithms as the core source of users’ skepticism, lack of trust, and slow uptake. As a result, lawmakers have been moving in the direction of requiring the availability of explanations for black-box algorithmic decisions. Indeed, a near-consensus is emerging in favor of explainable AI/ML among academics, governments, and civil society groups. Many are drawn to this approach to harness the accuracy benefits of noninterpretable AI/ML such as deep learning or neural nets while also supporting transparency, trust, and adoption. We argue that this consensus, at least as applied to health care, both overstates the benefits and undercounts the drawbacks of requiring black-box algorithms to be explainable.
Get full access to this article
By: Boris Babic, Sara Gerke, Theodoros Evgeniou and I. Glenn Cohen
Products and services that rely on machine learning—computer programs that constantly absorb new data and adapt their decisions in response—don’t always make ethical or accurate choices. Sometimes they cause investment losses, for instance, or biased hiring or car accidents. And as such offerings proliferate across markets, the companies creating them face major new risks. Executives need to understand and mitigate the technology’s potential downside.
Machine learning can go wrong in a number of ways. Because the systems make decisions based on probabilities, some errors are always possible. Their environments may evolve in unanticipated ways, creating disconnects between the data they were trained with and the data they’re currently fed. And their complexity can make it hard to determine whether or why they made a mistake.
A key question executives must answer is whether it’s better to allow smart offerings to continuously evolve or to “lock” their algorithms and periodically update them. In addition, every offering will need to be appropriately tested before and after rollout and regularly monitored to make sure it’s performing as intended.
Read the full version of this article on Harvard Business Review. The article was picked to HBR’s 10 Must Reads 2022.
by Boris Babic, I. Glenn Cohen, Theodoros Evgeniou, and Sara Gerke