How Meta decides when AI can remove posts and when humans step in

Meta's automated systems do not send every flagged post to a human reviewer at the same stage. Its technology helps determine which content should be reviewed by people and in what order

The government on Thursday asked Meta to ensure that content is not taken down solely based on decisions made by artificial intelligence (AI), amid concerns that automated moderation can remove posts without considering their wider context.

 

The issue came up during meetings between senior officials from the Ministry of Electronics and Information Technology (Meity) and Meta executives between August 5 and August 10.

 

The discussions followed the temporary removal of a Facebook and Instagram post by Prime Minister Narendra Modi during the student protests in Delhi. Meta's global affairs chief Joel Kaplan apologised to Electronics and Information Technology Minister Ashwini Vaishnaw over the restriction.

 

 

During the meetings, Meity officials questioned Meta over the role of human reviewers in its AI-enabled content moderation system. Government officials, cited by Business Standard in a report published on Thursday, said the company's systems appeared to rely on “static” keywords to identify and remove content across its platforms.

 

One example discussed was a post by a government department about the extradition of pro-Khalistan fugitives from certain countries. The post was removed after Meta's AI-enabled moderation system detected the term “pro-Khalistan”, a government official present at the meetings told Business Standard.

 

“Without the AI system understanding the context,” the official said, the post was taken down because of the presence of the term. The official said a human reviewer could have prevented the error.

 

Meta, however, says its content moderation systems are trained to account for context when identifying potential violations.

How Meta's moderation system works

Meta, in its blog explaining explanation of its review process, said the process combines automated technology with human review. Its published material said AI is used to detect potentially violating content, make some enforcement decisions and prioritise cases for human reviewers.

 

Meta said that when its systems are highly confident that content violates its rules, they can remove it automatically. Where there is less certainty, the content can be prioritised for review by human teams.

 

“In an instance where our systems are near-certain that content is breaking our rules, it may remove it,” Meta said in the 2020 explainer, adding, “Where there is less certainty it will prioritize the content for teams to review.”

 

This means a human reviewer does not necessarily make every content-removal decision under Meta's stated process. The company's technology can take enforcement action when it determines that the likelihood of a violation is sufficiently high.

How AI identifies potentially violating content

Meta said its automated systems are trained to identify content that may violate its Community Standards. According to the company, the process does not rely only on a fixed list of keywords; instead, its technology is trained to assess different elements of a post and determine whether they indicate a violation.

 

This becomes more difficult when the meaning of content depends on context. In its blog explaining its Transparency Center, Meta said the meaning of some content can be immediately clear to a person but harder for technology to understand.

 

A post may contain text, an image or other elements that have different meanings when considered separately.

 

Meta said it uses training systems that learn from millions of pieces of current content. These systems help select training data that is aligned with the company's objectives. Meta said this approach is used to improve its ability to detect areas such as hate speech and content that incites violence.

How does Meta identify hate speech?

The difficulty of identifying violations is particularly huge in matters involving hate speech, according to Meta's blog titled 'Detecting Violations'.

 

The company has said that detecting such content requires technology to distinguish between words used in a prohibited context and the same words used in a different context. This is one reason Meta trains its systems to assess the broader content of a post rather than relying only on individual terms.

 

Meta's Community Standards define what content is prohibited on its platforms. The standards cover categories including violence and criminal behaviour, dangerous organisations and individuals, safety, objectionable content, integrity and authenticity, and privacy.

 

Once its systems identify content that may violate these rules, Meta can take enforcement action or route the case for further review. The company has said its technology is used both to detect potential violations and to determine which cases should receive additional review.

When humans enter the process

Meta has said that human reviewers are used for cases where context is important or where automated systems are less certain about a potential violation.  The company identifies three main factors: the severity of potential harm, how quickly the content is spreading, and the likelihood that it violates its policies.   

Severity refers to the potential for harm online or offline. Virality refers to how quickly content is being shared. The likelihood factor considers how likely it is that the content actually violates Meta's policies.

 

Meta said its systems use these factors to prioritise content that could cause greater harm or is spreading quickly. This means the order in which human reviewers examine content is also determined with the help of automated systems.

 

Human review is also used for cases where context is difficult to assess. Meta said in 2020 that some moderation decisions can be “extremely nuanced and contextual”. It gave bullying as an example because a reviewer may need to determine who is being targeted and whether the content is intended to cause harm 

 

Human reviewers also handle some appeals against enforcement decisions. Meta said its review teams spend more time on user appeals and that their reviews also help provide training data for its automated systems 

 

Meta has also said that language and regional expertise are part of its review process. In 2023, the company said dedicated teams reviewed Facebook and Instagram content in more than 70 languages. It said local understanding can be important when assessing the meaning of words and the context of posts. 

 

Meta said its technology is therefore trained to first examine the different components of a post and then determine its meaning. This includes looking at text and images together rather than assessing them independently.

 

In its 2020 explanation, the company said some moderation decisions can be “extremely nuanced and contextual”. It cited bullying as an example because reviewers may need to establish who is being targeted and whether the content is intended to cause harm.

 

Human reviewers also handle appeals against some enforcement decisions. Meta has said that feedback from human reviews can be used to improve its automated systems.

 

Meta has also said that its review teams have language and regional expertise. In 2023, the company said dedicated teams reviewed Facebook and Instagram content in more than 70 languages and that local understanding can matter when assessing the meaning of words and the context of posts.