Policy-As-Prompt: Rethinking Content Moderation in the Age of Large Language Models

Authors’ abstract

Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models KONSTANTINA PALLA, Spotify, UK JOSÉ LUIS REDONDO GARCÍA, Spotify, Spain CLAUDIA HAUFF, Spotify, Netherlands FRANCESCO FABBRI, Spotify, Spain HENRIK LINDSTRÖM, Spotify, Sweden DANIEL R. TABER, Spotify, US ANDREAS DAMIANOU, Spotify, UK MOUNIA LALMAS, Spotify, UK Content moderation plays a critical role in shaping safe and inclusive online environments, balancing platform standards, user expectations, and regulatory frameworks. Traditionally, this process involves operationalising policies into guidelines, which are then used by downstream human moderators for enforcement, or to further annotate datasets for training machine learning moderation models. However, recent advancements in large language models (LLMs) are transforming this landscape. These models can now interpret policies directly as textual inputs, eliminating the need for extensive data curation. This approach offers unprecedented flexibility, as moderation can be dynamically adjusted through natural language interactions. This paradigm shift raises important questions about how policies are operationalized and the implications for content moderation practices. In this paper, we formalise the emerging policy-as-prompt framework and identify five key challenges across four domains: Technical Implementation (1. translating policy to prompts, 2. sensitivity to prompt structure and formatting), Sociotechnical (3. the risk of technological determinism in policy formation), Organisational (4. evolving roles between policy and machine learning teams), and Governance (5. model governance and accountability). Through analysing these challenges across technical, sociotechnical, organisational, and governance dimensions, we discuss potential mitigation approaches. This research provides actionable insights for practitioners and lays the groundwork for future exploration of scalable and adaptive content moderation systems in digital ecosystems.

Paper

Authors

K. Palla, J.L.R. García, C. Hauff, F. Fabbri, A. Damianou, H. Lindström, D. Taber, M. Lalmas

Publication date

2025

Policy-as-Prompt

Rethinking Content Moderation in the Age of Large Language Models KONSTANTINA PALLA, Spotify, UK JOSÉ LUIS REDONDO GARCÍA, Spotify, Spain CLAUDIA HAUFF, Spotify, Netherlands FRANCESCO FABBRI, Spotify, Spain HENRIK LINDSTRÖM, Spotify, Sweden DANIEL R. TABER, Spotify, US ANDREAS DAMIANOU, Spotify, UK MOUNIA LALMAS, Spotify, UK Content moderation plays a critical role in shaping safe and inclusive online environments, balancing platform standards, user expectations, and regulatory frameworks. Traditionally, this process involves operationalising policies into guidelines, which are then used by downstream human moderators for enforcement, or to further annotate datasets for training machine learning moderation models. However, recent advancements in large language models (LLMs) are transforming this landscape. These models can now interpret policies directly as textual inputs, eliminating the need for extensive data curation. This approach offers unprecedented flexibility, as moderation can be dynamically adjusted through natural language interactions. This paradigm shift raises important questions about how policies are operationalized and the implications for content moderation practices. In this paper, we formalise the emerging policy-as-prompt framework and identify five key challenges across four domains: Technical Implementation (1. translating policy to prompts, 2. sensitivity to prompt structure and formatting), Sociotechnical (3. the risk of technological determinism in policy formation), Organisational (4. evolving roles between policy and machine learning teams), and Governance (5. model governance and accountability). Through analysing these challenges across technical, sociotechnical, organisational, and governance dimensions, we discuss potential mitigation approaches. This research provides actionable insights for practitioners and lays the groundwork for future exploration of scalable and adaptive content moderation systems in digital ecosystems.

Read the original paper

Transcript

A transcript has not been published for this episode yet.

Licensing

Source license

Creative Commons Attribution 4.0 International

Recording license

CC-BY-4.0

The authors and publisher do not sponsor or endorse this recording.