Azure OpenAI Content Filters: The Good, The Bad, and The Workarounds

If you regularly use the Azure OpenAI LLMs, you might have noticed two peculiar things: First, API calls for models on Azure sometimes take notably longer than for the exact same model on the 'ordinary' OpenAI platform. Second, you occasionally receive cryptic error messages about content filtering, even for seemingly innocent business content.

These behaviors aren't bugs - they're the result of Azure OpenAI's built-in content filtering system.

By default, every request to Azure OpenAI passes through multiple content classification models that analyze both your input and the model's output for potentially harmful content. These additional safety layers explain both the increased latency and the occasional unexpected rejections.

While content filtering serves an important purpose in preventing misuse, its default implementation can be challenging or even outright outrageous in business scenarios. Many developers find themselves wondering:

• Why are some perfectly reasonable business prompts getting rejected?

• Why do API calls take longer than with regular OpenAI?

• How can we optimize this for production use?

In this technical guide, we'll dive deep into Azure OpenAI's content filtering system. We'll examine what these filters actually do, why they might be overly aggressive for your use case, and how their synchronous nature impacts performance. Most importantly, we'll explore practical solutions - from configuring appropriate filter levels to setting asynchronous processing for better performance.

Azure OpenAI Content Filters are a set of AI classification models that run alongside the main language models (like GPT-4 or GPT-3.5). These filters examine both the input (your prompts) and output (the model's responses) for potentially harmful or inappropriate content.

The filtering system consists of several components:

• Hate and Fairness: Detects discriminatory language or attacks based on identity characteristics
• Sexual Content: Identifies explicit sexual content or inappropriate material
• Violence: Detects violent content or threats
• Self-Harm: Identifies content related to self-injury or suicide

• Protected Material: Identifies copyrighted content (like song lyrics) or code from public repositories
• Prompt Shield: Detects attempts to bypass the model's safety measures
• Groundedness: Checks if responses are based on provided source materials (only available in specific regions)

Each category operates on four severity levels:

• Safe: Content is acceptable
• Low: Mild concerns
• Medium: Moderate concerns
• High: Serious concerns

By default, these filters run synchronously, meaning:

1. Your prompt is first analyzed by the content filters
2. If approved, it's sent to the language model
3. The model's response is then analyzed by the filters - again
4. Only after passing all checks is the content returned to you

This sequential process explains why Azure OpenAI calls often take longer than regular OpenAI calls - you're actually interacting with multiple AI models for each request, not just the language model itself.

While this comprehensive safety system helps prevent misuse, it can be overzealous in business contexts. For example, a prompt discussing "database killing" in the context of IT operations might get flagged for violence, or a healthcare application discussing medical conditions might trigger self-harm filters.

That being said: How can you optimize content filtering for your use case?

The official answer: analyze your use case and adjust the filter severity levels appropriately. Azure OpenAI allows you to set different severity levels for each content category, from 'Safe' to 'High'. Then test your application, implement logging, and find where the filters might still be too restrictive.

The inofficial answer: These filters are a pain to work with and they rarely provide any meaningful benefit, with one exception: the 'Prompt Shield'. Therefore, for the time being our suggested strategy is to crank all filers to the minimum filter level (so that only the most 'offensive' prompts and answers are blocked), but keep the 'Prompt Shield' on.

Last but not least, they also slow down the whole inference pipeline, ruining performance of your applications. Thankfully we can also tweak this knob a bit: Instead of running these filters synchronously, we can set them to asynchronous mode. In this mode, they run in parallel while the answer of the LLM is streamed back to your application. This mostly speeds up the whole process quite significantly. Furthermore, as the answer is given WHILE the content filter evaluates, you have the opportunity to simply disregard it's results, meaning you have another option to turn down it's impact.

To explain our reservation against these filters: These filters block too many legitimate prompts and answers in business contexts - where you often simply don't need the AI safety/alignment/ethics checks. For example our prompts about electrical engineering guidelines got blocked due to 'self harm' concerns (I guess the filter models thought we're using current to kill ourselves). This over-blocking is critical for setting up repeatable and testable AI applications, as there is another highly probabilistic variable one needs to account for.

The prompt shield filter on the other side is - according to our tests and experience - useful as detecting prompt injection is quite hard and also leads to problems in business applications. At the same time we never got a false-positive from this filter - so let's keep this one on.

Before we continue...

Interested in how to train your very own Large Language Model?

We prepared a well-researched guide for how to use the latest advancements in Open Source technology to fine-tune your own LLM. This has many advantages like:

• Cost control
• Data privacy
• Excellent performance - adjusted specifically for your intended use

I hope our reasoning above convinced you to set the filters to the lowest severity threshold and set them to asynchronous mode. Here's how you can do just that:

1. Navigate to the Azure OpenAI Studio. There, select the Azure OpenAI resource you want to configure. Or create a new resource.

   Azure OpenAI Studio - select resource

2. In the resource dashboard, in the left-handside-menu select "Safety + Security". Click on "Create Content Filter".

   Azure OpenAI Studio - create content filter

3. Enter a name and click on "Next".

4. In the next screen, you can adjust the severity of the 'Input filters' - meaning the filters that run on your prompts. Set all the sliders to "High". Leave the "Prompt shield" at "Annotate and block".

   Azure OpenAI Studio - input filters

5. Click on "Next". Now more or less the same user interface appears, but this time for the "output filters" - the filters running on the model's output. Set all the sliders to "High" again and set the other settings to "Off".

   Azure OpenAI Studio - output filters

6. Don't click on "Next" yet! Scroll down a bit and make sure to set "Streaming mode" to "Asynchronous Filters".

   Azure OpenAI Studio - streaming mode

7. Click on "Next". In this final settings screen, select your model deployments for which you want to select this newly created filters. If you don't have any deployments yet, you can skip this step. Click on "Next" and then "Create".

   Azure OpenAI Studio - deployment settings

That's it - your Azure OpenAI models now will already use the new, much faster, less hindering content filters.

To use this new filter for deployments not already existing, simply select the filter during deployment creation.

Azure OpenAI Studio - select filter during deployment creation

• How to test your RAG pipeline?
• Advanced RAG: Recursive Retrieval with llamaindex
• DSPy: Automatic prompt tuning and creating
• Free and open source ChatGPT alternative: LibreChat