What is Red Teaming?

Red teaming is the practice of testing a system with an adversarial approach to assess that system's defenses. The goal of red teaming is to enhance the security of those systems by identifying potential vulnerabilities and assessing the effectiveness of existing defenses.

What is AI Red Teaming?

In the context of systems that leverage LLMs (Large Language Models), AI red teaming plays a pivotal role in assessing not only the security of the system, but also the safety of that system. Due to the indeterminism of LLMs and their interface being natural language, the possible inputs and outputs of any system that leverages an LLM are vast, including content that is toxic, biased, inappropriate, or otherwise harmful. This makes it especially important to incorporate adequate defenses in such a system and to assess those defenses on a regular basis.

Typically, AI red teaming is done by a team of experts that emulate potential adversary attacks. For an LLM application, however, effective AI red teaming is difficult to scale due to:

1. The vast amount of inputs and outputs possible from such a system. In other words, there are a vast number of potential attacks to emulate.
2. AI red teaming experts are hard to come by and can be expensive.
3. AI red teaming LLMs or LLM applications can involve a lot of toxic content which can be particularly taxing to the humans necessary to generate and assess such content.

However, we can alleviate a lot of those scaling concerns by leveraging automated (AI) red teaming.

What is Automated Red Teaming?

Automated red teaming is a system that emulates adversarial attacks against another system.

What is Automated AI Red Teaming?

In the context of red teaming LLM applications, an automated AI red teaming system may incorporate its own LLM, amongst other methods, to ensure emulation of efficient, diverse, and thorough adversarial attacks at scale against the target LLM system.

In revisiting the problems raised above, we can better understand how automated AI red teaming helps address scalability of AI red teaming efforts and ensures more robust assessment of LLM system defenses.

The vast amount of inputs and outputs possible from such a system. In other words, there are a vast number of potential attacks to emulate.

Leveraging an LLM in an automated red teaming system can much more efficiently generate attacks across a wide array of potential attack vectors, especially when done so with state-of-the-art prompt engineering techniques.

Red teaming experts are hard to come by and can be expensive.

The cost of LLM inference (the main cost consideration for using an automated AI red teaming system) is magnitudes lower compared to the cost of a team of AI red teaming experts. This is partially due to the amount of time necessary for that team of AI red teaming experts to generate the same number and quality of attacks as an automated AI red teaming system, which may be days versus minutes.

Red teaming LLMs or LLM applications can involve a lot of toxic content which can be particularly taxing to the humans necessary to generate and assess such content.

By leveraging an LLM in an automated red teaming system, we leverage the significant amount of toxic content LLMs were trained on to generate such attacks. This alleviates a lot of the mental toll on the humans that are typically necessary for that task.

What is Fortify?

Fortify is an automated AI red teaming tool. It automates safety and security vulnerability detection in AI systems by simulating real-world attacks at scale. Fortify is built on and kept up-to-date with rigorous data science research and the latest, state-of-the-art industry methods and techniques. It can be employed by non-technical users such as product managers and analysts to identify vulnerabilities in GenAI systems or to supplement and accelerate the work of red teaming professionals.

Roles

Admin

Admins have access to all options in Fortify:

Team Lead

Team Leads have access with slight restrictions on permissions in Fortify:

User

The Users have limited access and permissions in Fortify: