Skip to main content
Try Meta AI

Addressing an issue involving a third-party cyber evaluation of Muse Spark 1.1

5 minute read

A third-party model evaluator notified us of a cybersecurity issue while evaluating an earlier generation of our large language model, Muse Spark 1.1 - due to a misconfiguration. Following that disclosure, we completed a detailed retrospective to understand the incident and its implications. This post provides more information on what we learned.

Our cybersecurity evaluations are designed to measure a model’s capabilities by testing against specific threat scenarios, combining public and internal benchmarks with independent third-party testing. Some of these evaluations are conducted without the standard safeguard we deploy in production in order to measure the model’s underlying capabilities. This is one component of a broader evaluation process we conduct before releasing or deploying a model. Testing AI models for cybersecurity capabilities is essential to understanding what they can do before deployment.

What happened

We contracted with Irregular to conduct cybersecurity evaluations of pre-released models. As part of that work, we asked them to test a pre-released version of Muse Spark 1.1, according to our normal testing protocol. In early July, Irregular began an exercise to evaluate, in a closed testing environment with safeguards removed, whether our model would be capable of completing an adversarial cybersecurity task. However, when Irregular set up the testing environment, a misconfiguration allowed the model to access the open internet, and instead of using a fictional name of the “target” of the fictional exercise Irregular unintentionally provided the model with a name of a real website as its target.

Believing the real website was the intended target, the pre-release version of Muse Spark 1.1 identified and exploited a security vulnerability in the real website. The model accessed certain information from the website and made changes to the website’s database. Meta provides model access via API and the evaluation ran entirely on Irregular’s infrastructure, therefore we have limited information related to the third party company and are taking steps to ensure that their data is not on our systems. Several other companies’ AI models were being evaluated by Irregular around the same time and exhibited similar behavior. The misconfiguration issues were contained, Irregular disabled the affected evaluation, and notified us as well as ensured that the affected party was also notified. We appreciate Irregular’s prompt disclosure of this issue and their ongoing partnership.

What we found

Once Irregular identified the issue, they disabled the affected evaluation. We then began our own independent investigation. Our security teams reviewed over 10,000 records of Muse Spark 1.1’s activity during testing and conducted deeper analysis to understand the full scope of what occurred.

Key findings include:

  • No other instances of the model exploiting a third-party company’s system were identified beyond this evaluation, proving the isolated nature of this incident.
  • The model operated within the scope of its assigned task based on the instructions it was given and the environment it encountered and this was not a sophisticated offensive cyber attack or sandbox escape.
  • Our review also surfaced additional areas for improvement in the testing environment and the integration with it, which we shared with Irregular to support remediation efforts.
  • We’ve also identified monitoring improvements that would help identify these types of issues earlier in the future.

Strengthening evaluation security

Testing AI models for cybersecurity capabilities is essential to understanding what they can do before deployment. We conduct these evaluations as part of our broader preparedness approach, which includes multiple layers of internal and external testing before a model is released or deployed. Our published evaluation reports detail the stress-testing we conduct across a range of areas, including cybersecurity and model alignment. But as models become more capable, these evaluations surface a specific challenge: models that demonstrate the ability to find and exploit vulnerabilities require proportionally stronger containment during testing. This is an industry-wide challenge that labs are actively working to address, including through improved containment during testing, better separation between evaluation and production environments, and ongoing research into how to evaluate safely without reinforcing unwanted behaviors in the model.

This issue reinforces the importance of appropriate guardrails and coordination around the use of the internet in these environments. Irregular has confirmed the misconfiguration has been corrected and that evaluations do not reference real website names. Going forward, we are implementing independent verification requirements for test environment isolation and scenario review before evaluations begin and that test scenarios do not reference real companies. More broadly, we continue to invest in our evaluation and preparedness infrastructure. As model capabilities advance, evaluation environments and practices need to keep pace, both within our own infrastructure and across the third-party partners we work with.

Looking ahead

We’re contributing to industry-wide efforts to help improve security and evaluation safety practices, model alignment, and how models behave in adversarial testing scenarios. We plan to continue to strengthen how evaluations are conducted in a secure manner going forward, together with our partners. We appreciate Irregular’s partnership and look forward to continued efforts to work closely with them on security.