Claude is an AI assistant developed by Anthropic for work involving text and code. According to its Cyber Verification Program (CVP) announcement, generally available models can help review code, patch known issues and triage security alerts; some cybersecurity operations are blocked.
CVP is an access arrangement that qualifying security professionals can apply for, rather than a separate chat model. It shows us that AI task outcomes depend on both capabilities and safety settings. Beginners can use this example to learn how to read tool descriptions and test figures.
Start by separating three questions
Capability: Does it know how to do the task? For example, can it understand a piece of code, identify potential vulnerabilities and suggest how to fix them?
Permission: Does the system allow it to proceed? Finding vulnerabilities can help people fix them, but can also help attackers. Anthropic therefore applies different safety restrictions to different scopes of work.
Reliability: Does the result hold up when checked? Even when a model provides a patch suggestion, someone still needs to check whether it is correct and whether it introduces new problems.
These are three ways to assess a tool, rather than a fixed internal process used by every AI product. A single refusal cannot establish that a model lacks the capability; a single answer cannot establish that its content is reliable. A platform allowing an operation also does not mean the user has obtained authorisation from the target system.
Three uses, each with necessary conditions
The following examples are based on the official scope of work. They are demonstrations, not hands-on tests conducted for this article.
Review code you manage. Developers can ask Claude to flag potential issues, explain the reasons and suggest patches. The output is a draft for human review and cannot be treated as proof that the code is secure.
Organise security alerts. A team can ask the model to summarise alerts and list missing information, then pass the results to staff for assessment. Before starting, confirm which data the organisation allows to be processed, and remove passwords and access credentials.
Conduct authorised penetration testing. This means simulating an attacker to find weaknesses in a system. The officially described Red Team Access tier covers this work, but the source explicitly states that the tier is for organisations only and requires authorisation to test the target systems. Actions that could cause physical harm or mass disruption remain subject to blocks.
General readers do not need to apply for CVP to understand these concepts. It is an access programme for professional security work and should not be treated as a feature available to every Claude account.
“50 trials” does not mean 50 types of work
According to the official testing description provided, Anthropic evaluated Claude Opus 5.5 on CyScenarioBench under different CVP safety settings. This evaluation requires models to plan and execute cyber operations involving multiple steps under specified constraints.
Each access tier was tested on ten challenges, with five attempts per challenge, making fifty trials in total. The company reports that without CVP access, every task was blocked on the first prompt. Under the Defense Access settings, forty-six of the fifty trials were blocked at some point, while the remaining four succeeded.
Keep two distinctions in mind when reading these figures. First, the fifty trials include repeated challenges, so they cannot be described as tests of fifty different types of work. Second, “blocked” describes an intervention by safety restrictions and cannot simply be rewritten as “the model gave a wrong answer”.
These results support the conclusion that different safety settings affect how these tasks proceed. They do not represent Claude’s overall accuracy or predict its success rate when you organise documents. The figures were reported by Anthropic; this article did not repeat the tests.
Third-party involvement still calls for reading the actual report
In its update on independent evaluators, Anthropic says it has given a team from the nonprofit evaluation organisation METR access roughly comparable to that of its internal risk assessment teams.
The same announcement also describes limits: some information is excluded. Anthropic cannot veto publication, but it can require the removal of confidential information in predefined categories, and the company decides whether information falls within those categories.
When you read that a product has undergone “third-party evaluation”, look for the evaluator’s published report. Check what was tested, what information the evaluators could access and what limitations apply to their conclusions. This description of the partnership alone is insufficient to establish that the CVP figures above have been independently verified.
Practical exercise: break a promotional claim down into evidence
Open the CVP source linked above, find the testing section for Defense Access and write down four things:
- Who ran the tests? Anthropic, rather than the author of this article.
- What was tested? Ten cyber challenges involving multiple steps, with five attempts each.
- Under what conditions? Claude Opus 5.5 with specified safety settings.
- What cannot be inferred? Overall accuracy or refusal rates for everyday work.
At work, you can use the same approach to request evidence when someone introduces a tool by saying, “This AI performs very well.” For example, when assessing a document summarisation tool before procurement, first look for tests relevant to document summarisation. Then try a short passage that can be shared publicly and whose expected summary you already know, checking each omission and error. This is an evaluation method recommended in this article, rather than CVP’s testing procedure.
When a task fails, first record whether the tool refused to proceed, lacked information or produced an incorrect result. Describing the problem clearly makes it easier to decide whether to provide more information, adjust the scope of work or use another method.
To practise assessing tools, asking clear questions and verifying results through everyday tasks, explore Fluent AI HK’s free twelve-chapter AI course. This article was checked on 30 October 2026; it does not claim that our company repeated the security tests described above.

Join the discussion
CommunitySign in to share an idea or question. Comments appear after review.
Sign in to comment