Your internal AI assistant answers questions correctly during the demonstration. It also reads documents, remembers conversations and can update a customer record through a connected tool. Before you give it to staff, someone asks whether it has been tested. A handful of good answers is not enough to answer that question.
For organisations in Luxembourg and the Greater Region, a useful AI assessment needs to cover both security and intended performance. AI pentesting investigates exploitable weaknesses. AI testing measures whether the system does its job reliably. This guide explains how to scope both, what evidence to request and where models, agent harnesses, MCP integrations and bias fit into the work.
AI pentesting and AI testing answer different questions
A pentest asks what an attacker could make the system reveal or do. A quality evaluation asks whether it produces an acceptable result under the conditions your users will encounter. A system can perform well on routine questions while exposing another user's documents. It can also resist a jailbreak while confidently inventing an answer to a customer.
| Assessment | Main question | Useful evidence |
|---|
| AI pentesting | Can an attacker cross a security boundary? | Reproducible attacks, affected permissions and business impact |
| AI testing | Does the system perform its intended task? | Representative test cases, measured results and acceptance criteria |
| Conformity readiness | Which applicable requirements have supporting evidence? | Control mapping, technical records, gaps and remediation ownership |
Our AI pentesting service covers the security boundaries of the complete application. It can be combined with quality and bias evaluations when the engagement needs a broader assessment. Keep the objectives distinct in the statement of work so neither disappears into a generic promise to test the model.
Define the system boundary before selecting a tool
Start with a diagram showing the user, application, model, document sources, memory stores, tools and downstream systems. Mark where identities change and where information moves between organisations. Include the administrative interface, background workers and monitoring pipeline; they may have access that the visible chatbot does not.
Then name the intended users, permitted tasks and prohibited actions. An assistant that drafts a purchase request has a different risk profile from one that submits the purchase. Write down whether it may send email, modify records, run code or access documents across departments. These decisions create the boundaries the assessment should try to cross.
Agree on authorisation, permitted test environments, third-party restrictions, synthetic data, escalation contacts and stop conditions. Use controlled destinations for any exfiltration simulation. If a production dependency must be involved, define the allowed actions explicitly rather than assuming that permission to test your application covers somebody else's service.
Consider an illustrative support assistant that can prepare an email but cannot send it. The assessment should check the draft, the proposed recipient and the execution record separately. If the conversation says that an email was sent, establish whether a real tool call occurred. If execution was blocked, retain that evidence too. This avoids confusing an invented success message with a completed action and identifies which component needs a correction.
Test the model through realistic entry points
Model testing should include the languages, file types and interaction patterns your application actually supports. For a multilingual Luxembourg workflow, that may mean more than translating the same English prompt. Different document formats, mixed-language messages and domain vocabulary can change what the system understands and how its safeguards behave.
OWASP distinguishes direct and indirect prompt injection: an instruction can arrive through user input or through material the application retrieves. That distinction helps choose realistic entry points instead of testing only the chat box.
For each test, record the application and model versions, prompts, supplied context, configuration and observed result. Distinguish an offensive response from an actual breach of a defined boundary. Revealing a harmless system instruction and disclosing customer data have different consequences, even if both look dramatic in a screenshot.
A generic automated scanner is useful for breadth and repeatability. It still needs application-specific cases: what data exists, which tools are available and what the business regards as an unacceptable action. A clean scanner run is evidence about those tests, not a guarantee that every interaction is safe.
Assess the agent harness and its action controls
The agent harness is the application layer that supplies context, manages memory, selects tools and executes actions around a model. This is where a plausible suggestion can become a real database update. Review the path from model output to execution and identify which controls are enforced by code.
Use a synthetic business workflow to test approvals. For example, an assistant may prepare a customer-record change but require a human to approve it. Confirm that the approval is tied to the exact action and arguments, expires appropriately and cannot be reused for a different operation. Check the result after execution rather than accepting the assistant's description of what happened.
Exercise cancellations, retries, tool failures and session changes. A cancelled approval should remain cancelled after a retry. A failed lookup should not silently broaden access. Conversation memory should not transfer sensitive material to another user. Bound repeated tool calls so an unproductive agent loop cannot consume an unlimited operating budget.
Include MCP servers, clients and downstream permissions
Model Context Protocol, or MCP, connects AI applications to tools and resources. Inventory each client and server, how it is hosted, which identity it uses and which downstream permissions it holds. A useful test target is the complete path from a user request to the final API call.
The official MCP security guidance covers protocol-specific risks such as token passthrough and confused-deputy behaviour. Review the guidance against the transport and authorisation flow your implementation actually uses.
Build explicit positive and negative cases. A permitted user should be able to perform the approved task. A different user, an expired credential or a request outside the authorised scope should fail without exposing extra data. Review tool descriptions, results, local execution and secret handling alongside server authentication.
Do not treat a trusted connector as proof that everything it returns is trustworthy instruction. A support ticket or document can contain hostile text even when retrieved through an authenticated service. Test how that material affects the subsequent action, using synthetic records and harmless test destinations.
Evaluate retrieval, quality and bias separately
A retrieval-augmented generation system, or RAG system, should be assessed at both retrieval and answer stages. Check whether the permitted documents are found, whether disallowed documents are excluded and whether the answer faithfully uses the retrieved material. A correct-looking answer can hide a failed access filter.
For quality testing, create a reviewed dataset from realistic tasks. Include answerable questions, questions for which the knowledge base has no answer, conflicting sources, outdated documents and requests that require a human. Keep expected outcomes and scoring rules independent of the system being assessed.
Bias evaluations need a stated context. Define which user groups, languages or circumstances matter for the intended use, and what an unfair difference would mean. Where appropriate, compare otherwise equivalent cases while changing one relevant attribute. Record uncertainty and sample limitations rather than turning a small test into a universal fairness claim.
The NIST Generative AI Profile is a voluntary resource for managing AI risk across the lifecycle. It can help organise an evaluation programme; your release criteria still need to reflect your actual workflow. Our AI testing service turns those criteria into repeatable evaluations and regression checks.
Connect testing evidence to EU AI Act work
Technical testing contributes to regulatory readiness, but a report needs the correct context. Establish the intended use, your provider or deployer role and the applicable requirements before deciding what evidence to collect. A chatbot, a recruitment system and a model integrated into another regulated product need different assessments.
Article 15 of the EU AI Act addresses accuracy, robustness and cybersecurity for high-risk AI systems. Map relevant test results to applicable obligations, with their scope and limitations visible. An isolated prompt test cannot stand in for the wider conformity process.
Keep evidence about datasets, human oversight, logging and remediation together with the technical findings. Our EU AI Act guide explains the wider compliance context. The assessment should identify gaps and responsible owners, without describing a pentest or bias score as a certificate of legal conformity.
A useful test record links one case to an expected outcome, the observed result and the supporting trace. Store sensitive evidence with controlled access and redact copies shared more widely. Attach the remediation owner and retest result to the same record. This lets a release reviewer distinguish an open issue, an accepted limitation and a verified correction without reconstructing the assessment from chat transcripts.
Request a report your team can act on
Each security finding should identify the affected component, prerequisites, reproduction steps, observed impact, severity rationale and proposed correction. Quality findings should show the test population, scoring method and failed acceptance criteria. Both should identify untested areas and dependence on a particular model or configuration.
Agree on retesting and keep the regression cases. Changing a prompt, model, document index or MCP tool can alter behaviour even when the user interface stays the same. The deployment and administration checklist explains how those tests become part of ongoing operation.
If you need to assess an AI system, bring its architecture, permitted actions and deployment stage to a scoping discussion with ObsidianCorps. We can define a focused pentest, a quality and bias evaluation, or a combined assessment with clear evidence and remediation deliverables.
