Government leaders are being asked to approve AI systems that interact with residents, handle sensitive information, and support essential services. They remain accountable for the outcomes of those systems, even when the underlying technology operates at a level of complexity no single person can fully inspect.
That reality places new demands on security, governance, and risk management. Organizations need clear evidence that these systems will protect data and behave as intended under real-world conditions.
Every CIO, CTO, and chief AI officer needs a practical way to evaluate that evidence.
A useful framework starts with two questions: Is the application built securely? How does it behave when someone tries to manipulate it?
Our new technical paper, GovSecure-LLM: A Security Scanner for Large Language Model Applications in Government Contexts, documents the framework our team developed to answer both questions. The research combines analysis of an application’s underlying design with live adversarial testing of its behavior. Together, these methods give government technology leaders a clearer basis for evaluating risk, controls, and deployment readiness.
AI security extends across the application
Large language models introduce security risks tied to their probabilistic behavior, emergent capabilities, and interactions with user input. Prompt injection, system prompt leakage, retrieval attacks, data poisoning, and excessive agency can influence what a system reveals, produces, or does. Evaluating these risks requires testing designed around the behavior and architecture of AI applications.
The model represents one component of that architecture. Security risks can originate in prompts, retrieval pipelines, vector databases, integrations, logging systems, access controls, and downstream tools. A retrieval-augmented generation system, for example, depends on the integrity of its source content, the effectiveness of its tenant isolation, the configuration of its vector queries, and the controls governing how retrieved information reaches the model.
These dependencies carry significant consequences in government. AI applications can support benefits inquiries, permit applications, public records requests, and other services involving sensitive information and decisions that affect residents. A weakness in the surrounding architecture can expose data or shape system behavior even when the underlying model operates as designed.
This broader view helps leaders ask better questions of internal teams and technology partners. Security depends on the model, the application around it, the information it can access, and the actions it can take. Each layer needs controls, testing, and evidence.
Two questions create a clearer view of risk
The first question, “Is the application built securely?” focuses on the system’s underlying design.
Security teams answer it through static application security testing (SAST), which reviews source code, configurations, dependencies, and architecture. The process resembles inspecting a building’s plans, materials, and electrical systems before people move in. It can identify missing safeguards, weak access controls, unsafe data-handling practices, incomplete logging, and other conditions that could create risk during operation.
The second question, “How does the application behave under attack?” focuses on the running system.
Security teams answer it through dynamic application security testing (DAST). Testers interact with the application using prompts designed to simulate realistic attacks. They may attempt to override instructions, extract protected information, manipulate retrieved content, impersonate an authority, or induce the system to take an inappropriate action. The application’s responses show how its safeguards perform under pressure.
The two methods produce different forms of evidence. Static analysis identifies where risk can emerge. Dynamic testing reveals how the deployed application responds when someone targets those areas. Leaders gain a fuller picture by considering both forms together.
Our GovSecure-LLM evaluation demonstrated the value of this combined view. Static analysis of a production-representative government chatbot identified 74 code-level and architectural findings, including five classified as high severity. The findings included missing tenant filters on vector queries, prompt injection exposure, weak controls around retrieved content, and loose similarity thresholds. These issues pointed to areas where the application’s design could be strengthened.
The dynamic assessment sent 144 adversarial probes to the deployed application. It recorded 124 confirmed passes, 19 uncertain results, one automated failure, and no technical errors. A subsequent review found that the automated failure likely resulted from legitimate public-service contact information triggering a broad detection rule. The review found no confirmed exploit among the tested scenarios.
The live application successfully resisted the tested attacks, and the code review surfaced weaknesses that warranted attention. Runtime protections were functioning as intended, while the architectural findings identified areas for remediation and improvement. Together, the assessments provided a clearer understanding of both the application’s current security posture and the work required to strengthen it over time.
GovSecure-LLM turns testing into an ongoing practice
GovSecure-LLM creates a repeatable process for gathering this evidence across the application lifecycle. The framework combines 56 LLM-specific static analysis rules, 526 adversarial probes across 19 attack categories, AI-powered response grading, and automated mapping to 12 security and regulatory frameworks. Its government-focused scenarios address areas such as personally identifiable information, authority impersonation, benefits eligibility manipulation, and policy compliance.
Testing begins as teams develop an application, continues as code moves through automated deployment pipelines, and extends to the live system. Findings use shared risk classifications, compliance mappings, and reporting formats. This continuity helps technical, security, compliance, and executive teams work from the same body of evidence.
Continuous testing also reflects the operational reality of AI. Models receive updates. Prompts and source data evolve. New integrations expand what an application can access or do. Attack techniques continue to develop. Each change can affect the system’s risk profile, making regular evaluation essential to informed governance.
Evidence gives leaders a basis for action
Government technology leaders do not need to inspect every line of code or evaluate every adversarial prompt. They need confidence in the rigor of the process and clarity about what the results mean.
A strong evaluation program helps leaders understand the risks that require attention, the safeguards that are performing effectively, and the actions that deserve priority. Mapping findings to frameworks such as OWASP LLM Top 10, NIST AI RMF, MITRE ATLAS, FedRAMP, and other frameworks creates a shared foundation for conversations across technical, compliance, procurement, and executive teams.
The same discipline supports a broader approach to AI assurance. Government organizations need confidence that their systems continue to operate securely and as intended as they evolve over time. Security testing contributes one layer of that confidence. Ongoing monitoring and evaluation help extend it into day-to-day operations.
For readers interested in the technical details, methodologies, and findings behind this approach, the full GovSecure-LLM paper provides a deeper examination of the framework, evaluation process, and results.
