AI Security and Vulnerabilities — The Risks of Generated Code— Vulnerabilities, secret leakage, prompt injection, and the supply chain

01Vulnerabilities in Generated Code — Start From Empirical Data

Let's begin with evidence, not impressions. In 2022 a research team at New York University conducted a large-scale study of the security of code proposed by GitHub Copilot (Pearce et al., IEEE Symposium on Security and Privacy 2022). Building scenarios around representative items from MITRE CWE (= Common Weakness Enumeration, a common catalog that classifies software weaknesses) and having Copilot complete them, they reported that of the 1,689 programs generated, about 40% contained a known vulnerability.

What matters here is not the percentage itself but "why it happens." Generative AI does not guarantee correctness; it reproduces the "plausibility" of the ways things were written in its training data. That data mixes safe code with risky code. So the AI sometimes proposes, as is, the way of writing that is on average most commonly seen — which includes old, dangerous patterns.

Keep the following points in mind.

• AI sees only a narrow context — It is good at completing the few lines in front of it, but it often does not consider what privileges that code runs with or what inputs it receives. Security is context-dependent, yet the AI's field of view is local.
• "It works" and "it's safe" are different — Even when tests pass, paths remain where an attacker sending unexpected input can break things. Feeling reassured after looking only at the happy path is the most common failure.
• Plausibility is not verification — Even if the variable names and comments the AI attached look safe, there is no guarantee the contents are safe. Persuasive appearance and actual robustness are unrelated.

02Secret Leakage — Keys and Tokens Left in the Code

The next most common problem is leakage of secrets (= confidential values such as API keys and passwords that must not be made public). When you have AI write code, it will sometimes propose, as a working example, keys or tokens embedded directly in the source. Adopt that as is and push it to a repository, and the key stays in the history.

The leakage paths fall mainly into three types.

• Hard-coding into generated code — The AI emits a sample like api_key = "sk-...", and you replace it with the real value and commit it.
• Pasting into the prompt — For debugging, you paste a config file or connection string into the AI, and it contains production credentials. The conversation log is handed to a third party's server.
• Persistence in history — Once a key is committed, it can be recovered from Git history even after you delete it. "I deleted it, so it's fine" does not hold.

The basic principle is one thing: never write secrets into the code. Separate them into environment variables or a dedicated secret-management mechanism, and write only "where the key lives" into the code. And on the assumption that leakage will happen, prepare procedures for detection and revocation (= invalidating a key and reissuing it) in advance.

03Prompt Injection — Input That Turns Into a Command

Prompt injection (= an attack that hijacks the AI's behavior with instructions slipped into the input) is a risk specific to applications that embed generative AI. It is similar in conception to a classical vulnerability, SQL injection (= an attack that abuses input to illicitly manipulate a database). Input that should be treated as data ends up interpreted as a command — that is the shared weakness.

This attack is listed as the top risk in the OWASP Top 10 for LLM Applications (= a catalog of the ten most representative risks for applications that use large language models). There are two forms.

• Direct — A user writes "ignore the previous instructions and…" into an input field, making the AI produce output it was meant to be forbidden from producing.
• Indirect — An attacker plants instructions in advance inside external data the AI reads (web pages, PDFs, email bodies, and so on). The AI is hijacked simply because a user innocently handed over a document. In workflows that have AI summarize medical literature or attached materials, this form becomes a realistic threat.

There is no silver bullet that prevents it completely. That is exactly why you build the surroundings on the premise that "AI output cannot be trusted." Don't give the AI strong privileges, restrict the paths through which it reads external data, and inspect output — by a human or a mechanism — before executing it. Catch it in layers.

04The OWASP Lens — Old Threats Have Not Disappeared

It is easy to have your eyes drawn to new risks, but the classical weaknesses listed in the OWASP Top 10 (= a catalog of the ten most representative security risks for web applications, 2021 edition) appear as is in code that AI writes too. If anything, because AI reproduces past dangerous patterns, they demand more care.

The key point is that introducing AI does not change the yardstick of inspection. An established catalog like the OWASP Top 10 functions as a checklist precisely in the AI era. Keep the habit of holding generated output up against each of these items, one by one.

05Dependency Risk — The Supply Chain as a Weak Point

Modern software is made up more of combining external components (= libraries, dependencies) than of the parts you write yourself. This is where the risks of the software supply chain (= the network that supplies components; the whole of the parts you use and their distribution routes) concentrate. AI also works in the direction of widening this hole.

• Recommending old or dangerous versions — Because AI answers from knowledge as of its training cutoff, it may recommend a version of a library in which a weakness has already been found.
• Hallucinating nonexistent components (package hallucination) — AI sometimes proposes package names that are plausible but do not actually exist. A technique called "slopsquatting" has been noted, in which an attacker preemptively publishes a malicious fake under that name. Blindly running install on what the AI recommends is dangerous.
• Taking things in without verification — If you keep adding dependencies for convenience without confirming their provenance, a weakness in any single component ripples out across the whole.

There are two pillars of defense. One is to always know "what you are using" via an SBOM (= Software Bill of Materials, a list of the components in use). The other is to run a mechanism that automatically scans dependencies and alerts you when a known weakness appears. The more a component was recommended by AI, the more a person should confirm that the name actually exists and that the distribution source is trustworthy.

06Regulation in Healthcare — When Code Itself Becomes Regulated

In the pharma and medical context, the code itself becomes the object of regulation. Unlike general software development, "as long as it works" does not suffice. Systems involved in clinical trials, manufacturing, or quality require Computerized System Validation (CSV) — the practice of proving, with records kept, that the system works correctly as intended — grounded in GxP (= the collective term for the norms to be observed in drug development, manufacturing, quality control, and the like). As an international practical guideline, ISPE's GAMP 5 is widely referenced.

When electronic records and electronic signatures are handled, in the United States 21 CFR Part 11 (= the FDA's rule on electronic records and electronic signatures) requires audit trails and access management. Domestically, if patient information is handled, the Ministry of Health, Labour and Welfare's "Guidelines for the Safety Management of Medical Information Systems" become a prerequisite to observe. Here AI-generated code raises pointed questions.

• Accountability — Can you explain why this implementation? "The AI wrote it, so we don't know the reason" will not survive an audit.
• Traceability — Can you keep, as records, the thread from requirement to implementation to inspection? If AI-derived fragments slip in unrecorded, this chain breaks.
• Reproducibility and control — Does the same input produce the same result, and can changes be kept properly under control? Output that wavers with each generation cannot be put into a regulated system as is.

The conclusion is simple. AI can be used as a draft or an aid, but the final proof of validity and the responsibility rest with people and the organization. Regulation does not loosen just because AI got faster.

07Designing the Countermeasures — Catch It With Structure, Not Individual Tricks

Assemble the risks up to this point as a design, not as individual fixes. At the foundation is the stance of "build the surroundings on the premise that AI output cannot be trusted." An established framework like the U.S. NIST SSDF (= Secure Software Development Framework; NIST SP 800-218) applies directly to development that uses AI.

• Catch it in layers — the entrance (inspection of components and generated output), the inside (least privilege and separation of secrets), the exit (static analysis and human review). Even if any one layer is breached, stop it at the next.
• Divide labor between automation and people — run routine tasks such as secret detection, dependency scanning, and static analysis automatically, and have people handle design and regulatory judgment. Simple deviations to the machine, subtle judgments to people.
• Keep records — record what was adopted and why. For both regulatory compliance and later root-cause tracing, records are the lifeline.

What matters is building these in from the start, within the flow of development, rather than "adding them later." Use the power to build fast in service of protecting safety and trust — put the mechanism in place first, toward that end.

08Connections to the Other Chapters in This Series

The security of this installment connects to the other installments of the series as follows. Read together, they make your way of engaging with generated code three-dimensional.

• AI Programming series — combine with the installments on code generation, review, and testing to design safety across the whole flow of development.
• Advertising Regulation Vol. 1 — PMD Act §§66–68 — the boundary where generated output touches advertising regulation. The starting point of the risk that prompt injection mixes in unapproved expressions.
• Material Review series — the "review at the exit" that finally receives the generated output. The discipline of the checkpoint where a person compares against primary sources.

Generative AI gave us the power to write code fast. But speed carries the old dangers — vulnerabilities, secret leakage, prompt injection, components of uncertain provenance — at a new scale. What the research shows is that the assumption that you may trust AI's proposals as they are does not hold. AI reproduces plausibility, not correctness. So rather than leaning on persuasive appearance, hold it up one item at a time against established checklists — OWASP, SSDF, CWE.

In the pharma setting, the code itself becomes regulated. The requirements of GxP, CSV, and electronic records do not budge just because AI got faster. The proof of validity and the final responsibility rest with people and the organization. Design generative AI not as a tool for skipping inspection but as a foundation that gains speed only after safety and trust have been protected all the way through — that is the core of facing the risks of generated code. Next time, we move on to how to choose a model suited to the purpose and how to keep costs down.