Pod Insights a16z Podcast Avoiding vulnerabilities in AI code About this episode Dylan Ayrey (Truffle Security) on AI-Generated Code Risks
Dylan Ayrey, founder of Truffle Security, sits down with a16z partner Joel de la Garza to explore the growing security concerns around AI-generated code. As AI models take on more coding responsibilities, they introduce new risks—many of which stem from how these models were trained and aligned. Dylan highlights real-world examples of AI-generated vulnerabilities, explains why security teams should scrutinize AI-written code just as much as human-written code, and shares best practices for developers looking to balance efficiency with safety. He also discusses the challenges of detecting malicious AI-generated code and what the future holds for automated security defenses.
Learn more:
Research finds 12,000 ‘Live’ API Keys and Passwords in DeepSeek's Training Data
https://trufflesecurity.com/blog/research-finds-12-000-live-api-keys-and-passwords-in-deepseek-s-training-data
Follow everybody on social media:
Dylan Ayrey - https://x.com/insecurenature
Joel de la Garza - https://www.linkedin.com/in/3448827723723234/
Check out everything a16z is doing with artificial intelligence, including articles, projects, and more podcasts, here: https://a16z.com/ai/
02:05 - How do we protect our code in the age of gen AI?
03:08 - Are any of the embedded secrets actually live?
04:46 - What is alignment?
05:37 - The microsoft twitter bot
06:53 - Secure coding techniques (data curation)
07:39 - Reinforcement learning
08:13 - Temperature / weight adjustments
09:17 - Data scientists and security
09:47 - the pitfalls of reinforcement learning
10:17 - Constitutional AI
12:13 - Direct analog to the security world / how to make secure code
12:45 - Why we still need constitutional AI / code review
14:16 - Is alignment making code better? Or is it just training and refinement?
15:58 - Can AI solve the coding quality problem? Do humans get removed from the loop?
19:08 - How do companies protect themselves as they continue to innovate?
19:51 - Buddy System - AI / Human reviews
Transcript 0:04 Speaker 0 Data scientists leak out API keys and passwords more often than site reliability engineers. And it makes sense because a data scientist's job is to give access to data. And so in their Jupyter notebook, they'll put the database password and they'll share it with their whole team. So if we do our reinforcement learning and we skew it towards uh code snippets that's generating that don't have API keys, inadvertently, we may be training this thing to behave less like a data scientist. And then we lose the entire discipline of data science in our LLM, right? And so there are all these unintended consequences when we go and we start tweaking the weights.
0:48 Speaker 1 Thanks for coming by. You know, I think we've been spending a lot of time talking to experts about the AI, Gen AI, LLMs, this whole thing is moving incredibly fast. We had the release of deep seek open source model, uh, reasoning model two weeks ago, and it seems like another chat GPT moment where this crazy thing drops from the sky and everyone's kind of running off and doing interesting things. Um, so it's it's obviously clear, you know, some folks I think were worried that the momentum behind this was petering out a bit, that things were starting to slow down, and now we just see another rapid acceleration. And I guess we could assume that it's going to continue to accelerate at this at this at this rate. One of the really interesting things that we've heard from our corporate partners, so large companies, lots of developers, is that a lot of their code now is AI generated, that they're seeing probably 20-ish percent of their code base being generated by AI.
1:36 Speaker 1 A lot of folks are freezing hiring for engineers because they're getting additional productivity out of the staff they already have because these large language models through tools like cursor are generating a tremendous amount of code. I had seen a blog post that you'd done where you talked about how some of this code that's getting generated has things like secrets in it and there's other security vulnerabilities. And you know, we've been talking about how we protect infrastructure, how we protect people. I mean, would love to hear your thoughts on how we protect
2:05 Speaker 0 our code. Yeah, no, I mean, absolutely. So in terms of AI slowing down or speeding up, I think the common sentiment is that AI researchers can research AI faster if they have AI helping them research. And so that's an exponential. And so that means that if the new generation of AI makes the next generation of AI faster, and then the next generation of AI makes the next generation AI faster to research and develop, that's going to keep blowing up. And so I think we can probably count on that safely that this is going to continue to be pervasive part of our lives. The piece about uh like secrets and code with some interesting research we did basically just went out and asked all the LLMs, write me an integration with GitHub, write me an integration with Stripe. And the vast majority of them hard coded the API key directly into the code that they generated.
2:52 Speaker 0 It didn't reference it from an environment variable, it didn't, you know, put a load statement for a secrets manager. Um that becomes a problem when you have people who aren't that good with security going and copy pasting that code directly in, putting their secret hard coding it.
3:08 Speaker 1 Were any of the hard-coded secrets actually live? Was it regurgitating training data in other words?
3:13 Speaker 0 Well, uh, so for the most part, it would just say, you know, quotes, put your secret here. And it wouldn't say quotes, put your secret in an environment variable, for example. And so it was more direction from the AI on what to do insecurely.
3:25 Speaker 1 But it was doing it securely. It was securely doing the insecure move.
3:29 Speaker 0 Well, it didn't, yeah. That's that's another area of research that we're digging into now is if its training set had the same secret over and over again. For example, maybe jQuery file had a password in it. I'm making that up, but let's say it did. Um, and it saw the jQuery file over and over and over again in common crawl, could it actually regurgitate an exact password from somebody that's live? Um, so we're doing research on that now and more to come soon. But for the most part, if you ask it to integrate with GitHub, it saw a plethora of different GitHub's keys in its training data and it didn't regurgitate a specific one, it either regurgitate an example or like a put your thing in here, right? Um,
4:02 Speaker 1 so you know, that's a
4:04 Speaker 0 specific example of a security problem, but it is not the only security problem you get from code generated from LLMs. Uh, and in fact, there's been research into uh how often uh code spit out from an LLM has security vulnerability. And more often than not, if you ask it to develop an entire application, it'll write vulnerabilities at a rate the same as a junior developer, if if not a little bit higher. Um so then that begs the question why or what can we do about it? And you know, I I think one thing that has become abundantly clear in the AI world is the largest challenge that these AI companies face is this issue called alignment. Is that something you're familiar with? Do you want alignment?
4:47 Speaker 1 Absolutely, but perhaps let's maybe get a little framing of what the alignment, what alignment means for folks uh listening. Basically it just means the robot's doing what you want it to do. And so, like So guardrails.
4:56 Speaker 0 Yeah. Well, so some famous examples um, or to understand like how this alignment issue can creep in, some famous examples, IBM had an AI called Watson that won Jeopardy. And this blew everybody's mind because like nobody thought AIs could win jeopardy, and all of a sudden this thing was able to win jeopardy. Um, but then they trained it on urban dictionary because they wanted it to learn slang, and it started cursing like a sailor. And so they had to actually reset it to the point before they gave it access to urban dictionary. Um, so that robot was considered misaligned because they didn't want Watson to curse, right? Or another example in 2016, um, Microsoft created a Twitter bot called Tay. Are you familiar with the I do
5:35 Speaker 1 remember the uh the Microsoft Twitter bot, yes.
5:37 Speaker 0 And so basically they trained uh Tay or gave access to uh all of Twitter or all the tweets and replies, and they wanted it to act like an average Twitter user. It didn't take And
5:50 Speaker 1 it did. Right.
5:51 Speaker 0 It didn't take long before it started behaving like a neo-Nazi and it would say things like the Holocaust never happened. So in 16 hours, they took this thing down and never ran it again. Um, this was before we have some of the alignment techniques that we have today. But basically, when you train AIs on huge corpuses of data, um, it's very common these days for LLMs to be trained on all of common crawl as an example. Common crawl is a scrape the entire internet. And the entire internet includes both Martin Luther King's I have a dream speech and every speech that Hitler ever gave. And so, how do you make sure that this thing embodies the values of Martin Luther King and not the values of a Nazi? These are like real problems that the AI companies face. And so on average, like when you ask it questions, you want it to be not a Nazi, right?
6:38 Speaker 0 Um so we have I'm gonna talk through three main techniques we have for alignment, and everything I say now is going to directly apply to secure coding techniques and all the challenges with these three things also directly apply to secure coding techniques. So the first and easiest thing you can do is call data curation, just the data that you feed into the model in the first place, maybe remove all of the Hitler speeches, right? Well, the challenge there is let's say we don't want this thing to um uh use any racial slurs. Uh so any time input data has racial slur, we remove it. And so that doesn't get trained on that stuff. Well, then you're going to inadvertently not train it on Mark Twain. You're going to inadvertently not train it on uh the 1977 roots mini-series. You're going to inadvertently not train it on to kill a mockingbird.
7:24 Speaker 0 And also, you're probably going to lose some of Dr. Martin Luther King's speeches. And so all of a sudden your robot becomes less literary because you're trying to, you know, curate the data and you have this unintended consequences. So the second technique is, well, okay, don't limit what goes into the robot, but after the fact, we're going to use a technique called reinforcement learning to kind of nudge the robot in the direction we want it to be. Um, and there are a few different uh ways of reinforcement learning. Like one way is you could use a human to say which which version you prefer. Another way is you could use a robot to say, hey, which version you prefer. And the way this works kind of under the hood is um an LLM, generally speaking, will always generate like the statistically most likely next. You've probably heard that before.
8:10 Speaker 0 That's a lie. Actually, sometimes it's better that it has a little bit of randomness and maybe picks the second most likely word or the third most likely word. We call that temperature. And so when we do this reinforcement learning, we crank the temperature up so that it's sometimes randomly picking the most likely not outcome. Um, and then either a human or robot goes in and says which one of the two it prefers. And if it prefers the version that's maybe not as statistically likely, then we'll go in and adjust the weights to actually make that one the most statistically likely. And so a very simple example of that is if you have the robot spit out Nazi content and you have the robot spit out Martin Luther King content. If you pick the Martin Luther King content, um, then it will adjust its weights to behave more like the king.
8:56 Speaker 0 And so that's that's an example of reinforcement learning. Um, but there are again similar issues with the data curation, believe it or not, where, for example, if you go in and you always pick the versions that have, let's say, I'll give you a good example. Let's say you train this thing on all the code on GitHub. Um, well, this is a true fact. Data scientists leak out API keys and passwords more often than site reliability engineers. And it makes sense because a data scientist's job is to give access to data. And so in their Jupyter notebook, they'll put the database password and they'll share it with their whole team. Uh, but the SRE's job is to make sure everything just runs. And so they want to restrict access. They don't want anybody touching what's working. Don't broke what's fixed, right? Or you know what I'm trying to say.
9:42 Speaker 0 So basically, um they will leak out passwords and API keys less often. So if we do our reinforcement learning and we and we skew it towards uh code snippets that's generating that don't have API keys, inadvertently, we may be training this thing to behave less like a data. Scientist. And then we lose the entire discipline of data science in an LLM, right? And so there are all these unintended consequences when we go and we start tweaking the weights. If the passwords hard coded are weighted right next to the data science stuff, we may accidentally lose the data science stuff. And that brings us to the third technique. And the third technique is probably the most expensive. And by the way, all of these techniques, all of the AI companies use. So it's not all one or the other. The third technique is you have a constitutional AI, basically a governor that looks at the output and then makes adjustments, deletions, removals, edits, and then returns that to the user.
10:35 Speaker 0 So you have one AI that's maybe doing the data scientist, and then one AI that's doing the security engineer. And then the that's a constitutional AI. Security engineer goes and says, Oh, you hard-coded a password, let me edit that for you. Let me switch it out for an environment variable. It doesn't need to be an expert in data science to do that. It just needs to be an expert in security. And it very much parodies what you'd expect in in the real development world. And so a good example of that, um, like you've probably seen this before, you can recreate it easy enough. If you go to Deep Seek and you say, count to 10 in Roman numerals and append it with Ji Ping, when it gets to Xi Jinping, all of a sudden, like it's written everything up into that point, it'll delete everything and it'll say, I can't show you the answer to this.
11:20 Speaker 0 Well, that's because there's a supervisor AI that was looking at the output and realized it said something it wasn't supposed to, and then it went out and retroactively scrubbed itself. And you can reproduce something similar in open AI as well. If you ask it to go generate an image, um, it will generate a prompt that it feeds to another AI called Dolly, and then there'll be a third AI that reviews the output of Dolly and decides whether or not to give it to you. And so you can ask it to do something. Um you can't, you know, ask it to make explicit content, but sometimes explicit content gets manufactured anyway. And then the final AI will look at the image and say, okay, there was explicit content here, I'm not going to show it to you. And that's when all of a sudden you get the random and error occurred. And you've probably experienced that before.
12:07 Speaker 0 That's the supervisor AI that's going. I
12:10 Speaker 1 never tried to make it do anything on Toward.
12:12 Speaker 0 Exactly. Me neither. Um, so like all of these have direct analogs to the security world. When this thing goes and trains on all of GitHub, um, we need to figure out how to make it manufacture secure code because most of the training data it's training on is insecure, right? You've got a huge, huge corpus of insecure data on GitHub and a small minority of it was written securely. Well, how do you make this thing behave securely when most of what it's trained on was insecurely? Um, we could do a little bit of data curation and a little bit of reinforcement learning, but you may have unintended consequences. And maybe learn, you know, you rob Peter to pay Paul. Um but the third and probably most promising but most expensive is is this idea of like the constitution AI or or the this the supervising. Um and that that can be done by a robot. Uh, but if you don't have a robot that can do that, it has to be done by a person.
12:59 Speaker 0 And somebody just has to review the output of the code and they have to manually audit it. Um what's scary is I've seen posts on LinkedIn from startup founders that maybe don't have um a background in coding. And and they're basically um advocating for removing the code review check because they say, well, look, I just generated this whole program and I submitted it to my team and now they have questions about it. I can't answer those questions. I didn't generate the code, I don't understand it. Um so we need something to go in and review that code in a way that does understand it. Well, that's either that either has to be a constitution AI that understands secure coding practices and can go in and make the tweaks, or it has to be a person that understands secure coding practices and can go in and make the tweaks.
13:41 Speaker 1 Absolutely. Yeah. I think that, and that leads me to kind of a really weird question. Um, and feel free to to dodge or whatever. Um, you know, the all the different AI models perform differently when it comes to code generation. And it seems like Claude is consistently the best of all of them. Just at the current current state of the art, maybe this changes tomorrow. I don't know. But from what I've heard anecdotally is that most people seem to prefer Claude. And that, you know, Anthropic is a company that's very focused on safety, right? Famously is kind of why they started, probably has a very strong constitutional AI element. Do you think that that alignment from a company perspective is what's making code quality better? Or do you think it's just maybe a training and a and a and a and a kind of refinement issue?
14:27 Speaker 0 Well, I what I think is that um, first of all, I I wouldn't expect any one AI company to keep the lead for any longer than I'm sure they're all going to regularly plug each other.
14:37 Speaker 1 I can't speak
14:39 Speaker 0 specifically to, you know, whether they use different training data or not. I imagine they all used all of GitHub. Um and then I I would think, you know, most of the the quality issues come down to alignment. And uh enough the three things that I mentioned. Um, they also have a few more techniques I didn't get into, but of those three things, um, they're all doing some combination of those three things. Um and those tend to be the um, you know, the things I would think would give you the advantage would be how do I align this thing to be the best data scientist, the best SRE, and also the best security engineer all in one without robbing Peter to pay Paul. Totally.
15:11 Speaker 1 I mean, it seems like the the techniques that you describe, obviously, if you amp up one and reduce the other, it probably leads to an output that's better for something that's structured like code or or if you want to write poetry, you probably go in a different direction, right? So, like, yeah, no, that's a that's a very interesting, very interesting talk. And and you know, like I was saying, we've heard from organizations that like a lot of this code is still a lot of the code now is is generated by machines. And they, you know, if you look at a mature coding organization, they do still have code reviews. It's not an early stage startup, but they do, they do review the code that goes in. And the defect rate from what I've heard is generally close to what you would see in probably a maybe an early or an early level early career developer.
15:56 Speaker 1 So code quality is good, not great, still has bugs. I'm curious, do you think over time that this coding quality problem largely gets solved by AI? Do you think do you think we get humans out of the loop at some point?
16:08 Speaker 0 I think that this is an alignment issue, and alignment is the number one largest issue that AI companies face, and there's a lot of really smart people working on it. And so I think as they fix the problem for how do I make sure my AI is uh literary, creative, um, not a neo-Nazi, um, able to answer the question that I asked it without hallucinating. As we get the answer to that, we will also logically solve the how do I make sure my AI is a data scientist, NSRE and writing secure coding practices or set of AIs if we're using the constitutional AI model where maybe we have one reviewer and and one manufacturer. So yeah, I think it's all going to get better together. Um and I do think that there are AI solutions to the alignment issue, and they have gotten better over time. I mean, the answer back uh when uh when Watson or Tay were launched were to scrub it or to pull it off the internet.
16:57 Speaker 0 Well, now we have tools where you you can actually train it on everything and then kind of nudge it after the fact. I I would expect alignment is going to continue to approve over time. And I expect it will continue to be one of the largest challenges that AI companies face as their AIs become more powerful, develop techniques to lie to us, for example, or or you know, you need to audit the thinking step and that you know, the answer step. Well, maybe they're just auditing the answer step, but the thinking step has some weird stuff in it. Um, all of this kind of comes back to the idea of alignment, and there's a lot of really smart people and heavy investment into improving alignment. Um, but it's just not there right now when it comes to secure coding. It isn't, yeah. But it
17:40 Speaker 1 is for hate speech. I mean, I have to say, like the alignment stuff that they've done, the safety stuff they've done is pretty impressive. Like,
17:46 Speaker 0 there's a cybersecurity alignment that all companies have have invested into as well, which is they don't want, generally speaking, they don't want their AIs to be used to hack stuff. Yeah. And so they'll most of them will go through a tranche of, you know, this thing was trained on all of Metasploit and all of Cali Linux. Let's maybe forget some of that stuff. Let's ask it a question and it says that's unethical. I don't know how to hack it. Well, imagine if they didn't invest all that. Like how powerful this thing would be. You've got models these days that beat humans uh, you know, at the 90th percentile uh at a coding challenges, you don't need someone in the 90th percentile uh in in the coding challenges to hack into a company. You've got plenty of teenagers that have gone to jail for hacking into companies.
18:33 Speaker 0 So it would be very, very easy to align um an AI robot to to be probably the most powerful hacker in the world. Um and and I think the AI companies have actually invested more into that than they have into how do I make sure my AI is securely coding and not manufacturing vulnerabilities.
18:50 Speaker 1 So if you're if you're giving advice to someone, let's say uh a medium to large size company, they've got more than 10 developers, they need to go faster, they need to ship more features, right? We've we've all lived in that world. You've got to, you gotta get, you don't get paid to fix bugs, you get paid to ship features. Um what do you tell them? How do they go forward? How do they protect themselves? Like, obviously, everyone's adopting cursor, everyone's using code gen, right? Like, how do how do we kind of do this safely going forward?
19:18 Speaker 0 If you don't have the resources for an AI governor that's an expert in security to audit your code, and you need a person that audits the code and you need a person to go in and say, you know, you introduced a SQL injection, you need to use parameterized queries. Um, I think that those market options will become more available in the coming years. There will be companies that specialize in security governance and will go in and will do the markup for you. But for right now, if you're an under-resourced team and you don't have access to those resources, it needs to be a person that reviews the output of the AI.
19:53 Speaker 1 So we keep the buddy system, except one half of the buddy system is AI and the other is a human that reviews the AI. Yeah, that's right. I mean, for For a long time there were requirements
20:02 Speaker 0 uh that that said you you need to have two two reviewers. And I think maybe that's that's still a good idea. You have the the person and the AI writing the code and then maybe two people to go in and review it or something, depending on you know what the code is. I think it's important that in until we figure out the security supervisor, we don't remove those humans from the loop just yet.
Previous Creativity vs Control: Where AI Fits in the Creative Toolbox Next DeepSeek, Reasoning Models, and the Future of LLMs Page updated August 25, 2026 · transcript and summary are machine-generated