The Standup with ThePrimeagen
Original podcast
Where to listenCopy share link
The gang discusses Test Driven Development and laughs a lot. As usual ;) Don’t forget, ssh terminal.shop
9,017 words · 4 speakers
So today's stand-up, we are gonna talk about TDD. Okay, we're not gonna have some topical topic. We're gonna go after the big stuff, okay? TDD, is it good? Is it bad? Do you use it? TJ's just smiling, looking like an absolute champ in that brief frame. Dude, somehow TJ's the only human I know that gets freeze-framed, and it's a good picture.
Yep.
Yeah, look at TJ, look at that Labrador.
I can't handle this right now. I cannot handle this right now.
Alright, Casey, you probably have the strongest opinion, potentially, out of all of us.
Okay. I doubt that, but okay.
Alright, cuz I also have a strong opinion, but I'm...
It's probably you, then.
Yeah, I think Prime, you gotta lead up the strong opinions here.
I'll lead up with the strong one, which is, so, I recently just tried another round of TDD.
Prime, Prime, you have to say, "Hey, everyone, welcome to the stand-up. We got a great show for you tonight. I'm your host, the Primagen. With me, as always, are TJ, hello, Trash, hey, everyone, C. Muriatory. Great to be here, Prime, thanks. Today's topic is test... Have you ever heard a podcast before?
Thanks, everybody, for joining. You're watching the hottest podcasts on the greatest software engineering topics, The Stand-Up. Today, we have with us Tej, who's in the basement fixing something.
Is that a Home Depot ad?
Look at him, he's helping us, his router. We have with us Trash Depp, broke a toe, winning a judo competition, still living off that high. And C. Muriatory, the actual good programmer among us. Oh, alright, I got a lot to live up to. Yeah, you got... Don't worry, it's not hard to live up to, at least in this crowd. I'll kind of start us off on this one. So, we've been developing a game called The Towers of Mordoria. We came up with the name from an AI, and AI generated that name for us. And effectively, I decided that, at one point, I have this kind of really black box experience. I have a deck of cards that I need to be able to draw cards out. When they're played, I either need to go to the discard pile or the exhaust pile. When there's no more cards in my hand or not enough to be able to draw cards,
I need to be able to take all the discard cards... Dude, Tej is just flexing on us, touching grass right now. Put all the discard cards, shuffle everything up, and then draw new cards back out, replenish the draw pile. Right? This sounds like a TDD pinnacle problem. There's black boxes, there's a few operations, I'm gonna be able to do this. And so I just did this like a week and a half ago on stream. It was amazing. I did the whole thing, designed the entire API in the test, then set up all the tests, set up it all to fail, then it did work, or then it failed, then I made it work, then it worked, and it was great, and everything was perfect, and I was like, there we go, we just did it. I just TDD'd and it was a good experience. So then I took what I made, and I went to integrate it into the game,
and realized I made an interface that's super good for testing, and not actually good for the actual thing I was making, and I had to redo it. And so it always, you know, that's kind of been a lot of my experience with TDD, is that either the thing is simple enough, and is super black box enough, that I can make it, and both the testing interface and the usage interface is the same, or I create an interface that's really great for testing, and it's actually not that great for the practical use case. And I know people talk about how it's a great way to develop your, you know, your architecture and all this kind of stuff. I've just yet to really buy it, but I will say the thing that I do like doing is on problems that are hard to develop and have many moving constraints, I do like to co-create tests with it.
Meaning that I come up with an idea, how do I want to use it? I program it out, then I go create a test, which is backwards from TDD, go, okay, this is what I want to see. Am I even correct on this? Can I get a fast iteration loop being like, no, no, no, yes, okay, got it. Next thing, I want to make sure it does this. No, no, no, yes, okay, got it. And so I do like the fast iteration cycle of getting tests in early, but few problems feel like I can actually have that. And I feel like this, all things need to be tested immediately and need to be written first. You just write dumb interfaces, everything becomes inversion of control, everything's an interface, and you just have this abstract mess at the end of it, because testing was the requirement, not, hey, use it when it's good.
So that's my TDD experience, and I feel fairly strong about that. And if you say TDD's good, I feel like I'm gonna jump kick you.
I can respond to that, or Casey, if you have strong feelings.
No, I think Trash, let's go to Trash take first. Yeah, no blockers on my end.
First, you couldn't like deter me from doing any task than to tell me I need to like start off by writing tests first off. Second, I kind of had a very similar experience recently building like a CL.
Hold on, can you rewind for a second? What, what did you just say for your first point?
I said, "You couldn't deter me from doing work if you let off with me having to write tests first."
I don't even know what that means.
You mean you couldn't deter me anymore? Like that's the most deterring thing or?
Man, am I stupid?
Can you try using a different word to describe what the issue is? Okay, let's, let's reword this. Flip, please take this out.
Flip, leave it in.
I don't like test-driven development, however.
Okay, got it.
However, however, I just built something recently at work, which was very kind of similar to what Prime was saying.
I ended up doing a little bit of test-driven development, but
you wouldn't notice this, but I'm actually really kind of passionate about testing. So I'm gonna kind of like, when I test, I would say
2% of the time, I actually end up with test-driven development and the rest is usually post-implementation because one,
requirements and all these things change afterwards, right? So it's like, why write tests for things that are potentially gonna change and you don't really know what's gonna happen? But I do write tests for very granular things that I think are gonna be used all the time. So for instance, like if I wrote a parser for like my CLI, the parser's gonna be reused by multiple commands, probably never gonna change. So I typically will write that and I'll also also write tests for things that take multiple steps to test. So like, you know, people argue for end-to-end tests because you test just like a user. So that's kind of how I like to develop. I like to use it like I would use it for anybody else. But if I gotta do like three steps to get to the point I want to test, I'm like, alright, screw this. I'm just gonna write a test specifically for this like instance and then I save time that way.
So that's usually the time I arrive at test-driven development is specifically in those scenarios. Otherwise, it's purely just like, you know, uh, what do you call it? Integration testing and stuff like that because you end up with stuff like, like Prime said, like you, when you write tests, you write everything so granular and everything works so modular that you just don't even know what it's like when you actually put them together. So that's why, that's kind of like the way I approach things. That being said, like people that kind of like
Live and Die by Test Driven Development? I don't, I don't get it. But I was hoping someone would be pro TDD in this group, so I could argue with them.
That's all I got for right now.
I'm, I'm, uh, pretty much right there with trash on this one. My, my feeling on TDD is pretty much the same as my feeling on OOP, right? The problem is not if you end up with an object that, or something that looks like an object in your code in OOP, that's fine. The problem is the object oriented part, the like thinking in terms of that wastes a lot of your time and leads you to bad architecture, I think. TDD is the same way. I think testing is very good, but test driven development, meaning forcing the programmer to think in terms of tests during development, like that is what your primary thing is, is I'm making tests to drive the development, the TD part. That's actually the problem. I think the reason for that's pretty straightforward.
Uh, and it's kind of weird because usually the people who advocate for TDD are the same people who would poo poo any similar suggestion, like any suggestion that like, oh, we should measure performance during development. They'll be like, that's ridiculous. You're wasting so much program time. Like, what about the 8,000 tests that no one ever needed because we ended up deleting that system anyway? And they're like, well, you know, test driven development is very good. So I think like, the problem with all of those sorts of things, any of them, is just putting emphasis on something instead of talking about it like what it really is, which is a trade off. If you're spending time making tests or you're orienting things towards making tests, that is necessarily programming time that isn't being spent doing something else.
Like, programming is zero sum. If you spend time doing one thing, you're not spending time doing another thing. And so, for example, if you end up spending a lot of your time making the interface revolve around tests, you weren't spending a lot of your time making the interface revolve around actual use cases, because actual use cases don't usually look like tests. Therefore, sometimes you get the exact situation that Prime was talking about, where you do test driven development and you end up with a completely unusable mess, which is actually very common in my experience, right?
I wouldn't call it a mess. You know, I write pretty good. God, I mean, it was like pretty dizzy. I just want to say.
No, the API, the API is a mess. Not the, not the code, right? Like, the code has been tested and it probably works. But the API is a mess, right? I mean, that's what happens. Uh, because it wasn't designed specifically for the kinds of use cases it was meant to tackle. And so you end up with that problem. And again, it was shifting the developer's focus was all it took to make that happen. Good APIs are hard to make in the first place, right? So if you're adding more complexity to the programmer's workflow when they're trying to make an API that's good for the, you know, the actual use case, you're just going to get a worse result. Again, zero sum. So what I would say is like, testing is better something that you do as the thing takes shape. When you're like, okay, we now have figured out like how this system works properly.
We're really happy with it. The API is working nicely. We checked the performance and we think we've got like reasonable, we've got a reasonable path towards having this perform really well. Now let's start seeing what are the, what is the overlay? Like, what are the parts we can now start to add some little substrate to, to test that we think are going to have bugs that will be hard to find? Because that's usually the way I look at like, what are the parts of this system whose bugs will be either very hard to find or really catastrophic? Like, they'll cost us a lot of money or they'll crash the user's computer or whatever, right? And then you put the tests for that, right? And that's how I've always approached it. And I think that works pretty well. And, uh, you, you do want to do that step. But I don't think you want test-driven development. It just seems like a really bad idea.
And to be honest, when I use software that comes from people who are like, TDD, I believe this organization, CD, whatever, it's full of bugs all the time. Right? Like, I'm not getting these pristine software things. I mean, I'm, I would imagine that the people at YouTube do a lot of testing and code reviews. And yet for the past eight years, the play button cannot properly represent whether something is playing or not. For eight years, right? Sometimes it's paused and it has the play thing. And sometimes it's playing and it has the play thing. And, and sometimes both of those times it shows the pause thing. It's just, it doesn't seem like the testing is working, guys. So again, I'd say, you know, I think you have to kind of rethink how you're allocating that time. That's what I would say.
Can we get TJ in here? Can, can, uh, Playboy TJ? We don't know this one.
Can we? Am I, am I coming in loud and clear?
Yeah, your voice sounds good. Your voice sounds good. You're looking good. The brass looks good.
Thank you, thank you. So I would, I'm, I'm, uh, I think I agree mostly with what everybody is saying, although I would say for a few kinds of problems. Uh, the problem is, I, I, I think people, when you say TDD, they just mean 10,000 different things. So this is like one of the problems, right? There's like people who are like, TDD is only if you do red, green cycle before you write the code. You have to write the tests and like, you can't write any code till you've written a failing test and blah, blah, blah. Which like I get, uh, that, that maybe they're the people who invented the word so they can define it that way. Uh, but like, I find that not, not very useful for some stuff.
But, uh, there are like testing or like projects where I think I test like basically everything. And there's just ways that you can do it that are like more effective versus less effective. Like my favorite kind of testing is if you can find a way to do like snapshot or like golden testing, whatever, you know, people have different names for it. But where you can basically just write a test and then say, assert that the output is the same as it used to be. Right? Uh, like, so in you, I usually I've seen those called snapshot tests or expect tests. I find that those are like super.
Often.
Yeah, yeah. Or like people will use them in regression, but like the kinds that I've done it before, like, uh, maybe I'm building up a complex data structure to like represent the state of like a bunch of users, or maybe it's the parse tree of something, or maybe it's the type system, like whatever it is. If you can find a way to represent that in like a human readable way, usually just by like printing it in tests that I've done. You can just like compare the diff between this one and before you made the changes. And they're, they should be fast at least. If they're not fast, then that's a separate problem. Like your tests have to be fast for people to run them, otherwise they don't run them.
Then then those kinds of things tend to be, I find like very effective and give you a lot of confidence that you haven't like scuffed anything throughout the system with your changes. Right. Because you're sort of like asserting the entire state of something. And if something changes, it's really easy to update because you just say, oh, snapshot, update this. Cool. Done. One second, all your tests are updated. You walk through them and change them. You get some other side benefits too, like in code review, you literally see the diff of the snapshot and you can be like, um, that looks wrong. That used to say false and now it says true. Those things that shouldn't have been flipped between those. So there are like certain problems where I've employed like
nearly TDD in this snapshot style, in the sense that the snapshot always fails on the first time you run it. Because there's nothing to expect. And then you accept that snapshot if it looks good or keep changing the code until it does look good. And then you like move on to the next thing. So but that but it's kind of like a unique case. Like I wouldn't want to snapshot test a like
HTML page, right? It would just like, there's no way that they're reproducible, it feels like anyways. Uh, but like, so there's, there's, it's only for like certain subsets of problems which have been more effective like for me because I've made a lot of dev tools in my career and stuff like that. So you want to do like verify that all of the references of something look like XYZ. Okay, very easy to do with snapshots.
That's so that's kind of the main, the, the one thing that I wanted to throw out there is like that style of testing I find really powerful. And I've used it a lot, like across projects you wouldn't really expect as well.
I've always had the opposite experience with snapshot tests. I, I feel like it's okay, maybe if like it's you and maybe some other people that actually know the expected output of a snapshot. My experience is It changes and they're like, oh, let's just update it. And then the test pass again and then you're kind of just left with this snapshot and you're just like, alright. So it's to the point like where it's like, alright, let's just get rid of them because they're just changing every time requirements change and no one actually knows what the expected output is. One more point I want to make, someone said something about like test helper refactoring. I think that's true after a product is actually mature. Because you're just going to be, it's kind of like a snapshot test where you're just going to keep updating your tests every time something changes
to the point where it's kind of pointless. So I always find when tests are like that, I usually just delete them immediately. Okay, go ahead, Trash.
Alright, alright.
I called you Trash.
Yeah, I know you call me Trash.
Your, your comment filled Prime with so much joy, it translated into physical up and down motion.
Yeah.
I, I was literally watching him as I was talking and I was like, let's just see how hype we can get Prime.
Yeah, dude, I was high.
I wish I could see you guys. I see nothing.
Yeah, yeah, we see you. We see you guys. Yeah, we could, we could see you, DJ, enjoying grass.
TJ, if you see a strange mushroom in the grass, do not eat it.
Oh yeah, don't eat it.
Do not eat the strange mushroom, DJ. When you're mowing, there's a strange mushroom.
Guys, do I, I don't have my AI glasses. Do I eat this or not? Do I eat it or not, guys?
No, don't eat the mushroom.
Eat the mushroom, that's all I heard. Okay, I'm going for it. Prime, say your thing.
Alright, so the reason why I actually agree with Trash a lot on this one is that I've done a lot of snapshot testing and I think there's probably does exist a world where this makes a lot of sense. It's just all the times I've ever done it. Whenever a requirement change or I have to make a change that warrants changing the snapshot. I've also had to make some decent, some non minor change also to some logic throughout the program. So when my snapshot breaks, I also don't know if I got it right. So now I'm like trying to hand eyeball through this, this thing like one at a line like, okay, is this, is this really what I meant it for it to be in this moment? How do I know I should change this from true to false? Like I have to re, I have to re reason about everything and I find that true snapshot testing on big things is, is, feels very difficult.
Though I love it, like I want, like ultimately, that's what I always want. I fall into the hey, let's just make a golden trap constantly. Like that is like my default go to is just like, it's golden time. And then I always get sad. As far as like the whole um, unit test breaking and mature product thing, I just find that testing generally, I do not think testing helps for refactoring unless it's a true end to end test. Like you use the product from the very tippy top and you only have the very bottom of the output. You have nothing else you look at. But often I don't like, that's really hard to test because I'll write a function that I know is very difficult to get right. And I just want to test those five things. So I'm gonna make a, I'm gonna make a test that's highly coupled to that
piece. And if you decide to refactor it, yeah, the tests all break. Like that's just a part of it. I wrote it so that this one thing works this one way. And I wrote a test that runs that one thing to make sure you didn't screw it up if you had to make a minor bug fix to it. I don't try to do that whole um, I just really think this whole idea of unit testing and oh, it will prevent you from having headaches while refactoring, has just been alive from the beginning. And then what you'll hear is people do the exact same thing every time. Oh, that's because you're testing at the wrong level. It's just like, I'm just testing the things that I want to test. I don't test all the things. And what happens if you're testing on the wrong level just simply means that you broke a whole bunch of tests, but then some of your tests didn't. And that gave you the confidence that you did the right thing. And I just don't, I just am not going to
test the universe. I'm gonna have a few end to end. I'm gonna have the hard parts and that's that. No more.
I also think like, I mean, when I'm thinking back to times when I've, when I've done testing, uh, like, a lot of times when I, when I will break out testing, it's for things that I know aren't really all that refactorable or changeable anyway. It's like, okay, I, you know, I made this memory allocator. And we pretty much know what it does, right? You have these like two ways you can allocate memory from it and you have one way to free or something like that. And then I just have these tests to make sure that, you know, it doesn't end up in bad states or whatever it is. And, and that's a really useful thing, right? I've definitely done that. I do that for math libraries. Like, we kind of all know what the sign of something should return.
So if I'm making a math library that implements the sign function, that's not like, I don't have to think too hard about refactoring sign because math has been around for like, you know, hundreds of years and isn't going to change tomorrow. Like, hell, bad news, guys, like React updated and now the definition, the mathematical definition of sign is different today. So I'm so happy that's not even a possibility, that React does stuff with math because we'd be screwed.
You know, guys, there's still time. There's still time.
Yeah. So, uh, so I do, I do like them for that. And, and actually, like, there was a, I can think of a specific time, uh, when I was shipping game libraries, when, uh, we had a really bad bug, uh, that caused a problem for one particular customer that took forever to find. And it was because it was in a thing that I had added to the library just for that customer as like a backdoor, right? They, they like, we need this one thing really quick. And I was like, okay. And I put it in there and it had a bug in it. And of course it wasn't in any of my standard testing or anything. It wasn't even really in anyone's use case, right? And, uh, and so like, you know, that was a, to me, I was like, okay, that's a pretty,
that's pretty good evidence that like the testing that I was doing before was good and like wasn't a waste of time because like the one time it didn't happen, it caused a significant problem, right? And so, you know, presumably I would have had more of those in actual other customers had the main, had all the stuff that normally shifts to the library not been tested in the way that I thought it should have been. And, you know, from then I was like, so like, you know, don't, never do a one-off ad. Like, if I'm going to add something for a customer, I'll add it like the right way and actually put in the testing for it, or I'll just tell them, I'm sorry, I can't do that right now because, you know, I'm too busy or whatever. And so I do think like, yeah, when you can actually talk about this is isolated, this is what this thing does, I'm pretty sure it's not going to change.
I think the tests are great, but before then, yeah, it's just, and also. This was just kind of going off on a random anecdote, but just to circle back. The thing that Prime was saying is absolutely true. It's like refactor testing? Not really. Like, that's only... You can only really get like a very high level confidence from that. You can't really know. There could be all sorts of things that aren't really working. Because whole system testing like that is gonna miss... It's gonna miss like weird edge cases that can overlap. And the only way you catch those with test-driven development is knowing about those edge cases and making tests to kind of target them. And you can't do that if you just refactor everything, because now the edge cases are different, right? So, yeah, I think TDD people overstate the degree to which that really works.
That's why I'm a huge fan of PDD, prog-driven development. I think that every time you release and you have a bunch of people that use your product, you should do a slow release, you should measure the outcome, because it's just gonna test all the weird things you're never gonna be able to think of or set up. And if it starts breaking, you know that you screwed up. Previous version worked better than the new version. You can't release this new version. You gotta back it up. Not P Diddy, PDD, okay? Stop saying P... There's no baby oil in this situation, okay? But I mean, to me it's like-
Shots fired at Sonos, or Sonus, or whatever that company is called.
I don't know what they did.
You guys are supposed to be the web dev people. What's going on?
What the hell are you talking about, Casey?
Sonos, the company that makes the speakers.
Nobody? Boz? Are you kidding me? You guys are the web dev guys!
Beats by Dre?
No, this is legendary. Like, everyone knew about this. So, there's this company called Sonos. They make like the most popular internet-powered speakers in the world. Like, everyone has these things. I mean, not me, because there's no way in hell I'm connecting speakers to the internet. That's just a way to make them not work, which Sonos went ahead and demonstrated for me. They had a product that everyone liked. They were extremely popular speaker. They then did a big software update where like they were like, "We're rolling out our new system." Like, you know, kinda like how Google does every couple of years. They're like, "Hey, guess what? Gmail's gonna suck way harder than before today." Like, you have five, you know, you'll get five months of being able to switch between them and then you're on to the new bad one. They did that, only there was no five months. They just basically said like, "Here's
the new rollout." And it was awful. Like, tons of feature regressions. It didn't really work. Customers couldn't do most of the things they used to. And it like tanked the company. Like, it was just like, immediately, everyone hated it and nobody wanted to buy their stuff and they had to do this walk back. And they couldn't roll back because they had redeployed all their servers and I guess they didn't know how to like, undeploy them. So they couldn't, you couldn't go back to the old app. It was, "Go read about this. I can't believe no one knows about this but me." It was a massive failure.
Casey, I know it. I know it. Never heard of them. I heard it. Casey, I don't know if you guys can hear me, but I heard that. Okay.
We can hear you. We can hear you, TJ. We can't really see you. You have a very few expressions.
That's fine.
But I do, okay. Your point is that hardware is definitely different than software. You know, if Twitter has a small breaking thing because they made something, Twitter, all these major sites, they break all the time, right? I have much more forgiveness for that. This is, you know, that's obviously different than, you know, you gotta play the field a little bit different. When I say break production, if I'm building a game that only, you know, so many people play, you don't want it just crashing non-stop while you're testing features. There's probably some levels. But if you're Twitter and all of a sudden you can't post for five minutes and only .001% of the people couldn't post for five minutes, it's just not gonna make a huge splash and it's probably not gonna deter them from actually using the product again.
Well, I was actually, no, I was agreeing with you with the Sonos thing. What I was saying was, they should have done exactly what you're talking about. They should have And deployed a small server bank for what they were doing.
Oh, I see.
Given it to 1% of their users and their users would be like, this is utter garbage and I do not want it. And they would have been like, okay, pull that on back. Nobody saw it. We're fine. Right? So exactly what you're saying, I was totally agreeing with you. I'm like, so no, if they had done that, they would have totally saved their company, literally like six months of hell. Had they just given a slice of users this thing for a while, they would have been, you know, some significant percentage above beta test, right? Like, so like you said, you know, 5, 10% of the people got this. They would all been like, nope, nope, nope, nope, nope. And then they would have like, you know, been like, okay, okay. So.
That's all.
I do want to, can I, can I say one thing just about the testing part though? Is that okay? From before?
Maybe.
Sure. That's what the podcast is about. Exactly.
I like this. I like this. Go for it, TJ.
My favorite project that I've ever worked on, NeoVim, by the way.
Can you go back to the grass? You were much more like better in the grass.
Really?
Your house is blocking the towers.
I got a notification that said, your phone is overheating because I was in the sunshine.
I was actually gonna bring that up in the beginning.
Like it literally said, your phone is overheating.
Your body's the one covering. Okay. All right. Sit over, create a shadow TJ. So they can't, the sun can't get your phone.
You gotta plank, plank over your phone.
Hey, what's up, boys? Hey, yo, what's up, boys?
This is like getting into like TJ profile pic territory where it's like on the dating app. Can you tell us some of your pet peeves, TJ? Do you like long walks on the beach?
I'm a, I'm a long walks enjoyer. I'm really not a fan of short conversations. I want to really find out about the real you. Um, I don't know what else.
I don't even know if any of my words are coming through.
TJ is actually just describing himself right now. That's all TJ is doing.
I love laughing. I love having fun. I like to make you laugh. This is just me to prime. This is my pitch for getting on the, on the podcast.
I'll make fun of trash.
Yo, if, if there's a guy named Trash on the podcast, I will make sure to make fun of him every episode. He uses one password. Are you joking?
Alright, alright, alright, alright. TJ, what's your take?
My take is my favorite project that I've worked on and like definitely the one with the most weird edge cases and probably users, Neo Vim. Like, the test sweep for Neo Vim is amazing. We have tons of, we have different like, styles of tests, we have different ways to run them, we have all this different stuff. Tons of work went into making it. And maybe that's just partially because Neo Vim is like, trying to maintain compatibility with Vim on some stuff like literally forever, right? So there's some other requirements that are different. But man, every time I did a PR and a random test broke, it was my fault. It was not the other tests, it was not the snapshot tests, it was nothing else. Like, and like random
stuff. I mean, maybe it's just because Neo Vim's also like a really old C project. So you like, change something and then a random global is destroyed. I don't know, you know, it's like, whatever, it's maybe not always the best, but like, so I don't, in, in my experience where people like actually cared about it and like cared to maintain snapshots and make those happen, like, no one's even getting paid to work on Neo Vim. You know what I'm saying? And like, the snapshots were valuable, the unit tests are valuable, the functional tests were valuable. And I used them in a ton of different PRs and across like really large refactors, adding Lua to auto commands, like doing lots of different stuff where I had to change tons and tons of the code. I was just wrong every time until the test went green.
I think that probably goes to show that tests are as valuable as the people who wrote them. I was literally gonna say that.
Yeah, that's what I've been waiting to say too, is, I just wanted to say, but since I've been lagging, I didn't want to yell. I just wanted to yell "skill issue" to everything Trash has been saying.
I would, I would disagree, actually. I will defend Trash.
Yes!
Actually.
Casey's smarter than you.
Because here's the thing.
That's true, dang it!
I suspect, I suspect that some of what you're seeing, though, is not so much the skill of the people who did it, although they may have been very skilled. It's more just about when the tests were added. So, at least for me, a lot of times what I'll do is I'll go, okay, if I, like, let's say I have a bug that I actually work on for a day. Like, like, this is, because most bugs are like, oh yeah, I know what it is, and you just fix it, right? But you get one and you're just like, what the hell is happening? And you actually, it's like that once every few months where you're just like, okay, this is really weird and I'm not sure what it is. When I find those, I typically try to add a test, or assertions, or whatever I think would have caught it.
Right? Because I'm like, okay, this one's nasty, and I wasn't expecting it. What can I add that will trip this kind of thing up? And I'll add it. And I suspect that much of what gets seen in like, mature products like NeoVim are just like, hey, if you've got some post-test discipline, like, after we made the product, when we're finding bugs, we add the test that would have caught the bugs. Those are often very good. Because they come out of, what are the things that are likely to break when we do stuff? And we put in a net there to catch them. And so, those are like, very good. And they're the opposite of TDD. They're like, oh, they're TDD! They're test-driven debugging. Uh, maybe you could say, or, or something like that.
But, but, uh, so, so I do think there's something to that, uh, that, that's a lot more. I had a couple other hot takes that I'll get to in a second, but I just wanted to leave it, I wanted to say that one, based on what Tej said. I don't know if that's true about NeoVim, but I, I might suspect it, right?
It definitely is. The thing, though, that like, I was more pushing back, I was like, I find the snapshot tests in NeoVim invaluable. Like, I, they're amazing. I find a bunch of these good end-to-end tests. Truly end-to-end, like, literally, it runs a test and does user, I don't know if I just dropped, uh, it does like, user input, and then asserts the outputs. Right? Like... Though I found those super valuable in Neovim, probably just dropped.
I also, no, you didn't, you didn't drop. We saw, we heard your point. I think also to put on top of it is think about the, the horizon of Neovim versus you starting up a new dev tool at work. And maybe there, maybe there's something to be said about when are the Goldens introduced during exploration phase or like Neovim is just not going to be changing. My assumption is that a lot of Neovim doesn't change super hard. It's not like, hey, guys, let's do a whole new data model today.
Wrong.
Changes all the time.
All the ones, that's all the ones that ever, everything you guys have poo-pooed on that I found them the most valuable. When I changed literally the entire data model and how all auto clans work all inside of Neovim, it was the most useful to have the snapshot tests then.
I mean, your testing is going to depend greatly on what you're building 100%.
Sure. Yeah. Yeah. But I'm just saying like, we were fundamentally changing the data models.
Yeah. I'm just saying, that's what happened.
I thought it was a pretty smart statement. Come on.
You're really smart, Trash.
I wouldn't poo-poo that. I wouldn't poo-poo that at all. I, I, you've noticed I have not poo-pooed snapshot testing.
True. Good job, Casey. I love you. You're smarter than Trash.
I love that I'm now getting gung-hoed on it. Can we start rearranging the windows based on who Tej thinks is smartest at any given time? And I'm guessing that it has a very high correlation to who is agreeing with him. It's, it's like I'm getting that sense.
No, I'll just always put Trash at the end. It's fine, Casey. You don't have to worry.
Today I'm going to tweet that Casey protected me from Tej.
Yes.
This is a pinnacle moment of my career.
I can't even see your guys' videos, so I have no idea what's happening. I'm talking to a black screen right now.
I've been just cracking up at your feed this whole time. I just can't handle.
It's pretty good. It's very good. Like the result is, is top notch of the feed.
Unfortunately, I think this will be a stream only feature. I don't know if TJ will actually have this effect on the video because Riverside will be uploading it.
That's true. Yeah, that's true. He'll still be in, in weird like, uh, it is, uh, profile, is dating site profile pick phase, but, but it will be much smoother. Yeah. So, so anyway, uh, I, I wanted to add, I, I, I would agree that snapshot testing can be good. It depends, it just depends on whether those snapshots are going to be stable. I mean, that's the bottom line. If you have. If you're, I would imagine what Prime and Trash were talking about is if you're trying to do snapshot testing at a level below something that's actually consistent across the refactor, obviously that's not going to work, right? And so if your snapshot testing is like someone mentioned a parse tree, well, if it's a parse of something that could be parsed different ways depending on how you chose
to implement that subsystem, those snapshots are useless. If the parse always has to be the same because it's like parsing for the C language specification and it can't change because like, you know, it's hardcoded, so if the thing's in C99 mode, then it had better produce a exact parse tree, that's a great snapshot test, right? And so I think the difference boils down to those things. It has to be about something that actually is, you know, not going to change, item potent to the, to the release of the project, otherwise it's a limited time test. But, you know, that's just how it is, so.
Probably also the scope of the breaking, right? If only 10% of your golden is invalid because you made this change, that's a lot different than 100% of your goldens are invalid and you have to like rework all of them because of big kind of refactors. That was my last, my last experience is that I had to do a bunch of testing against this games reporting thing at Netflix. And to be able to do this autodoc and to be able to do it, I had to like create the data model of the incoming events. And then they were changing the incoming events like once a week because it was still in exploration phase. So it was just like, well, not only is my input incorrect, now my output's incorrect. So I have to like change both sides. So I'm just like, I guess I'm just redoing all the things again. Here we go.
And so then it just became super hard to know if I was actually right or if I was breaking things.
So I, I think at the outset, I made it pretty clear, I don't like test-driven development as an idea. I don't like focusing the developer's attention on tests. I think they should understand tests and know how to deploy them smartly, but I don't think, you're right. I do want to say there is a place where I do recommend literal test-driven development. And it is as follows. That is, if the programmer is finding that they are low productivity on a particular task and that task does not provide feedback. So, for example, if you have to implement some low-level feature component that has no visual output, no auditory output, no anything,
it's just like an abstract thing that happens inside the computer, right? Then sometimes programmers, like, they get bored working on these sorts of things. But a lot of programmers will like focus on things like numbers that come out of a thing. Like if a thing, if you run a thing and the number like, you know, 37% pops out, suddenly the programmer's like super motivated to get it up to like 45% and 50%, and so, right? So sometimes it's like, oh, okay, if you can turn this problem, if you can make a test that turns whatever you're working on into something with a concrete output that you feel some reward about when it goes up or improves,
that can be a valuable use of test-driven development. And we do this in performance a lot. We'll put, make a little test around something that outputs things like, "How many cache misses did it have?", or "How many, you know, cycles did it take to complete?", or whatever. And it's like, "Ooh, now I'm like trying to get that number down", and that's a lot more fun than I'm just trying to make the system faster, or whatever. So like, a lot of times you can get a, you can make it so that those things go together, and then you get the test, or the good statistics gathering, or whatever it is that you actually wanted anyway, and you get this motivation for the programmer, and it, they all kind of work together in this nice way. So I just wanted to put that out there. There's one place where I think it does kind of work to drive development by a test of some kind.
First thing that comes to my mind is like code coverage reports. And you see that number really low, and I have the opposite reaction where I just don't care, and I want to do it.
I don't think I've ever saw like a low code coverage report and I was like, you know what? Today, we're making that number higher. I'm kind of just like, guys, we suck and I'm not gonna help us get better. There's no way.
Oh, no. Alright.
I, I mean, to be fair, if I go to a place and it's like 100% code coverage, I also go, uh-oh. Right? Like, I'm also kind of worried about the 100% code coverage, uh, code cases as well.
I'm not gonna say the company I was at. I've been at many.
Netflix.
And there was a team that required 100% test coverage to the point where people were just like writing the most worthless tests just to get like this like PR bot to just shut up and was
and to make it even like funnier, there was like bugs every day, like in production. It's just like, I was just like, what's happening here? I don't know if anyone's ever experienced like 100% like test coverage quota by their team.
Yes, I did it with Falcor when I was at Netflix. 100% code coverage, bugs every day of my life.
Oh, for Falcor, the one line production takedown?
Wait, Fa- do you say Falcor?
Uh huh.
Like the luck dragon from the never ending story?
What? Why was it called that?
You don't know what Falcor is, bro?
I do know what Falcor is. It's the luck dragon for the never ending story.
Yes, it was named after the luck dragon from the never ending story.
Tell the story, Brian. Tell the story. Tell it all so Casey can hear and then we'll be done.
Alright, I'll do a quick one. Casey, Casey Snow, how, how it works is that you would provide effectively an array with keys in it as the path into your data model in the backend. So say you want videos, some ID, title. It would return to you the title in that kind of as a tree. Videos or as a graph, or maybe as a forest is a better way to say it. Videos, ID, title. So if you wanted, say, a list, I want list 0 through 10, title, it would go list 0 through 10, which would be IDs, which would then link to videos by ID, title. So it's kind of like a graph-like data structure that makes it so that if you make duplicate requests, it's able to go, oh, okay, I already have videos, whatever. I already have list item 0. I can kind of not request so much from the back. The problem was is that what happened if list seven doesn't exist?
You don't know, so we have to return something back to you that says it does not exist. So we returned something called like a, it's a boxed value. It's an atom of undefined. It lets you know, hey, there is a value there and the value there is nothing. You don't need to request it again. It's not going to be something. So we created something called materialize. On the backend, that means you need to materialize missing values. Well, what happens if you request list zero through 10,000, zero through 10,000, zero through 10,000 title? Well, you just created a gigantic number that were going to materialize everything for you. And so with one simple line of code, a while loop in bash, you can DOS or at one point 2016, you could have DOSed down and took down Netflix permanently. There was no rolling back. It was in production for like two years.
You could do it for all endpoints, for all devices, for everything and just it would never run again. And that's it. It was a simple while loop. And I wrote a lot of beautiful code about it and I did a lot of beautiful stuff with it and it was very nice code. And then I also discovered it and reported it and then later on it got named the repulsive grizzly attack.
Repulsive grizzly?
So has no one at Netflix watched the never ending story? Shouldn't it be called the Gomork or something like it's a movie. It's probably on Netflix.
Okay, that, no, no, this wasn't some, this wasn't some 9,000. It was a completely separate team that when they found out about it, they had to do a bunch of work and and it ended up being, you know, a fix on the Zul gateway and all this kind of stuff.
Alright, because there are no grizzly bears in the never ending story. I didn't read the original book because I think it's in German or something. I don't know.
Yeah, yeah, but the nice news is I originally didn't come up with the materialize idea. But I did do a lot of refactoring that while refactoring and recreating everything, I stopped and went, wait a second. Something seems funny here. So then I trashed staging for a day where no one could use staging for a day. And then that's when I realized we made a mistake.
When I say we, I mean me.
And that's it, and he's off.
He's like.
I told my story and I'm not participating in this podcast anymore.
The transitions at the end of this podcast are just getting worse and worse. He's end stream, he walks off frame.
End stream?
What's happening?
All right, TJ, do you have anything you want to add to this all?
He's frozen with a smile.
Is stream done? Are we still live? Yeah. We're still live?
TJ's smile, dude, find a friend that smiles at you like TJ smiles at us. I think today's big takeaway is don't bring Trash Dev into a project that needs help. He'll just quit.
Yeah. No, actually, I will help your, your product get better.
Bye.
He'll drive your code coverage metrics down to zero.
Code coverage is a lie anyways.
Are we live? And then quit. Oh, TJ's back again.
All right. Well.
What is happening?
Don't know.
See you later.
No blockers on my end.
I think we had a really productive meeting today, everyone.
Everybody enjoy your weekend.
Next quarter is gonna be big.
It's gonna be the biggest quarter of our lifetime.
Absolutely.
We gotta hit our OKRs.
Well, okay then.
Another professional podcast from the stand up.
All right, I'm ending it. All right, hold on one second, let me, I'm gonna, I'm gonna end the stream. Thanks, everybody for being around here. TJ is back, big, whatever TJ is.
Page updated November 3, 2026 · transcript and summary are machine-generated