Pod Insights Syntax - Tasty Web Development Treats 950: Even SCARIER Web Dev Nightmares (Spooky Stories Pt. 2) About this episode In part 2 of this year’s Spooky Stories special, Wes and Scott discuss the most chilling developer horror stories—from six-month-old unprocessed donations and runaway dog-food orders to vanishing databases, DNS disasters, code that literally tore apart a mall’s ventilation system, and more!
Show Notes
00:00 Welcome to Syntax!
01:33 A Bug Beyond the Grave
04:16 NHL SPOOKS
06:36 White Space Ghost Faced
10:54 Over Order Nightmare
16:50 Alaskian
21:16 Brought to you by Sentry.io
22:50 Rackspace’d Out
25:02 Fired
26:52 WordPress Woes
33:21 What does the P in VPS stand for?
34:18 Beyond the Grave II
35:39 The Hottest Hot Fix
37:54 Bad Redirect
40:03 Instead of Making Money, You Spend Money!
41:26 Certbot Certain Death
43:55 It’s Always DNS
50:02 Cache Ruins Everything Around Me
51:52 Fiber F-Up
56:18 More Spooky Stories
Don’t Drink and Deploy 😬 | Spooky Dev Stories Pt. 1
Submit your Spooky Stories
Hit us up on Socials!
Syntax: X Instagram Tiktok LinkedIn Threads
Wes: X Instagram Tiktok LinkedIn Threads
Scott: X Instagram Tiktok LinkedIn Threads
Randy: X Instagram YouTube Threads
Transcript Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Wes Bos? Welcome to Syntax.
0:04 Wes Bos? We've got our second episode of 2025 Spooky Stories. These are stories of web development and software development in general of just awful things that have happened in your career. You've taken down websites. We've got some real doozies coming up, just awful stories, and we are going to read them. If you have your own spooky story, please submit it at any point. Go to syntax@fam4/s spooky and you can submit your story to us and we'll read it on next year's episode. We're always collecting them and always love hearing them. So we have a sentry plug in one of these stories, so I'm going to leave it for that. Is that, you think that's all right, Scott?
0:49 Wes Bos? It's actually a really good one.
0:51 Scott Tolinski? I want to read the ad.
0:57 Wes Bos? I want to read the ad. So if you didn't catch the last one, and if you're listening on audio, Scott has like a very good vampire costume, real blood.
1:08 Scott Tolinski? Thank you. I was doing this costume and my wife said, "Do you have a plan?" And I said, "No, I don't have..." What is this? I'm losing the accent. "I don't have a plan." And so I just started mashing the face paint around and what you see here is what you get. And you know what? I think that's a pretty good job, actually.
1:30 Wes Bos? You look great.
1:31 Scott Tolinski? Thank you.
1:33 Wes Bos? All right, let's get in on the first scary story is a bug from beyond the grave. By the way, I wrote all the titles for these ones, so please appreciate them. "Once, I built a website for a foundation that did medical research. People could donate money for special occasions like birthdays, but mostly in memory of somebody who has passed away." That's nice. "Everything worked perfectly at launch and I even tested it myself. The payment showed up as reserved on my card." All good. You know, like it reserves the money in your card. It hasn't actually gone through. Sometimes you need to do that. That's a part of credit card processing. Six months, oh my gosh. "Six months later, I'm at my summer cabin with a beer in hand and the grill going when the phone rings.
2:18 Wes Bos? None of the donations have come through. A donator was complaining that their donation never came through." Strange.
2:28 Wes Bos? After some panicking and digging through, all the test suites were still passing. I realized I'd done everything right, except one tiny thing. I never hit capture. The payment provider, thankfully out of business today, had a special endpoint for capturing the payments. That's Stripe is the exact same way, by the way. Auto capture was optional, but not default and very hurried in the documentation. So all those donations and memory for dead people have only been authorized, but never actually charged.
3:02 Scott Tolinski? Oh, no.
3:03 Wes Bos? Oh, so people have been donating for months to honor loved ones and the money never moved. Total panic. Imagine your entire business is taking money for people. You don't do the one thing you're supposed to do. But luckily, the payment gateway kept pending transactions for six months. It had been five months and 20 days.
3:26 Scott Tolinski? No.
3:27 Wes Bos? I wrote a quip script to capture everything and sent out an email to everyone affected. In the end, it all worked out. Ooh.
3:36 Scott Tolinski? Spooky. Oh, oh, this one, because it involves the spirit of lost ones. I am going to say this one is a nine out of 10. That is very spooky to me. Oh, man.
3:48 Wes Bos? I'm going to give that one a nine out of nine and a half. That was, I just, the fact that it not only is awful and includes money, but also includes like something sensitive, which is donating and memory of loved ones.
4:02 Scott Tolinski? Yeah.
4:05 Wes Bos? Yeah, next one's about the NHL, though. Maybe those make us feel better.
4:09 Scott Tolinski? Oh, go about the NHL. Go grab my hockey stick I got over there. I got a hockey stick in my office, Wes. Oh, yes. The next one here is NHL spookiness.
4:22 Scott Tolinski? I built a very nice website for the National Hockey League. A whole team of content specialists, about 10 people, start filling the site with content. The day before launch, the whole team even starts working from 6:00 in the morning for their last big push of content. I export a single table from our staging site. Not content for a special feature. To SQL and blindly import it in the production database. Oh, blindly import it in the production database. What a sentence. The whole site goes down and my new table is left. Only my new table is left. Oh, man.
5:03 Scott Tolinski? It turns out the dot SQL export didn't just contain a create table query, but also a drop or all drop table queries.
5:18 Scott Tolinski? That sounds like an AI style migration, if you're asking me. That's a... Oh gee, so they ran a SQL command that just dropped all the tables and then added their new table. The latest database table was from three days before, so all of the content was gone. Yes, the launch was delayed. Yes, I was ashamed. Yes, I always check my exported dot SQL files from now on. This is now more than 10 years ago and I still get chills thinking about it. Oh, there's no backup strategy here? What, everybody had to re-enter their stuff or what? It's just gone?
5:53 Wes Bos? Probably. Yeah, that's awful. If that was me, I'd just be like constantly backing that thing up, you know? Just every, every 20 minutes backing it up. Cause that's, is very spooky to have that happen, especially when it's such a big, like the NHL, you know? Like there's 10 people working on it. This is, that's, that's a lot of money in, in paying people salaries.
6:17 Scott Tolinski? Yeah.
6:18 Wes Bos? Like, brutal. I mean, I want a seven out of 10 spooky, spooky bookies.
6:24 Scott Tolinski? Spooky bookies. I'm gonna give it five Jason masks out of seven Jason masks. Uh, he did wear a hockey goalie mask, if you don't know about Jason.
6:34 Wes Bos? I never know what you west, so. The next one is entitled white space ghosts faced. I'm a full stack dev that writes mostly typescript, of course, like any good full stack dev. I'm a seasoned staff with lots of experience writing backend code, managing infrastructure, writing helm charts, setting up alerts, yada, yada, yada. So I'm not afraid to jump into some Python code base and make some changes. This is sounds like something I would say. I've been writing a lot of Python lately. The Python service I've been working on was originally written by a messy ex-Google engineer, and I'm stuck with them. And a broom cleaning up the mess. Yeah, typical Google employee writing a bunch of messy Python.
7:16 Wes Bos? This particularly messy Python service was hitting Google's cloud video debt generation model. Okay, so this is relatively new. So I was cleaning up some of the messes. Every time video creation fails, multi-line logs are barfed up over the standard out in using a print statement instead of our structured logger. So I updated the main.py file in our code base to use our logger instead of print. So doing so required moving around a few things, etc. Makes sense. I opened up a PR, I ping my trusted front-end dev to rubber stamp it, and logs are pretty and people can create VO videos through the API. Everything is good. All right. Three weeks go by. That's a long time, folks.
8:00 Wes Bos? That's a long time for this, the just wait part. And I see a group of people crowded by a laptop on my way to the office, and the CFO is standing by them, looking moderately perturbed. They are wondering why our VO2 costs last month has shot up 5X. Certainly nothing to do with my innocuously logging change, right? Well, time to find out. My little cleanup PR introduced a line that unindented the message acknowledgements outside of the try catch block, resulting in failed video generations. Example, fail due to safety checks. Yeah, like you try to create a video with something that is is not appropriate, they'll they'll fail it.
8:43 Wes Bos? To be tried, retried five times. Just for reference, a 30-second video generation on Google costs about $10.
8:54 Scott Tolinski? Oh, yeah.
8:55 Wes Bos? It's it's an expensive API, so this literally had cost the company hundreds of thousands of dollars. The crazy part is it was just a white space change. It didn't even show up on our code review platform, which is Graphite. So the DevI Ping rubber stamped the PR, didn't even notice the change. Friggin' Python with his indentation. I used to be such an indentation fan, you know? I used Pug and Stylus and CoffeeScript and everything was indentation based.
9:23 Scott Tolinski? Same, yeah. I liked it.
9:25 Wes Bos? No more.
9:26 Scott Tolinski? I liked it. I don't know if I'll still like it, but I liked it at the time. That was the style then.
9:31 Wes Bos? I would blame him. I would blame Graphite, but ultimately I have to be forthcoming and put my ego aside in such situations. There is only one person to blame here. The Python programming language and its effing syntactic white space.
9:48 Wes Bos? He queued it up to blame himself. He just says it's the Python problem. So, yeah, lots of yapping to say one invisible line change caused us hundreds of thousands of dollars to the Google Abyss. Thankfully, my colleagues were quite forgiving and I'm still working hard to redeem myself in the finance department. I've definitely been given a shorter leash. The head of engineering.
10:12 Wes Bos? Oh, you gotta have some alerts set up or something if things were to to go awry.
10:21 Scott Tolinski? Oh, bro. Yeah, I get that. I get that one about four Dracula fangs of which I'm going to be taking out. So that way I don't have permanent damage done to my lips here.
10:31 Wes Bos? That is not the most expensive goof up we've had. We've had a couple of stories in the past that were in the millions of dollars because they shipped out physical products to people who did not pay for them. But that is still a seven oofs out of eight, I think.
10:52 Scott Tolinski? You think seven oofs out of eight? Okay, next one here. Overorder nightmare.
10:58 Scott Tolinski? I had been working at a premium pet food e-commerce store. To give you an idea, we average about 4,500 orders a day with about 300 of that amount were subscription orders. Well, I subscribe to some dog food, maybe I'm a customer. The customer had an option to start a subscription of their pet food orders. This would create an order on the interval the customer set him or herself. However, the person that created the whole subscription system went into a burnout and eventually left the company. I joined when that person was already burnt out. Let me tell you, working in reoccurring billing without like tools to do it for you can absolutely
11:42 Scott Tolinski? drive you to burnout, absolutely.
11:44 Wes Bos? Billing. What's the worst? Can we, can we ask this? What's the worst parts to work in? I'm gonna say.
11:53 Scott Tolinski? Reoccurring billing.
11:54 Wes Bos? Single sign-on. Reoccurring billing. Single sign-on and maybe auth in general. Access control.
12:02 Scott Tolinski? I don't mind access control either.
12:04 Wes Bos? I think those are probably the worst.
12:06 Scott Tolinski? Anything that involves kind of like ping ponging back and forth, but like with reoccurring billing, like testing it, you have to be, it has to be done through webhook. So you can send a test webhook and then like when an actual subscription like webhook comes in, yeah, you could have tested it, whatever, but it's still like, it's, it's still always very scary for, for like what actually is, is doing that. There was a big demand of the customers that they could choose which day of the week they would receive their food as hybrid remote work was very popular in our country. They wanted to receive their food when they worked from home. This would be for regular orders and subscription orders. For example, for regular orders, you could choose that you would receive it the first Thursday after placing the order.
12:50 Scott Tolinski? This got really complex as we sold it to about 20 countries. Oh yeah. Oh, oh, oh. We worked together with different shipping companies depending on the warehouse that order came in from. These shipping companies all had different lead times for each country. Eventually, we worked this out and thinking about the implementation of the existing subscriptions.
13:14 Scott Tolinski? For running subscriptions, we had to calculate what was the last day of the week they received their last order. Then we calculated the next shipping dates that would receive on the same day. But with the given subscription interval calculated into it, we had to keep in mind the subscriptions were running so that we could need to calculate the time from their last order until now and subtract that from the interval they had specified. We then took the closest day calculated from the next shipment date and would then change the shipment date for the existing subscriptions. Oh my gosh. Bro, I hate subscriptions with digital products. With shipping and then shipping dates that things have to arrive in, take me out back. I'm done. Logistics, man.
13:57 Scott Tolinski? Logistics. I don't like logistics. I'll tell you about that. I don't know. I'm good at planning. Jeez, that's crazy. At the time, I had been there for about a year. The subscription system was a bit of a gray box for everyone, where we could change some things, but nobody on the team had full context on how it all worked exactly. And there was no documentation. As I said earlier, the original creator had left the company. I had been fixing some bugs in the system at my time there and felt confident to implement this new feature. I took some days to get familiar with the system and wrote out exactly how I would change it in the areas that would be touched. From running subscriptions, I wrote a migration strategy that discussed everything with the team and my manager. Everybody gave the green light.
14:42 Scott Tolinski? We tested the implementation. We put this into production. Everything ran smoothly and we were getting positive feedback from our customers. Oh man, the hammer's about to drop here. But after 10 days, we were getting complaints. People who had subscriptions and had their next order delivered in these days had been paying every day since the feature launched.
15:05 Scott Tolinski? That's, that's the hang I knew was waiting for it. They'd been paying every day. Turns out, we had been calculating the next order date wrong, and once the new date was calculated, the next order date was placed on the current date, as this is a running system and had been going for 10 days before we noticed. Thousands of extra orders created, packed and shipped.
15:28 Wes Bos? Oh no! And it's, it's not even like you're getting like, like a hard drive or something fun. It's like, like dog food. It's so huge.
15:39 Scott Tolinski? I bet. I would take 10 extra orders of dog food because you know they're going to refund it at this point. Luckily, I created a backup of the database when I launched the feature into production so we could see who had upcoming orders, and we then had to spend the next week with the whole team figuring out which customers had placed how many extra orders to reimburse them all.
15:58 Wes Bos? Oh.
16:00 Scott Tolinski? This, I give this one, uh, 9 demon pups out of 10 demon pups.
16:06 Wes Bos? Brutal. That is, and you know, like, the poor support people as well, they're just going to be going through it for the next, like, weeks or months, and like, probably lost a bunch of money on the dog food. And I, I would be mad if I got all that extra dog food, because like, where do you put that, you know? I guess you could give it to a, like, donate it to a shelter or something, but oof. Hopefully that's the silver lining here. Some shelter got a bunch of free dog food.
16:38 Wes Bos? By the way, we keep all of these anonymous. Um, this one includes the company name, so I'm just going to change the company name for the sake of anon- anonymity.
16:48 Scott Tolinski? Anonymity.
16:50 Wes Bos? Anonymity. The year was 2019. I was on the experimentation platform team at the time, but was given responsibility over everything analytics after being told it should be in the "keep the lights on" mode, which is minimal development, just keep it running. The service API was based on Segment.io in order to leverage Alaskan-specific wrappers. Change the name.
17:17 Wes Bos? Wrappers around the clients. Alaskan developers had been given a halt order for feature development after being plagued with reliability issues in the front end that devs were flying blind to. It was decided that the best way to give front end quick visibility into performance and reliability was to leverage the analytics platform. The front end JS client used a library called segment local storage retry in order to capture as many events as possible. This is pretty common when you're sending events. You'll throw them in like a variable or you'll throw them in local storage so that if there is, if they're blocked by like a thing, if the person goes offline for whatever reason, at least they're in local storage and you can retry over and over again.
18:02 Wes Bos? This uses multiple local storage entries to manage the queues for processing and to be able to recover queues from tabs that have been closed. That's another good reason for it. We started getting reports from internal users that local storage was filling up and it was due to the analytics client. After digging around, I found that the client would detect that local storage was full and switch the entire storage solution to memory, which prevented any further cleanup inside of local storage. Once we got that fixed out, we pushed it to all products only to start causing more instability to our backend analytics service. So it took us a while to realize that we would occasionally be seeing single analytics
18:47 Wes Bos? event be able to be duplicated tens of thousands of times, which was also causing additional slowdowns on the other clients. It was a hard case for us to get budget to add more fixes for a service that should have been in maintenance mode, but the volume it was needed for processing grew from 1 billion a day to 3 billion with the influx of additional monitoring front end teams that were trying to be shoehorned in. Yeah, this is, this is monstrous. Like we've had folks on the past from Sentry when it's just like. Like you're essentially getting DDoS. Like being able to just receive the events from everybody is a massive, massive thing.
19:27 Wes Bos? Soon, it became apparent that the fix was causing more slowdowns on client browsers and also sometimes prevented garbage collection from running due to the frequency of local storage checks. This is really interesting because local storage is a synchronous API. Because so every time you're touching it, you're, you're actually running a synchronous function, which if you do it often enough, can start blocking the main event loop. In testing, I even managed to crash my entire laptop.
19:58 Wes Bos? I was thinking, we should do a video of who can crash the browser tab first. Who can write the worst code to crash a browser tab? I think that would be good.
20:07 Scott Tolinski? I can crash your browser tab in seconds. Melissa, I can crash faster.
20:12 Wes Bos? Oh, you know how to do it?
20:14 Scott Tolinski? You just do a, I mean, just use, use effect, huh? Just use effect.
20:19 Wes Bos? Use effect. We tried a few hacks to get on top of this, but at the end of the day, we were trying to treat local storage like a transactional database and it became very clear that only one thing we could do is move the functionality to index DB, which removed a lot of complexity and overhead and almost all of the issues went away after that, except for the fact that many devs kept throwing more and more load at it because the analytics pipeline turned into a hammer for many problems, but that's a different story. Moral of the story, don't treat local storage like a transactional database. Yes. Yeah. As much as, especially like you can, you can run out, you can run out of space in local storage and every time you write to it, it only holds strings.
21:01 Wes Bos? So you gotta, there's like a JSON parse you gotta pay for there. And a second story here is the analytics is not designed for platform monitoring. I think there's another tool for that, like, dot, dot, dot, sentry.io. Why, thank you for the, the ad there, dear friend. I don't know what your name is, but yes, you guys should be using Sentry at sentry.io forward slash syntax and get two months for free using the coupon code Tasty Treat.
21:31 Scott Tolinski? Yeah, take a bite out of your errors.
21:35 Wes Bos? That's good. P.S. The main problem was Overlord Alliance on the analytics problem kept growing despite being in maintenance mode and it hit a peak of 13 billion events a day before Dedicated Team was assigned to rein in the cause. Oof, I can't imagine the bill for that, man.
21:54 Scott Tolinski? Be with a billion.
21:56 Wes Bos? Billion. P.P.S. I'm no longer at Alaskan.
22:02 Wes Bos? To this day, there is still some code that will attempt to reclaim events from local storage despite no new analytics being added to local storage since the start of 2021.
22:15 Wes Bos? Oh, man. Man, man, man. I give that nine and a half out of 11. That was good.
22:24 Scott Tolinski? Yeah. Yeah, I would, I would say that is, I would say that is a nine out of 10 great white sharks that are infected with some kind of zombie disease. So they're
22:34 Wes Bos? Zeke in the chat just said that's 150,000 events per second.
22:39 Scott Tolinski? Oh, what?
22:40 Wes Bos? It's a big, that's a big company, Alaskan. I think they have every person in Alaska as a customer.
22:47 Scott Tolinski? They, a lot of them do go skiing, for sure.
22:50 Wes Bos? Yes.
22:50 Scott Tolinski? Alright, the next one, rack spaced out. I'm gonna little Arnold there. I run the dreaded R.M.R.F. forward slash star when I meant to run the R.M. hyphen R.F. dot forward slash star.
23:11 Wes Bos? Never, never do this.
23:16 Scott Tolinski? What? On the production server of a large commercial real estate company. I didn't use Git or any other version control. Backups had been offline for years. The server was completely trashed.
23:33 Scott Tolinski? But the rack space tech was entertained. Can you imagine getting this email? They'd be like, hey, brother, I just ran RMRF on my entire server. 23 year old me was thoroughly impressed that the dude stayed on the phone with me for hours. Ended up spinning up a whole new dedicated server and moving over what pieces he could to the new box. I lost a few days of data in the end, but got most of it back up. Well, that's, that's a lucky end result. One time I ran get clean on my root directory. I did that one time. I think I talked about it in one of these spooky episodes.
24:14 Scott Tolinski? And then I deleted a bunch of video production files I had going and kind of borked my system, but didn't. I noticed what I was doing, so I, I was really quick on the command C. But I still, it's still chomped through some stuff. Either way, very scary.
24:28 Wes Bos? Everybody go right now and check that your backups are working because man, they often are not. And when you need them, it's not the time to find out they are not working. And also like shout out to like support engineers or like the technical support staff that actually know what they're doing. Cause they're able to, Sentry has a really good support team. I've been to the Toronto office several times and you just like listen in on them trying to help everybody with all their problems. And they know about everything. It's crazy.
25:01 Scott Tolinski? Yeah.
25:02 Wes Bos? Next one's called fired. Before I worked as a web dev, I worked in IT doing desktop network and server support. I had a job working for a financial company in San Francisco. My job was to prepare laptops and other hardware for new fire, new fires, new hires. The process involved wiping the ORM images and loading custom images on my colleague and I maintain depending on the laptop or in rare case desktops. My job was very routine and robotic. I loved it. One day, however, the CEO's laptop was brought in for servicing and definitely not a wipe sadly. In the routine of my job, I formatted a CEO's hard drive. So true, you do something every single day.
25:47 Wes Bos? We have a backup of his laptop before, but it was not one of the CEO's laptop that was backed up. I was looking for work the next day.
26:00 Wes Bos? Oh, that sucks. That sucks so much. Oh, man.
26:05 Scott Tolinski? CEO's hard drive. I can't think of like a worse thing to make an impression with. Oh.
26:13 Wes Bos? Man, but like, I don't know if these big companies can do this, but like I have rolling backups running in multiple places all the time because this is bound to happen at one point.
26:26 Scott Tolinski? Yeah. Yeah. I mean, you gotta have a backup. The IT people should have been making sure that that CEO had backups or whatever, but still, uh, very spooky. Yeah. Getting fired for something very spooky. I would say that this is a, you know, a six out of eight killer snowman for sure.
26:43 Wes Bos? I would give the same as well. Killer snowman.
26:47 Scott Tolinski? Killer snowman. Jack Frost's, I believe. I don't, that's a movie, but I haven't seen it. WordPress woes. I was doing some freelance work for WordPress when I was on a hot streak with quite a few jobs done and all five star reviews until this job. It was actually pretty simple. It was just a WooCommerce site that needed some minor changes. So I did what I always did and created a backup, a staging site, et cetera, and got to work. I completed everything, uploaded onto staging and then got the client to overview it with a green checkmark. Awesome. They were using SiteGround, so I was able to just hit the sync button, merge the staging sites to production, forgetting that this was an ecommerce site.
27:31 Scott Tolinski? I just overrode the database entries for all of the last few days of purchases. There was no way to tell who had placed the orders or when or for what. Suffice to say, the client was very upset and I still got lucky. I still got away with a three star review, even though it should have been a lot worse. I would imagine what they would have to do is like go through amounts and like guess and then contact the customers and be like, our dumbass web developer screwed the pooch here. What did you order? Yeah, that, there's no way to fix that but manually.
28:05 Wes Bos? We've had this, this many times, this story of someone who did it and they've had to like re- Build everything from like emails sent out.
28:15 Scott Tolinski? Yes.
28:15 Wes Bos? Hopefully you can go into your, your sent emails or logs or something like that. But just, it's always a tricky one, especially like, it's not a WordPress specific one, but a lot of people hit this issue where I have config that needs to change in the database, you know?
28:30 Scott Tolinski? Yeah.
28:30 Wes Bos? But I also have content that lives in the remote database. And sometimes you bring the remote one locally and sometimes vice versa. And you gotta be very careful doing these migrations.
28:43 Scott Tolinski? I always found that to be very scary when you have that much, like with WordPress or Drupal, like doing migrations in that way where the database is such an essential
28:52 Wes Bos? Yeah.
28:53 Scott Tolinski? part of the whole thing. I, yeah, there was like a Drupal features and ways that you would commit things and whatever to get things in to get. And it was just like always very scary when you did any kind of deployment process.
29:05 Wes Bos? Another WordPress whoa. One time I was working on converting an old shop site. This guy submitted like 10, 10 WordPress horror stories like he'd never learned from his thing.
29:21 Wes Bos? Because the guy wanted his own site. I was doing everything in WP CLI to speed things up because there was tons of data. Well, I didn't realize that WP CLI was going to trigger the WooCommerce email system as I was migrating the data over. It sent out emails to people about their purchases. There weren't actual charges made, thanks goodness, just re-triggering of email notifications. And the guy I was doing the job woke up to over 18,000 emails from angry and confused customers. Luckily, I was able to salvage our relationship and get the job done with a five-star review. This guy's got five stars. And the site is still running to this day, five years later. It made me think a little more carefully.
30:05 Wes Bos? Well, apparently not, because the next one is from the same guy.
30:10 Wes Bos? But like, if you worked with somebody and they really screwed the pooch, but then they fixed it, would that give you confidence in working with them in the future?
30:21 Scott Tolinski? Well, you know what they say, you know, fool me once, shame on you. Fool me twice, you can't be fooled again. But
30:31 Scott Tolinski? Did you ever hear that? That's a George Bush quote. He like, he forgets what he's supposed to say. So he's like, "You can't get fooled again."
30:39 Scott Tolinski? It's a, it's a train wreck. But yeah, I give one time. One time you give them a break and then two times, yeah. J. Cole said something about that.
30:49 Wes Bos? Well, the third time, you got the next one.
30:52 Scott Tolinski? Yes. "I was working for a small WordPress-based company when we had a client who was thinking of hiring us on Retainer. So we said we would spend a lot of time to look at their site and their project to get up and up to speed and discuss what we could offer them. One of the big things they needed was help with when they approached us was dealing with Stripe. The way it was set up was not correctly doing something. I don't remember why exactly. So we asked if they had any custom code within the site dealing with Stripe and they told us no. When I pull up the site to my local machine and spun up in my dev environment at the time, and I start to look through some things, well, it turns out that they were wrong about custom Stripe stuff.
31:35 Scott Tolinski? And there was some code that automatically ran a bunch of webhook stuff when loaded locally. All of their customers, around 5,000, got a second charge to their credit card for the month, no matter how recently it was paid. All in all, they lost about 500 customers by the time we talked to them the next afternoon. We were not hired on Retainer and we learned some very valuable lessons that day. You didn't even get hired and they lost 500 customers for, that's a bad day at the office."
32:10 Wes Bos? You know what? They probably had the Stripe API keys in the database and he pulled the production database locally.
32:18 Scott Tolinski? Yeah.
32:19 Wes Bos? Holy, don't put API keys in your, I guess in some cases you have to, but man, that sucks. A 10% loss of all their customers. And on top of that, how many of those 5,000 people that got the second charge? Like filed a, like a charge back? Cause that's 15 bucks out of your pocket. Every, like regardless, if, if, if you, the way that Stripe works is if you can refund it before they do a charge back.
32:48 Scott Tolinski? Charge back.
32:49 Wes Bos? Then you're in good shape. But if you don't, if they file a charge back and then you immediately refund it, you still pay that $15, even if you, you win or regardless. So like, I don't know, 5,000 people, maybe of 1% of them charged back. That's still 750 bucks out of your pocket.
33:11 Scott Tolinski? Yeah. If you want to get pissed off, people, your customers, start messing with their money. Yeah.
33:21 Wes Bos? What does the P in VPS stand for? This is a really quick one. I switched my VPS provider and forgot to point my portfolio to the new IP address. How do you do that? A month later, a friend texted me saying, oh, my portfolio was turned into an adult website.
33:43 Wes Bos? How can you not check it?
33:46 Scott Tolinski? Dude.
33:47 Wes Bos? Oh, I guess that IP address was reused and you just never changed where it was pointing to. That's great.
33:53 Scott Tolinski? Yeah. One of the first agencies I worked for, we were doing WordPress sites. And the only site I'd ever worked on, it wasn't even my, my client, but um, it had gotten hacked somehow. I don't know how. And that's, that's what had happened to it. They just replaced every URL with a bunch of redirects to adult sites. And uh, it was a crazy day at the office. I'll tell you that.
34:15 Wes Bos? Never, never do that.
34:18 Scott Tolinski? Beyond the Grave 2. When I first joined my company as a junior developer, I was testing out an issue with one of our alert systems. By tagging a very popular artist to my test event, I managed to send everyone a push notification stating that artist was going to be performing in their area. Oh, no. The only issue being that that artist had died a few months prior. Oh, no.
34:47 Scott Tolinski? That's a much bigger oh no than I had initially given that. Oh, my God. Luckily, I worked for a nice company. No blame was placed on me and the new guardrails were put in place before tagging deceased artists and test events dispatching real alerts. Bro, that's two now that I have dealt with people who have passed away. That is very spooky. Ah, that I give that one 10 ghosts out of 10.
35:15 Wes Bos? People talk about like vibe coding and building an app in an afternoon. This is the stuff that you're not told about. How these companies have unfortunately sent push notifications about a dead artist.
35:31 Wes Bos? And now there's some code in their code base that checks if an artist is deceased or not before it sends events. The hottest hot fix. I once worked on new features that introduced API calls to a somewhat complex web component. From everything my team tested, it worked great. We thought we had accounted for everything. So we push it up to our repo, get it approved, publish the version and get it installed in our front end applications. Fast forward a smidge, the reports start to roll in that tests in the front end application are failing and holding up CI and CD. The common thread amongst the reports was the test suite can't find any elements. I don't pay much mind because we ran a colossal Cypress test suite
36:16 Wes Bos? for a monolithic repro. So that could be anything. And that kind of thing is happening for. It's so true. Your Cypress tests are failing. Almost nobody goes, oh, must be actual error. Your initial thoughts are just like, probably Cypress is acting up.
36:34 Wes Bos? Then I see the error crap. This is our API call failing in their local dev environment because we didn't add a proxy or dummy data. We forgot to add a failure state. And because we didn't catch the error, it's throwing at webpack's red screen of death, which blocks Cypress from querying anything on the page. We rushed out a failure state, which solves a problem in our main development branches. Nope. The hotfix branch is running into the same issue. Let's get that fix out. Oh crap. They weren't supposed to get the absolute latest, so they did a hotfix on, not on their production, but on their, I guess their main branch, which is not yet released.
37:17 Wes Bos? So they pushed out a bunch of code that was not meant to be pushed out yet. Suffice to say, it was a comedy of errors that took days to fix. And if we had Sentry in our build system, we likely wouldn't have run into this issue. Thank you Sentry.io forward slash syntax use coupon code tasty treat. Hope you enjoyed this spooky story.
37:35 Scott Tolinski? Ah.
37:36 Wes Bos? Oh, that sucks. That sucks. I'm gonna give that one a one out of, no, a two out of three.
37:45 Scott Tolinski? Yeah. Yeah, I'll give it, I'll give it a five out of 14 Legos on the floor. Bad redirect. In a company I worked where everything was basically in shambles and barely making ends meet, a coworker thought it was funny to somehow magically redirect hackers to an adult site when he thought something was of malicious intent. The boss made a demo for the company. That was playing during the demo, he was redirected to the adult site several times. It was a miracle we didn't lose the contract and that he didn't get fired. You sent your, you sent your boss to an adult site?
38:28 Scott Tolinski? Don't put adult sites in your work. Don't do that.
38:31 Wes Bos? Never, ever.
38:33 Scott Tolinski? Never.
38:34 Wes Bos? Ever put a, even put a swear in. I was having some frustrations with my recording the other day. I was sent like a screen recording to some, someone and I sent it to Randy, our producer as well. And the name of the screen recording was F this. And guess what? They saw. Never, never put swears or any adult stuff or, or even like think that you understand. Cause also a lot of this, like trying to catch people doing bad stuff. You're going to eventually catch somebody who's not doing bad stuff for whatever reason. Don't do that. Yeah, that's not a good idea.
39:12 Scott Tolinski? You can, you can just send them to a blank page. That's just as easy. And then like, no harm, no foul. You can trap the, you can do honeypots. You can do all that stuff. And you can still annoy them by, I had a ban system on Level Up that would like act as if they weren't logged into the site. So then they would go to try to log in again, but then the login would fail because they were already logged into the site.
39:34 Wes Bos? Oh, yeah.
39:35 Scott Tolinski? Like, and that way, like if, if for some reason someone ever got accidentally stuck into the, that ban system. It would just be like, hey, I'm trying to log in. The login says this. Oh, let me fix it for you. You must've tried five different credit cards, 40 different times and the system failed. You know, like you must've hit these specific criteria, but like, don't send them to an adult site. That's bad, bad idea.
40:03 Wes Bos? Next one, instead of making money, you spend money. So back when I started web development, I was working for one of those sketchy cellphone game ringtone wallpaper companies. Um, and we're billing the victim, I mean customer via premium SMS messages. That's how it used to be. You want a ringtone? You, you send an SMS. For development and testing purposes, we had regular SMS numbers that we use that didn't bill our cellphone. Okay. So you had like a, you have a premium one, which you make money. You have a regular SMS, which you actually, you pay to send. One morning our CEO called us into a meeting to give us the bad news that there wouldn't be a bonus that month. As the SMS provider had screwed us over, instead of paying out the bumper sale, they were billing us for excessive use of the free SMS.
40:50 Wes Bos? I'm sure you had already, guess what happened? So at a moment, I had to sheepishly confess. I expected to be written up for having left the dev mode enabled in production, but I had an amazing CEO that just said, make sure it doesn't happen again and left it at that. Like you cost not just the company, but your coworkers real money by leaving a dev flag in production. That is three bones out of three for me. That sucks.
41:21 Scott Tolinski? Three bones out of three? I'll do eight cobwebs out of 10 cobwebs. Certbot Certain Death. How to Lose Hundreds of Hours of Meeting Recordings in One Easy Step. Just let Certbot fail silently and wait for the SSL to expire. Hmm. We record our Zoom meetings through their RTMP streams. All of these streams land on our loyal Injust server, which is tied to a domain name and secured with a Let's Encrypt SSL certificate. Like responsible grown up adults, we even let Certbot to handle renewals automatically. Yeah, that's, that's fine. Then one day during the mysterious deployment, probably a Friday, Certbot stopped working.
42:06 Scott Tolinski? No alerts, no warnings, just quiet little failure sitting there, waiting. Days later, when the SSL certificate finally expired, boom. Every single meeting stream started failing. And how did we find out? Not through fancy monitoring or alerts. An employee casually said, hey, that little streaming icon isn't showing up anymore. Translation, congratulations, you've lost recordings. A round of chaotic debugging later, we discovered the issue. The server was alive, but the SSL was not. We renewed the cert, the streams returned, and we immediately begged the leadership to approve buying proper SSL certificate with a longer validity.
42:50 Scott Tolinski? Because apparently relying on Certbot alone is a bit like relying on goldfish to guard your house. Moral of the story, if Certbot fails quietly, it's not the failure that hurts. It's the moment SSL expires that breaks your soul. What a wonderfully written spooky story value. Thank you for crafting that. That's why I just use a system, a host or something that uses Certbot behind the scenes or whatever to manage the SSLs. I don't want to do Certbot myself, but I certainly don't want to pay for SSL certificate in 2025. I'm just going to use whatever my host has got baked in. Because every host in the world's got that now.
43:34 Scott Tolinski? Who's not giving you SSL renewals for free?
43:37 Wes Bos? I actually just looked it up. Let's Encryp used to email you when your certs expired, but as of January 2025, they've stopped that. So they won't even, you need like a downtime, like a ping, like a pinging service would.
43:52 Scott Tolinski? Ping service, for sure.
43:53 Wes Bos? Easy fix that for you. All right, next one. It's always DNS.
43:59 Scott Tolinski? Is this about AWS?
44:01 Wes Bos? Probably.
44:02 Scott Tolinski? Was that DNS?
44:03 Wes Bos? Actually, it's about Bluehost.
44:05 Scott Tolinski? Oh, classic.
44:07 Wes Bos? It's always about DNS. I had been debating about staying with Bluehost or moving over to Netlify for my small business website. I was tired of paying for hosting and figured a free Netlify plan was sufficient for what I needed. I mean, my site was pretty simple, just a portfolio, some case studies, and why was I paying 120 bucks a year to Bluehost when Netlify could host it for free? WordPress was overkill for one I needed. Now, I should mention, I'm a designer who dabbles in front end development. I can build it, style it, make it look gorgeous, but deployments and server side stuff, that's not really my wheelhouse and I'll usually work with other developers that would handle those things. Still, I watched a few YouTube tutorials in Netlify and everybody made it look so easy.
44:51 Wes Bos? How hard could it be, right? So one Saturday morning in the air of confidence and iced coffee, I decided to cancel my hosting and begin migrating to Netlify. I signed up for Netlify, connected my GitHub repo and watched my site deploy in minutes. It felt like magic. Then came the DNS settings. Now, I know DNS was important, as we all do, but when I logged into my domain registrar and I saw that wall of cryptic records, A records, MX records, C name, TXT, SPF, my brain just went, yeah, I got this, shouldn't be too hard. I updated my name servers to point to Netlify as their documentation walked through and watched the little propagation may take 24 to 48 hours pop up.
45:36 Wes Bos? And I felt like a tech genius. Is this really what backend devs deal with? Super easy. As a cell certificate check working, I gave myself a little victory fist pump. What I didn't realize is I had removed my email, which was running through Google Workspace. And those MX records I casually deleted, those were the only things telling the internet.
46:02 Wes Bos? For two weeks, it felt like months. I'm sitting there thinking, wow, business is slow today. Nobody's reached out. Gets us a quiet week. Meanwhile, my email service is completely non-functional. And every email sent to me is bouncing into the void. Clients trying to send me project files, potential leads and time sensitive requests. All gone, never received. I get that way sometimes where I turn on do not disturb for the entire day. And then I go, wow, nice, nice, quiet day today. Oh shit, I do not disturb on all day. Hey, I've been trying to email you all week. Are you getting my messages? They keep bouncing back. The panic like that set in. Oh no, what?
46:43 Wes Bos? But that's where it gets worse. I had no idea to fix it. Remember, I didn't know backend network DNS stuff. So I did what any panic designer does. I started Googling. I spent the next two days, brother, you could have, you could have DM'd me and I would have fixed it for you in like 10 seconds.
47:05 Wes Bos? Oh, going down the most frustrating documentation rabbit hole of my life. I've read Netlify's DNS documentation, which would tell me to check my email provider settings. So I go to Google Workspace documentation, which would tell me to configure my DNS records with my hosting provider. So I go back to the Netlify docs. I was stuck in an infinite loop. I even went chat GPT and gave it to me back and forth in the same flow. I watched YouTube tutorials, read hundreds of Reddit thread. All with different variations of how to fix DNS records Netlify Google Workspace. I finally pieced it together enough, enough information to understand that I need to manually add Google's MX records back into the Netlify DNS settings. I copied those magic strings of text from Google's support page. I pasted them into Netlify.
47:49 Wes Bos? I waited for what seemed like another agonizing 24 hours for propagation. And then I finally emailed myself and started working again. You don't even need to wait 24 hours there.
48:00 Scott Tolinski? You don't, yeah.
48:01 Wes Bos? There's like two things with DNS is when you're changing the name servers, you're literally changing who manages your DNS. If you can change it from provider to provider, or you can just use your registrar to change the DNS, you know, that's what I do. I put them all in Cloudflare and then I just point them at whichever servers you actually need because that way you don't have downtime. But also like when you import a domain name into Cloudflare, it scans the domain name for as many records as it can because you could accidentally be bringing down so much stuff, more than email, if you don't know about it. That's the fun part. I crafted the most embarrassing, sorry for the technical difficulties email I've ever written and sending it to every client I could think of.
48:46 Wes Bos? I had to do damage control, apologize profusely and basically admit that yes, the UX guy broke his own website user experience. Talk about updating my services to we offer development, but only front end development.
49:02 Wes Bos? Oh, I actually, I did that once where I switched over somebody's name servers and I thought I had taken their email down, but it turns out that I didn't. But I was panicking for like probably an hour.
49:17 Scott Tolinski? Yeah.
49:18 Wes Bos? It sucks.
49:19 Scott Tolinski? It does suck. Why has DNS propagation gotten faster? It used to have to wait 24 hours. Like what's changed there?
49:26 Wes Bos? I don't know. How does the internet work? You know, there's, there are name servers around the world and when one of them has changed, they have to let the other ones know. And I've had it in the past where it updates instantly for me, but then you, there's, there's like a website you can visit where it will ping your website from like, like hundreds of servers around the world and tell you where it's pointing to. And you can see if it's updated. So like, it's just such a common thing and the, the infrastructure between all of these peers, as I understand it, has gotten much better. So things. Things update much more quickly.
50:02 Scott Tolinski? Cache ruins everything around me. Cream, get the money. Dollar, dollar bill, y'all. Uh, first day on the job was for a small startup, serving hundreds of thousands of users daily. I picked up an easy ticket, updated some dependencies, ran fine locally, ran the test. CI was fine. PR was approved and merged. I headed out for lunch with the CEO and founders who were in the office that day. Well, a nice little day at the office. Mid-conversation, his phone starts buzzing. He checks it and goes pale. Sites down, white screen. My stomach drops. That two-line PR? No way.
50:43 Scott Tolinski? Turns out, our monorepo had the same package at two slightly different minor versions, locally and in CI. Yarn's cache handled it fine, but Vercel's build didn't use the cache. White screen for a thousand users. The CEO stepped away to make some calls. I pulled out my phone, jumped into the dashboard and rolled back the deployment myself. 12 minutes of downtime. My second PR? Implementing sync pack lint to the CI to catch version mismatches. Oh, yeah. Build cache, bro.
51:19 Wes Bos? People always get annoyed by like lock files in pull requests. Or sometimes they'll just be like, yeah, I'll do it myself. I'll just delete the thing and reinstall it if there's like a, like a thing. But what's happening is that like, there's a possibility that eight levels down your package tree, there's some package that got bumped from 1.0.1 to 1.0.2. And, and it introduces a major bug in your system. I've, I've had it myself once in the past and that's where lock files are, are very, very important.
51:51 Scott Tolinski? Yeah.
51:52 Wes Bos? Next one's called a fiber F-up. I ripped out and destroyed the ventilation system at a US mall, along with $100,000 in equipment from a single retry. A number of years ago, I was working as an embedded engineer for Internet of Things products. This specific product is designed to guide sunlight into building. It consists of a large array of approximately eight-inch lenses and a big stack of fiber wire, uh, led into the building as special fixtures, which is lamps that scatter the sunlight in the room. That's crazy. So it's sunlight coming from outside, being piped through fiber and then into lamps.
52:35 Wes Bos? So like they're lighting the mall without any electricity. It's that easy. Sun hits a lens, travels through the fiber wire and lights up the room. The entire lens, okay, he just explained how it works. Yes. The entire lens. So I didn't read this whole thing because I was like, this is gonna be good. The entire lens array is programmed to follow the sun with fractions of a degrees of accuracy. Everything in these projects is expensive because of the insane tolerances required for good light. But the fiber wire especially needs to be custom made. Normal fiber for internet, audio, et cetera, do not work for visible light. They are made to transmit signal frequency data, not broad spectrum light. Interesting.
53:18 Wes Bos? One of the first of these devices was installed in a mall in the US. The fiber wire was simply fed down into the mall through the air ducts. The code had validation checks for everything, including valid encoder entries. That's the position of the motor, right? For the motor that spun the entire array of lenses. If any value was wrong, the application would just stop. Last minute, we make, these are always the best. Half of these stories are last minute, we made a separate decision. The operating system should just restart any application that crashed. Well, unfortunately, one of the encoders turned out to be faulty.
53:58 Wes Bos? The application would start up, validate everything, start to spin the device, realize the motor wasn't turning and crash. Then it would start up, spin for a bit and crash. For an additional context, due to the insane precision needed, the motor was mounted on a one to 65,000 gear. It took 30 minutes to rotate 180 degrees in full speed. It had crazy torque. It took a few days before the thing.
54:28 Wes Bos? It took a few days before the thick fiber wires finally ripped out large chunks of the ventilation system in multiple rooms and then finally snapping in half.
54:42 Wes Bos? Somehow I made those two decisions all by myself and I did not make the connection.
54:49 Wes Bos? This guy caused physical damage in the magnitude of hundreds of thousands of dollars.
54:56 Scott Tolinski? Physical damage.
54:57 Wes Bos? From one little decision to reboot.
55:04 Wes Bos? That's like fail safe. Fail safes, man, are crazy when, especially when they're real world where they could, like I often think about like self-driving fail safes and all of the code that needs to be written for that.
55:17 Scott Tolinski? I think unfortunately for humanity, I don't know how many of these companies are going to put fail safes over, you know, the dollar. So we'll see how that works out once that all, because there was like some, something where like a electric car got pulled over and it's like the car can't be held at fault. So like, what are they going to do? Like, what are they going to do when something crazy happens with electric cars? I better not hear a dev horror story about a
55:44 Wes Bos? No, I know.
55:45 Scott Tolinski? Or a self-driving car.
55:46 Wes Bos? Last minute, we decided it's okay to hit dogs as long as you're not killing a human.
55:52 Scott Tolinski? As long as the dog is 15 years old, we don't mind. The dog was on its way out anyways.
55:58 Wes Bos? We didn't account for the fact that the API could be down and the gas would never let up.
56:04 Wes Bos? I did one indentation and the thing drove itself into the ocean. Please don't make that happen.
56:10 Scott Tolinski? Our AI access to the self-driving code and had it rewrite it.
56:18 Wes Bos? Awesome. Well, we had more stories than we actually needed, but please keep them coming. Syntax.fm/Spooky. We'll throw the ones that we didn't read today onto the, onto the schedule for next year, because there are some brutal ones that have, have yet to come.
56:36 Scott Tolinski? Yes. Brutal. Yes. What a wonderful and spooky, spooky stories episode. Wes, thank you so much for making this happen live. I thought that was a really neat little tit you did. People who are watching on podcasts, we ended up streaming this live kind of randomly on Twitter. A lot of fun, folks. I hope you enjoyed yourself. If you're watching live, if you're listening to the episode in your car, if you're watching on YouTube, we do this every single year. So if you liked these dev horror stories, there are some millions and millions of dollar cost horror stories in the previous spooky stories episodes. So, folks, if you enjoyed these and you're looking for more bone chilling tales, check out the
57:20 Scott Tolinski? spooky stories episodes on Syntax. There are a ton of them. I don't know. We've done one, two, three, four, five, six, seven.
57:29 Wes Bos? Probably, we've done like seven years of them, and we've done at least two or three every single year. So there's, there's at least twenty episodes.
57:36 Scott Tolinski? Like sixteen or seventeen. Yeah. Yeah, including these two, probably twenty. So check them out. Tons and tons of spooky stories. If you have a spooky story, submit it on our website, Syntax.fm, and we might read it next year. So, shout out.
57:50 Wes Bos? Peace.
57:51 Scott Tolinski? Peace.
Previous 949: Web Dev HORROR Stories + Spooky Trivia! (Spooky Stories Pt. 1) Next 951: A first look at Remix 3 Page updated November 1, 2026 · transcript and summary are machine-generated