profile

codevev

Why I Don’t Just Trust Unit Tests Anymore

Published 4 months ago • 1 min read

Nothing too exciting happened in the last couple of weeks — I’m in the middle of a refactor that I need wrapped up before my surgery next week.

The work is around the eligibility logic for our ITL (Integration Testing to the Left) product. Originally, the logic lived where ITL runs were created and was triggered for each code review. That worked fine, but the algorithm always picked the pipeline itself, which wasn’t always the best choice. We added customization on the ITL website, which meant the eligibility logic had to move to a shared place between the test plan creator and the site.

I won’t dive into the refactor details — but here’s the interesting part: instead of relying solely on unit tests (too mocked to be trustworthy), I ran both versions of the eligibility logic side by side. Whenever they drifted, metrics were emitted. I built a simple CloudWatch dashboard with graphs and log queries, and this already caught 3 major issues (plus a handful of null pointers) that would’ve slipped by otherwise.

It’s more work than it sounds. You have to mimic the original implementation without touching it, add nested if-statements to check for drifts (e.g., did we used to find an eligible pipeline but now only ineligible ones are found?), and carefully wrap the new logic in try/catch so it doesn’t affect the main flow.

But in the end, it was worth it. Better I do the extra work now than have customers — or the poor on-call — discover it later.

Cheers!

Evgeny Urubkov (@codevev)

​

600 1st Ave, Ste 330 PMB 92768, Seattle, WA 98104-2246
​Unsubscribe · Preferences​

codevev

codevev is a weekly newsletter designed to help you become a better software developer. Every Wednesday, get a concise email packed with value:• Skill Boosts: Elevate your coding with both hard and soft skill insights.• Tool Tips: Learn about new tools and how to use them effectively.• Real-World Wisdom: Gain from my experiences in the tech field.

Read more from codevev

Last week, I ran into this tweet: the tweet It kinda triggered me. Why would someone pay $0.40 per secret per month when you could just use AWS Parameter Store and store them as SecureStrings FOR FREE? That’s what I use for oneiras.com, so I was determined to find out if I’d missed something. Am I unknowingly paying per secret? Or is there actually a reason to use AWS Secrets Manager instead? Turns out, there are a couple, but only if you really need them. The Big One: Automated Secrets...

about 2 months ago • 2 min read

Well, the global AWS outage happened just four days after I sent a newsletter about COEs and how “nobody gets blamed.” Great timing, right? I wish I could’ve been in the weekly global ops meeting to see the temperature in the room. That’s the one where teams present their recent issues and learnings. I can only imagine how lively that one must’ve been. Turns out the culprit was a DNS failure in the Amazon DynamoDB endpoint in the us-east-1 region. And while that sounds region-specific, it...

2 months ago • 1 min read

Someone pushes a new feature to prod the same day you go on-call. Hours later, your phone goes off - not a gentle buzz, but a full-blown siren that could wake up the entire neighborhood. You open the alert, and it’s for a feature you didn’t even touch. Maybe it’s unhandled NPEs, maybe something else. Doesn’t matter. You’re the one on-call, so it’s your problem now. When Things Break In those moments, it’s usually faster to just debug and fix it - even without full context. I’m pretty good at...

2 months ago • 2 min read