“something’s off”

— Toji Fushiguro, Jujutsu Kaisen

On how hope is bad

At a previous job, I worked on an entertainment app that looked a lot like Netflix, except it did not work like Netflix at all.

There was no subscription. People paid as they watched, episode by episode. I was working on a limited-time “free show of the day” feature. The idea was simple enough: pick a show, make it free for a day, and let people discover it without having to think too hard before pressing play.

At first, it looked like people were interested. Mixpanel showed plenty of clicks. People found the show, opened it, but then somehow disappeared somewhere along the way.

That was not ideal, but it was still a story we could tell ourselves. Maybe they got distracted. Maybe the show was not good enough

Then our monitoring tools showed us a stranger thing. Few people had paid for an early episode of the show we had made free.

And, for a brief and embarrassing moment, we were happy about it.

Maybe the user had finished their free viewing and liked the show enough to keep going when it was paid again, we thought. Maybe this was evidence that the feature worked. Maybe people really were willing to pay after getting a taste.

Hope is useful right up until it becomes an explanation.

Because if someone liked a show enough to pay for episode 9, why had they not binged it earlier while the whole show was free? The story did not make sense. Still, we kept finding ways to make it make sense. I remember blaming the “cache” .

The forgotten experiment

Eventually, a few more cases showed up. For users in a particular country, the free show was visible. The banner said free. The show page looked free. But when they tried to watch, they were asked to pay.

The pattern became difficult to ignore and then we found out that the free-show feature itself was not broken. It was being overruled.

Long before this feature, we had run another experiment: a discount on individual episodes. Not shows, episodes. The experiment had ended, it was no longer discussed. It was no longer on anyone’s roadmap. It had become one of those things that lives only in a few branches of code.

But the flag was still resolvable for that region (which was different than the region we tested in).

Our pricing logic gave the old episode-level configuration precedence over the newer show-level configuration. So the system did exactly what we had taught it to do. It saw a free show, then saw an old episode rule, and decided that the episode rule mattered more.

A branch is not dead untill its present in your code

I think feature flags make it very easy to confuse turning something off with removing it.

Turning a flag off means the happy path is quiet. The old/experimental code is still there. The old rule can still be evaluated. The old configuration can still exist for a strange cohort, a country-specific rollout, or an account that was somehow missed during cleanup.

Removing a flag means the system has forgotten it ever existed.

After this, I started thinking of old experiments as dead things. You do not leave them in the house because they are no longer moving. You bury them.

That means deleting the flag, the fallback, the stale configuration, and the branch of business logic that only makes sense if the experiment still has a future. It also means making the code unable to resolve that flag for anyone.

Give the cleanup a deadline

One small practice that helps is an expiry test: a test whose only job is to fail if the feature flag branch is still present after the planned removal/migration date. It is not elegant, exactly. It is a deliberately annoying note to your future self.

describe('daily free show experiment', () => {
  it('must be removed after 2026-10-31', () => {
    const expiry = new Date('2026-11-01T00:00:00Z').getTime();

    expect(Date.now()).toBeLessThan(expiry);
  });
});

As long as this test exists, the build fails on November 1. The fix is not to extend the date without thinking. The fix is to remove the experiment: its flag, the new/old code path, and this test.

It is a blunt tool, and it needs an owner. But that is partly the point. An experiment should make someone responsible for ending it properly, not just for launching it.

I still remember how reassuring it was to say “cache” before I had any proof.

So when an code path is done, do not just turn it off. Make it impossible for your application to remember it.