The Startup Bug That Could Have Gone Unnoticed for Years

There's a lot of talk about AI replacing software developers. I think some developers will be replaced; the job has already changed, and pretty dramatically. I don't think that's only because AI can write code. It changes what the job asks from us.

When I started in software about ten years ago, a big part of the job was knowing how to write code and knowing the languages, frameworks, and libraries you used. If you didn't know how something worked, you looked it up, copied an example from Stack Overflow, and figured out how to make it fit. AI can do a lot of that much faster now. It has seen enough common patterns to get you to a plausible first version quickly.

That doesn't mean the work has disappeared. Reading code and reasoning about what it does are becoming more important than expressing every idea by writing the code yourself. Being able to write code still matters because it helps you judge what AI produced. Writing good code matters too: AI tends to follow the patterns it finds, so we need to establish useful patterns first.

I recently ran into a good example at work. I'll keep the project anonymous, but the application lets each organization decide how long a user can stay logged in. A recurring background job cleans up expired sessions, so if that job stops running, users may stay signed in beyond the limit their organization chose.

There are two requests for this work: one during application startup and one on a recurring schedule. The startup request handles a narrow case where the application was shut down while the saga was running. Its persisted state could otherwise make the normal recurring request retry the existing run instead of starting fresh. We could have added more recovery logic to the recurring workflow, but that seemed like a lot of code for something expected to happen very rarely. The separate startup request keeps that case out of the recurring path.

That does mean the startup request has to be published before the regular recurring request. If the recurring request gets there first, the saga may already be marked as running, and the startup request won't replace that run.

The solution AI suggested was a shared asynchronous gate. The recurring job waits for startup to publish its request, and startup opens the gate afterward. It was reasonable, compact, and compiled. AI also wrote a focused set of tests for the gate and startup behavior. They all passed, but they didn't exercise a failed startup publish.

public sealed class StartupGate
{
    private readonly TaskCompletionSource _startupPublished =
        new(TaskCreationOptions.RunContinuationsAsynchronously);

    public Task WaitForStartupAsync(CancellationToken cancellationToken) =>
        _startupPublished.Task.WaitAsync(cancellationToken);

    public void MarkStartupPublished() => _startupPublished.TrySetResult();
}

public async Task ExecuteAsync(CancellationToken cancellationToken)
{
    await _startupGate.WaitForStartupAsync(cancellationToken);

    await _publisher.PublishAsync(runRequested, context, cancellationToken);
}

But the gate only opened after a successful publish. If publishing returned an error or threw, startup logged the failure and rethrew before signaling completion. The recurring job would be sitting at the gate, waiting for a signal that would never come. If the host remained up, it would stay blocked there until cancellation, and the cleanup wouldn't run.

try
{
    var result = await publisher.PublishAsync(startupRequest, context, cancellationToken);
    if (result.IsError)
    {
        throw new InvalidOperationException(result.FirstError.Description);
    }

    startupGate.MarkStartupPublished();
}
catch (OperationCanceledException)
{
    throw;
}
catch (Exception exception)
{
    logger.LogError(exception, "Failed to publish startup request");
    throw;
}

That's when I realized this wasn't just a scheduling bug. The cleanup enforces each organization's session limit. If it stayed blocked, someone with an already-issued session could remain authenticated beyond the configured duration, potentially indefinitely while the application stayed up, or until a restart or another cleanup path ran. That made this a serious security issue.

The fix was small: release the gate in a finally block. That is the right behavior for the gate: its job is to order the two requests, not leave the recurring job disabled if startup publication fails. If the host keeps running, the ticker can proceed with its normal path. The exception still propagates, so this doesn't pretend the startup request succeeded or guarantee that a fresh run was created.

try
{
    var result = await publisher.PublishAsync(startupRequest, context, cancellationToken);
    if (result.IsError)
    {
        throw new InvalidOperationException(result.FirstError.Description);
    }
}
finally
{
    startupGate.MarkStartupPublished();
}

When I pointed out the failed-publish case, AI immediately agreed and moved the signal into finally. The code change was small; noticing the problem meant tracing the gate through startup failure and understanding what the cleanup enforced.

AI was genuinely useful here. It got to a reasonable solution quickly and helped fix it once I pointed out what it had missed. It just didn't find that case on its own. Useful, but not magic.