System.Web.HttpException (0x80004005) in Optimizely MediaHandlerBase: Server cannot append header after HTTP headers have been sent.

I ran into this exception while looking at a custom media handler in an Optimizely CMS 11 website. The stack trace pointed at Optimizely trying to append an ETag header, but the interesting part turned out to be a redirect in our own code.

System.Web.HttpException (0x80004005):
Server cannot append header after HTTP headers have been sent.

at System.Web.HttpResponse.AppendHeader(String name, String value)
at EPiServer.Web.MediaHandlerBase.CheckIsModifiedAndAddETag(...)
at EPiServer.Web.MediaHandlerBase.NotModifiedHandling(...)
at EPiServer.Web.BlobHttpHandler.ProccessBlobRequest(...)
at EPiServer.Web.BlobHttpHandler.ProcessRequestAsyncInternal(...)

This is the old System.Web handler pipeline on .NET Framework. The code below belongs to CMS 11; it is not an ASP.NET Core middleware example for CMS 12 or 13.

The redirect was happening inside SetCachePolicy

The handler in our code inherited from BlobHttpHandler and added long-lived caching for versioned media URLs. A cachebuster in the URL identified the content version. If it no longer matched the current content, the handler redirected the request to the URL with the new cachebuster.

That check lived in SetCachePolicy. This looked like a reasonable place to put cache-related code, except that Optimizely still had work to do after calling the override. The relevant order was:

SetCachePolicy(context, modifiedDate);  // Our override could redirect here.
NotModifiedHandling(context, modifiedDate);  // ETag handling follows.
// Then continue with the media response, when appropriate.

This is a shortened illustration of the call order, not the complete Optimizely implementation. The important detail is that returning from SetCachePolicy hands execution back to code which may still append headers.

Why endResponse: true did not fix it

I initially wanted to make the smallest possible change to the existing code. Rather than move responsibilities around in the handler, I tried making the redirect explicitly end the response:

context.Response.RedirectPermanent(
    "/" + expectedString + context.Request.Url.PathAndQuery,
    endResponse: true);

I expected that to prevent Optimizely from reaching its ETag handling. There are two problems with that assumption. First, the single-argument RedirectPermanent(url) overload already ends the response. Adding true made my intention clearer, but did not change that behaviour. See the RedirectPermanent overloads.

Second, ending the response is not a reliable way to return from the current call stack. Response.End() attempts to abort execution with a ThreadAbortException. When it cannot do that, it can instead flush the response and arrange for the request to complete, then return to the caller. Microsoft describes both paths in the HttpResponse.End documentation.

This matters specifically for the asynchronous handler path. BlobHttpHandler implements IHttpAsyncHandler, using the older Begin/End asynchronous pattern. In the ASP.NET reference source, the step which starts an asynchronous handler is marked as non-cancellable. That provides a concrete reason why relying on the thread-abort path here is a bad assumption.

There does not need to be an await in our code, or two threads racing to write headers. The redirect can happen during the initial call into the asynchronous handler. If End() flushes and returns, the same call chain can simply carry on:

BlobHttpHandler calls our SetCachePolicy
  -> RedirectPermanent(..., true)
  -> Response.End() flushes the redirect and returns
  -> SetCachePolicy returns to BlobHttpHandler
  -> NotModifiedHandling tries to append the ETag
  -> The response headers have already been sent

A return inside SetCachePolicy only exits that override. Replacing End() with CompleteRequest() does not unwind the call stack either. It changes how ASP.NET proceeds through the request pipeline, but the code already running still needs to return.

Move the redirect before the normal blob handling

The fix was to check for the outdated cachebuster at the handler entry points. If the request needs a redirect, set the redirect response without calling End(), and return without entering the base handler. Both the synchronous and asynchronous paths need to make that decision.

protected override IAsyncResult ProcessRequestAsyncInternal(
    HttpContextWrapper context, AsyncCallback cb, object extraData)
{
    if (!TryRedirectOutdatedCacheBuster(context))
    {
        return base.ProcessRequestAsyncInternal(context, cb, extraData);
    }

    // This handler has completed the request without starting a stream.
    var result = new AsyncResult(cb, true);
    result.SetCompleted();
    return result;
}

protected override bool ProcessRequestInternal(HttpContextBase context)
{
    return TryRedirectOutdatedCacheBuster(context)
        || base.ProcessRequestInternal(context);
}

AsyncResult here is EPiServer.Web.Internal.AsyncResult. In the implementation used by this fix, the true constructor argument becomes AsyncState, which BlobHttpHandler.EndProcessRequest interprets as a handled request without a stream. It is not the endResponse flag, nor an instruction to start another thread. SetCompleted() marks the result complete and invokes the supplied callback. Because this uses an internal Optimizely type and its completion contract, check it against the CMS assembly version used by your application.

The synchronous version relies on the short-circuit behaviour of ||: if the redirect helper returns true, the base implementation is never called. The asynchronous version does the same thing explicitly, while also giving ASP.NET the completed result it expects.

Keep the access checks when moving the code

Moving the redirect earlier also moves it ahead of checks that the base handler previously performed before reaching SetCachePolicy. Those checks must not disappear in the process. The helper retains the routability, read-access and asset-owner checks before returning a redirect:

private bool TryRedirectOutdatedCacheBuster(HttpContextBase context)
{
    if (!IsRequestComingFromCdn(context) || context.Response.HeadersWritten)
    {
        return false;
    }
 
    var content = ServiceLocator.Current.GetInstance<IContentRouteHelper>().Content;
    if (content == null)
    {
        return false;
    }
 
    var cdnString = (string)context.Items[CacheBusterInitialization.CdnRequest];
    var expectedString = CacheBusterInitialization.Unique(content);
    if (String.Equals(cdnString, expectedString, StringComparison.OrdinalIgnoreCase))
    {
        return false;
    }
 
    // Preserve the checks BlobHttpHandler performs before reaching SetCachePolicy.
    if (!ServiceLocator.Current.GetInstance<IRoutableEvaluator>().IsRoutable(content) ||
        !content.QueryDistinctAccess(AccessLevel.Read) || !HasAssetOwnerAccess(content))
    {
        return false;
    }
 
    // Response.End can flush and return in an async handler. Return from the handler instead.
    context.Response.RedirectPermanent("/" + expectedString + context.Request.Url.PathAndQuery, endResponse: false);
    return true;
}

CacheBusterInitialization is application-specific code, not a built-in Optimizely API. In this application it provides the request marker and the expected cachebuster. The URL expression also depends on the existing rewrite setup: at this point the path is the media path to which the new cachebuster should be prepended. If your request still contains the old prefix, you need to replace it rather than keep adding prefixes.

The existing IsRequestComingFromCdn helper checks for an anonymous request carrying that marker. Despite its name, that alone is not proof that a request came from a trusted CDN. The access checks are still needed. If they fail, the helper returns false and leaves the request to the base handler and its normal access handling.

The HeadersWritten check only prevents this helper from attempting a redirect too late. It cannot repair a response that another component has already flushed, and returning false does not guarantee that later base-handler header operations will succeed. The actual fix for this path is making the redirect decision early and skipping the blob handling afterwards.

With that change, SetCachePolicy only sets cache policy. The existing CDN cache settings and the normal GetBlob implementation can remain in the handler; neither needs to end the response.

The part I would keep in mind when doing something similar is where the return happens. A redirect inside a nested cache-policy override leaves Optimizely with more response work to do. A redirect at the handler entry point lets us finish the request before that work starts.

It also leaves the responsibilities in a better place. SetCachePolicy should decide how the media response is cached; deciding to redirect the request belongs earlier in the handler. I had wanted to avoid moving existing code for what looked like a small fix, but in this case moving the redirect was the correct call.