AppendHeader Not Working In MVC?

I'm having problems getting AppendHeader to work properly if I am also using an authorize filter. I'm using an actionfilter for my AJAX actions that applies Expires, Last-Modified, Cache-Control and Pragma (though while testing I have tried including it in the action method itself with no change in results).

If I don't have an authorize filter the headers work fine. Once I add the filter the headers I tried to add get stripped.

The headers I want to add

Response.AppendHeader("Expires", "Sun, 19 Nov 1978 05:00:00 GMT");
Response.AppendHeader("Last-Modified", String.Format("{0:r}", DateTime.Now));
Response.AppendHeader("Cache-Control", "no-store, no-cache, must-revalidate");
Response.AppendHeader("Cache-Control", "post-check=0, pre-check=0");
Response.AppendHeader("Pragma", "no-cache");

[Code]....