How To Programmatically Change An ImageButton's Click Handler

I'm trying to use one ImageButton as both a login button and a logout button. Initially its a login button. I click the button once and it runs the login click handler and pops up a text box to type in a password. Then I enter the password to login. Inside the password box's submit handler I check to see if the password is valid. If it is, I enable all the password protected functionality on the page, change the button's image from the login image to the logout image, and change the button's click handler from the login Click handler to the logout Click handler.

So far my code just swaps the button's image but the click handler never changes to the logout handler. I have tried two ways of changing the ImageButton's click handler programmatically and both have failed:

1.

loginBtn.Attributes.Remove("OnClick");
loginBtn.Attributes.Add("OnClick", "logoutBtn_Click");

2.

loginBtn -= new ImageClickEventHandler(loginBtn_Click);
loginBtn += new ImageClickEventHandler(logoutBtn_Click);

Neither the login text box or the login/logout button are inside an UpdatePanel. EnableViewState is set to the default of true. Does anyone see what is preventing the Click handler from being changed?