Getting System.IO.FileLoadException - Couldn't Load File Or Assembly

I have a rather simple internal ASP.Net Website that has issues loading the Microsoft.Practices.EnterpriseLibrary.Data dll about once a week. Here is the Exception message:

System.IO.FileLoadException: Could not load file or assembly 'Microsoft.Practices.EnterpriseLibrary.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. Access is denied.
File name: 'Microsoft.Practices.EnterpriseLibrary.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null'
at foobar.Intranet.Logic.Data.UserDB.SelectByUserName(String userName)
at foobar.Intranet.Logic.Info.User.ValidateUser(String userName) in F:Developmentfoobarfoobarfoobar.Intranet.LogicInfoUser.cs:line 130
at Login.ValidateUser(String username, String password) in e:foobarfoobar.IntranetLogin.aspx.cs:line 32

Pre-bind state information
LOG: User = Unknown
LOG: DisplayName = Microsoft.Practices.EnterpriseLibrary.Data, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null
(Fully-specified)
LOG: Appbase = file:///E:/foobar/foobar.Intranet/
LOG: Initial PrivatePath = E:foobarfoobar.Intranetin Calling assembly : foobar.Intranet.Logic, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null.

LOG: This bind starts in default load context.
LOG: Using application configuration file: E:foobarfoobar.Intranetweb.config
LOG: Using host configuration file: \?C:WINDOWSMicrosoft.NETFrameworkv2.0.50727aspnet.config
LOG: Using machine configuration file from C:WINDOWSMicrosoft.NETFrameworkv2.0.50727configmachine.config.
LOG: Policy not being applied to reference at this time (private, custom, partial, or location-based assembly bind).
LOG: Attempting download of new URL file:///C:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/Temporary ASP.NET Files/foobar.intranet/668896d8/82d7e51c/Microsoft.Practices.EnterpriseLibrary.Data.DLL.
LOG: Attempting download of new URL file:///C:/WINDOWS/Microsoft.NET/Framework/v2.0.50727/Temporary ASP.NET Files/foobar.intranet/668896d8/82d7e51c/Microsoft.Practices.EnterpriseLibrary.Data/Microsoft.Practices.EnterpriseLibrary.Data.DLL.
LOG: Attempting download of new URL file:///E:/foobar/foobar.Intranet/bin/Microsoft.Practices.EnterpriseLibrary.Data.DLL.
ERR: Failed to complete setup of assembly (hr = 0x80070005). Probing terminated.

Things of note:

A simple IIS Reset fixes the issue. We had the same code running on a single webserver with no problems. We then moved the site to two new, load balanced (using sticky sessions) webservers (all three Windows Server 2003 R2 Standard Edition SP1). Now each of the load balanced webservers throw this exception about once a week just out of the blue.The only major difference I can think of, is I've installed the 4.0 .NET framework on the load balanced servers. The website is still running under ASP.NET 2.0 so I don't see how this would be an issue I've configured the indexing service to not search the "Temporary ASP.NET Files" folder, but it didn't help. We use the Microsoft.Practices.EnterpriseLibrary.Data dll everywhere on our site. Almost every single page uses our logic dll which in turn utilizes the EnterpriseLibrary dlls. While the errors where happening once, I even looked to see what processes were locking the dll in the "Temporary ASP.NET Files" folder, and it wasn't any different than the locks on the server that was working fine. Once the error starts, it errors every single time until an iisreset is performed.