C# - Loading DLL Into Separate AppDomain?

I am writing a plugin architecture. My plugin dlls are located in a sub directory from where the plugin manager is running. I am loading the plugins into a separate AppDomain as the following:

string subDir;//initialized to the path of the module's directory.
AppDomainSetup setup = new AppDomainSetup();
setup.PrivateBinPath = subDir;
setup.ApplicationBase = subDir;
AppDomain newDomain= AppDomain.CreateDomain(subDir, null, setup);
byte[] file = File.ReadAllBytes(dllPath);//dll path is a dll inside subDir
newDomain.Load(file);

However. newDomain.Load returns an assembly which the currently domain attempts to load. Because the plugin dlls are in a sub directory, the current domain cannot and should not see these dlls and the current domain throws a FileLoadException"ex = {"Could not load file or assembly ... or one of its dependencies."

The question is, can we load an assembly into a separate AppDomain without it returning the loaded assembly?

I know I can add a handler for the AssemblyResolve event in the current domain and return a null, but I would prefer to not to go this route.