registration free COM/Interop with SilverLight

I am trying to activate a COM-visible .NET assembly using the activation context API from SilverLight using the following configuration:

IE ->

SilverLIght component ->
AssemblyLoader.dll -
                unmanaged (not-COM)
                [manifest with dependency on AssemblyDLL's manifest] ->                                                                                                                                                                                              AssemblyNET.dll (COM visible)[manifest]

The manifests are not embedded. The activation context succeeds by loading the AssemblyLoader.dll.manifest but the result from CoCreateInstance of any object from AssemblyNET.dll is always 

I can successfully call the CoCreateInstance in the following configuration:

Test.exe using the AssemblyNET.dll.manifest to create the activation context -> AssemblyNET.dll (COM visible)[manifest]

Is this kind of activation context setup possible?