ServiceActivationException: Request cannot be dispatched because the virtual application is shutting down

We are having a problem with our virtual application shutting down. Site is running ASP.NET 4.5, IIS 7.5

Exception summary:

System.InvalidOperationException: Request to the service at '~/Services/ServiceExceptionLogger.svc' cannot be dispatched because the virtual application at '/Site' is shutting down.

Stack trace:

System.ServiceModel.ServiceActivationException: Request to the service at 
'~/Services/ServiceExceptionLogger.svc' cannot be dispatched because the virtual application
 at '/Site' is shutting down. ---> System.InvalidOperationException: Request to the service at '~/Services/ServiceExceptionLogger.svc' cannot be dispatched because the virtual 
application at '/Site' is shutting down.
 --- End of inner exception stack trace ---
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.End(IAsyncResult result)
at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)

Or here is another version of it that shows up in the event logs

Stack Trace:

System.ServiceModel.ServiceActivationException: Request to the service at '~/Services/Service.svc' cannot be dispatched because the virtual application at '/Site' is shutting down. ---> System.InvalidOperationException: Request to the service at '~/Services/Service.svc' cannot be dispatched because the virtual application at '/Site' is shutting down.
   --- End of inner exception stack trace ---
   at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.End(IAsyncResult result)
   at System.Web.HttpApplication.CallHandlerExecutionStep.OnAsyncHandlerCompletion(IAsyncResult ar)

Anyone have any ideas or seen this before? Seems to happen quite frequently on one of our environments, like 10+ times a day. On the QA environment its not happening at all. Idle timeout is set to 20minutes on both environments. Reason for 20 minute timeout is to avoid wasted memory

Apparently the app pool is not recycling when the above exception occurs. We are watching the process in task manager and it is not shutting down or reducing the memory when this exception occurs. Also, we turned on all logging, and no app pool recycle event log entry occurs in the System Log when this exception happens