#NetApp #MsExchange
Okay so nailed this one eventually.
[SERVER1] Initializing SnapManager server on remote machine [SERVER2]…
[SERVER1] Connecting to remote server [SERVER2]…
[SERVER1] Remote server [SERVER2] is connected successfully.
[SERVER1] Verify Sme Launch Exception, Error code: 0x80131501, Error description: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state., stack trace:
Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.ThrowIfFaulted()
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMes
[21:02:47.831] [SERVER1] Verify SME Launch on SERVER2 returned error 0x80131501
[SERVER1] Verify Sme Launch2 Exception, Error code: 0x80131501 [The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.], stack trace:
Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IM
[SERVER1] CCR/DAG Remote Rename Backup Exception, Error code: 0x80131501 [The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.], stack trace:
Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturn
[SERVER1] Error Code: 0x80131501
Please check the SnapManager-SnapDrive Reports and Windows Event Logs for more details. |
It was caused by the Cisco ASA firewall on our Inter DC link timing out idle connections after one hour.
When the SME service starts, it establishes a connection over TCP port 810 to other servers in it’s DAG. SME expects the link to be available all the time and not time out.
SME is a but stupid, in that as it EXPECTS the link to be open, and s.
Workaround: Restart the SnapManagerService before the Backups Starts.
Solution#1: Increase the timeout value on the ASA firewalls for port 808 and 810 from 1 hour to8 hours (we do backups every 6 hours).
Solution#2: Product change to SME, so before it initiates a backup, it actually re-establishes the connection before it starts. BUG/RFE 828920 has been raised to address this.