#NetApp #MsExchange
Okay so nailed this one eventually.
| [SERVER1] Initializing SnapManager server on remote machine [SERVER2]… [SERVER1] Connecting to remote server [SERVER2]… [SERVER1] Remote server [SERVER2] is connected successfully. [SERVER1] Verify Sme Launch Exception, Error code: 0x80131501, Error description: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state., stack trace: Server stack trace: at System.ServiceModel.Channels.CommunicationObject.ThrowIfFaulted() at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMes [21:02:47.831] [SERVER1] Verify SME Launch on SERVER2 returned error 0x80131501 [SERVER1] Verify Sme Launch2 Exception, Error code: 0x80131501 [The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.], stack trace: Server stack trace: at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen() at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: Exception rethrown at [0]: [SERVER1] Error Code: 0x80131501 |
It was caused by the Cisco ASA firewall on our Inter DC link timing out idle connections after one hour.
When the SME service starts, it establishes a connection over TCP port 810 to other servers in it’s DAG. SME expects the link to be available all the time and not time out.
SME is a but stupid, in that as it EXPECTS the link to be open, and s.
Workaround: Restart the SnapManagerService before the Backups Starts.
Solution#1: Increase the timeout value on the ASA firewalls for port 808 and 810 from 1 hour to8 hours (we do backups every 6 hours).
Solution#2: Product change to SME, so before it initiates a backup, it actually re-establishes the connection before it starts. BUG/RFE 828920 has been raised to address this.
flaphead.com 