NetApp SnapManager for Exchange and The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.

#NetApp #MsExchange

Okay so nailed this one eventually.

[SERVER1] Initializing SnapManager server on remote machine [SERVER2]…
[SERVER1] Connecting to remote server [SERVER2]…
[SERVER1] Remote server [SERVER2] is connected successfully.
[SERVER1] Verify Sme Launch Exception, Error code: 0x80131501, Error description: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state., stack trace:
Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.ThrowIfFaulted()
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMes
[21:02:47.831]  [SERVER1] Verify SME Launch on SERVER2 returned error 0x80131501
[SERVER1] Verify Sme Launch2 Exception, Error code: 0x80131501 [The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.], stack trace:
Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IM
[SERVER1] CCR/DAG Remote Rename Backup Exception, Error code: 0x80131501 [The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.], stack trace:
Server stack trace:
at System.ServiceModel.Channels.CommunicationObject.ThrowIfDisposedOrNotOpen()
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturn

[SERVER1] Error Code: 0x80131501
Please check the SnapManager-SnapDrive Reports and Windows Event Logs for more details.

It was caused by the Cisco ASA firewall on our Inter DC link timing out idle connections after one hour.

When the SME service starts, it establishes a connection over TCP port 810 to other servers in it’s DAG. SME expects the link to be available all the time and not time out.

SME is a but stupid, in that as it EXPECTS the link to be open, and s.

Workaround: Restart the SnapManagerService before the Backups Starts.
Solution#1: Increase the timeout value on the ASA firewalls for port 808 and 810 from 1 hour to8 hours  (we do backups every 6 hours).
Solution#2: Product change to SME, so before it initiates a backup, it actually re-establishes the connection before it starts. BUG/RFE 828920 has been raised to address this.

 

 

Leave a Reply

%d bloggers like this: