Exchange 2010 introduces a nifty new feature called Shadow Redundancy. Being one of the bigger changes in this version, it is well documented and discussed.
This post is on Delayed SMTP Acknowledgement, which is a subset of this feature – not Shadow Redundancy as a whole. However to fully grasp what I will be discussing, it’s important to understand a few basics about Shadow Redundancy to appreciate the purpose and spirit of Delayed SMTP Acknowledgement.
I encourage you to first travel to some of the above links, but most importantly, understand a few points:
Exchange 2007 sent messages to recipients through Transport Servers (Hub/Edge). If a Transport server were to fail with messages in its queue, these messages are lost. Generally, this is only a small amount of data loss, but loss is loss, and we want to avoid that!
To mitigate this risk you could:
A. Attempt to replay transaction logs (recover the database) from a separate disk; but this assumes the failure was limited to a single disk or database. More importantly the queue database uses circular logging by default so you cannot assume this approach will work anyway.
B. Backup your Queue databases. This sounds simple on the surface, but the database is changing each time a message is sent and received. Restoring a queue database is likely irrelevant unless you had truly continuous backups.
C. Leverage the Transport Dumpster. This feature is used for LCR/CCR environments only, but might resubmit messages in some scenarios.
Exchange 2010’s Shadow Redundancy sends the message down multiple SMTP paths (different Hub or Edge Transport servers) so that if the destination does not confirm successful delivery, another Transport server is able to submit the message. This means, we can sustain a failure of a Transport server/database, provided you have multiple servers.
Let’s take a look at a modified TechNet diagram to see an example:
Note that the Hub server sends the same message to two Edge servers. The lower edge server only submits the [shadow] message if it learns that the top edge server failed to do so.
Please understand I’m greatly simplifying this process. To fully understand all the steps involved, read the documentation linked above.
Ok, so now that we understand Shadow Redundancy, let’s ask the obvious question:
What about servers that do not support Shadow Redundancy?
A very valid concern, as this of course includes all previous versions of Exchange and most servers on the internet today.
Enter: “Delayed Acknowledgement”.
Delayed Acknowledgement is an attempt made by Exchange 2010 Transport servers to protect messages received from less sophisticated mail servers.
This is accomplished by making the sending server wait while the message is delivered behind the scenes of the 2010 environment.
Let’s explore this in more detail via the below illustration:
(Click for higher quality)
As you can see, this is a best effort attempt to protect email that does not support full Shadow Redundancy. This protection covers the scenario where your receiving Transport server fails after it accepts the message from the sending server, but before it delivers it to the user’s mailbox. If this failure were to happen, the original sending server would never get it’s acknowledgement and therefore it would be that server’s behavior to queue or resubmit the message.
See the below image to visualize this scenario:
(Click for higher quality)
So as you can see, while this isn’t as robust as true Shadow Redundancy, it does attempt to ensure messages are not lost when a Transport server fails.
Now that we see how it works, I’d like to point out some of the gotchas and configurable options:
As we saw in the first diagram, it’s possible for the sending server to think a message was delivered if the background submission takes more than 30 seconds. Because of this, messages that naturally take this long anyway (due to network conditions or latency, or whatever) will not be protected. Now, you can change 30 seconds to something higher, but you risk the sending server timing out on you.
There are additional reasons the Transport server might let the sending server “off the hook”, including:
· Submission queue in suspended state
· Message is in deferred state due to transient error
· Delivery queue is in retry or suspended state
· Delivery queue exceeds DelayedAckSkippingQueueLength value
· Message is routed to unreachable queue
So in closing, Delayed SMTP Acknowledgement is not as robust as it’s bigger brother Shadow Redundancy, but does a best-effort to protect messages in transport. You can configure the MaxAcknowledgementDelay via the Set-ReceiveConnector command.
You shouldn’t have to, but if you need to disable this feature, do so via:
Set-ReceiveConnector "ConnectorName" -MaxAcknowledgementDelay 0
See this sample scenario from TechNet:
Assume that all messages are typically delivered within 20 seconds in your environment, but due to performance requirements, you don’t want to delay acknowledgement more than 15 seconds for messages received from the Internet. After analyzing the message flow, you conclude that 95 percent of messages are delivered within the 15 second interval. This example configures the Receive connector from the Internet to delay acknowledgement for only 15 seconds. In this scenario, your environment provides shadow redundancy for 95 percent of messages received from the Internet.
Set-ReceiveConnector "From the Internet" -MaxAcknowledgementDelay 00:00:15.
· Understanding Shadow Redundancy
· Configure Shadow Redundancy
· TechNet Webcast: Deploying and Managing Microsoft Exchange Server 2010 Transport Servers
New to SP1:
Shadow Redundancy Promotion
Exchange 2010 introduced the shadow redundancy feature to minimize the loss of any message during delivery after it enters the Exchange organization. Exchange Transport servers achieve this by using the shadow redundancy SMTP protocol extension.
However, in any organization Exchange Transport servers need to communicate with other third-party SMTP servers that may not support the shadow redundancy protocol. This is especially true with Edge Transport servers that handle message traffic with various hosts on the Internet. When receiving messages from hosts that don’t support shadow redundancy in Exchange 2010 RTM, Transport servers delay sending acknowledgement to incoming messages until they verify final delivery within the organization. However, when a specific threshold was reached, the Transport server issued an acknowledgement even if final delivery wasn’t verified. This presented a scenario where messages received from hosts that don’t support shadow redundancy can be lost in transit.
To address this issue, a new feature called shadow redundancy promotion is introduced in Exchange 2010 SP1. When faced with the scenario described above, instead of issuing an acknowledgment without delivery confirmation, a Transport server now routes the message to any other Transport server within the site so that the message is protected by shadow redundancy.