MSMQ Vulnerabilities (Cont.) - Outgoing Messages blocked with 'Unacknowledged' state

This is another annoying MSMQ issue that consumed most of my time. One of my previous articles, I explained about a MSMQ message rejection issue due to a bug on Microsoft hot-fix and service packs. This time it was more complicated to find out the root cause since there were no errors logged at both client and server sides.


Symptoms:


If you have a WCF with MSMQ binding in your application, in certain instances, Messages get stuck on Outgoing Queues with ‘Unacknowledged’ state due to no reason. The application logs say, it has successfully delivered to the MSMQ. However, those messages were not consumed and also this issue occurred in a random manner.



Analysis of the problem:


Mainly, if the MSMQ Messages block under the Outgoing Queues with ‘Unacknowledged’ state, which means:

  1. The application successfully delivered messages to the Outgoing Queues.

  2. The Outgoing queue delivered those messages to the destination queue.

  3. However, since you use transactional MSMQs, the destination MSMQ Server didn’t send Acknowledgement back to the Sender MSMQ Server.

  4. Therefore, those messages remain under the Outgoing Queue’s with Unacknowledged State at the Senders side.


If you consider the possibilities for the above issue, mainly could be happened due to:

  1. Once the destination MSMQ Server received messages, it sends the Acknowledgement back to the Sender MSMQ Server, however, due to network issues; the Acknowledgement didn’t reach the Sender.

  2. Once the destination MSMQ Server received messages, it sends the Acknowledgement to a wrong IP address.

  3. Once the destination MSMQ Server received messages, it couldn’t send the Acknowledgement due to MSMQ component issues.


In this article, I am considering the above possibility #2, that sending Acknowledgement to a wrong IP address. Mainly, this can be occurred, if you have more than one cloned machines that work as client (MSMQ Sender) and they are belongs to the same master image.



Root Cause:


If you look at the following registry path of the Cloned machines (belongs to the same image), the QMid registry value is same on each those machines.

HKLM\Software\Microsoft\MSMQ\Parameters\Machine Cache


The root cause is, when Client communicates the Server, the Server cache the Sender’s QMid value (which come along with incoming messages) with the Clients IP address. This cache table gets cleared in a regular basis. So the Server sends the Acknowledgement back to the Client by looking at its cache after the 1st time communication.


Therefore, when more than one cloned Client machines belongs the same image send messages to the Server, the Server will send ‘Acknowledgement’ to the same IP address by looking at its cache.


So, one Client machine receives many additional Acknowledgements and it discards them, nevertheless, the other original senders never receive Acknowledgements for its outgoing messages. Hence the messages remain as ‘Unacknowledged’ in Outgoing Queues.


Therefore, it is clear that in certain instances you can see the Outgoing messages successfully transferred (during 1st time communication or when cache got cleared.) however, the outgoing messages get blocked in a regular basis for sure.



Resolution:


  1. Locate “HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSMQ\Parameters\Workgroup” registry key and make sure the value is “1” on each Cloned machine

  2. Stop the MSMQ Service

  3. Clear the QMId value completely

  4. Add a SysPrep DWORD (Under HKLM\Software\Microsoft\MSMQ\Parameters) and set it to 1

  5. Start the MSMQ Service


Next Steps:


  • Make sure to exclude the MSMQ component when you are cloning a machine.



Reference:


http://blogs.msdn.com/b/johnbreakwell/archive/2007/02/06/msmq-prefers-to-be-unique.aspx
3 comments

BizTalk Server Optimization & Maintenance

Download this post as Word | PDF

I have engaged with BizTalk Server 2006 R2 application development, maintenance & performance tuning processes during my past projects. I have learnt that when it comes to the production server maintenance, it is very important to proactively monitor and optimize the BizTalk & SQL Server in a timely manner.
There are many numbers of articles available under the BizTalk Server optimization. Following are the summarizations of some key steps that you need to be followed.
Mainly, I am categorizing it as:
  • Infrastructure Support
  • BizTalk Application Server Maintenance
  • BizTalk SQL Server Maintenance

Infrastructure Support

Having a good Infrastructure is very important due to its dependencies that are directly affecting your BizTalk Server performance.
  • Have servers with very good performance by considering CPU, DISKS (SANs), Memory (> 8 GB), Sufficient Network bandwidth, Etc.
  • Use 64-Bit Operating System and also BizTalk Server, SQL Server 64-Bit Enterprise Editions.
    • There are some significant performance differences between 32 Bit & 64 Bit.
    • For example the 64-bit virtual address space is exponentially larger than the 32-bit virtual address space. Moving to 64-bit will effectively resolve most of the BizTalk Memory Leaks issues.
  • Your BizTalk Server should be implemented in a Multi Hosted environment at least with 2/3 Servers by dividing Send, Receive, Process Host Instances respectively.
  • You should need to think about the I/O Contingency for heavy disk read/write processes and allocate separate disks such as SANs to isolate heavy I/O processes.
  • Maintain separate Backup Servers.
  • Customize your Anti Virus, File System Scan processes efficiently by excluding your BizTalk, SQL heavy read/write folders.
  • Keep your O/S & SWs up-to-date.
  • Implement monitors such as SCOM in order to proactively monitor the health of BizTalk Server and SQL Server.

BizTalk Application Server Maintenance
  • Create separate host instances & handlers for Send, Receive & Process. (If is it in the multi-hosted environment, create them respectively in separate servers). Also create a dedicated host for tracking purposes.
  • Disable tracking for orchestrations, send ports, receive ports and pipelines as it not required.

Note: - Tracking may cause the potential performance overhead for BizTalk Server.


BizTalk SQL Server Maintenance

Consider the following performance guidelines I gathered from Microsoft articles and I have added some additional details regarding configuring Microsoft SQL Server™ with BizTalk Server 2006:
  • Whenever possible, use a fast disk subsystem with SQL Server. Use a redundant array of independent disks type 10 (RAID10/0+1) or a storage area network (SAN) with backup power supply.
  • Isolate each MessageBox database on a separate server from the BizTalk Tracking database (BizTalkDTADb). For smaller deployments if CPU resources are available, it might be sufficient to isolate the MessageBox database on a separate physical disk from the BizTalk Tracking database.
  • The primary MessageBox database could be the bottleneck due to CPU processor saturation or latency from disk operations (average disk queue length). If CPU processing is the bottleneck, add CPU processors to the primary MessageBbox. If not, try to disable publishing on the master MessageBox database. This way the master MessageBox database can more efficiently handle routing of messages to the other MessageBox databases
  • If disk operations are the bottleneck, move the BizTalk Tracking database to a dedicated SQL Server computer and/or dedicated disk. If CPU processing and disk operations on the primary MessageBox database are not the bottleneck, you can create new MessageBox databases on the same SQL Server computer to leverage your existing hardware.
  • Follow SQL Server best practices to isolate the transaction and data log files for the MessageBox and BizTalk Tracking databases onto separate physical disks.
  • Allocate sufficient storage space for the data and log files. Otherwise SQL Server will automatically consume all of the available space on the disks where the log files are kept. The initial size of the log files depends on the specific requirements in your scenario. Estimate the average file size in your deployment based on testing results, and expand the storage space before implementing your solution.
  • Allocate sufficient storage space for high-disk-use databases, such as the MessageBox, Health and Activity Tracking (HAT), and Business Activity Monitoring (BAM). If your solution uses the BizTalk Framework messaging protocol, allocate sufficient storage space for the BizTalk Configuration database (BizTalkMgmtDb).
  • Configure BizTalk SQL Jobs properly. There are almost 12 SQL Jobs available for BizTalk. If they were not configured them properly or not running, there can be some severe BizTalk performance issues. Ex:- BizTalk Backup Jobs, DTA Purge and Archive Jobs
  • Depending on business needs, such as data retention periods, and the volume of data processed in your scenario, configure the Archive/Purge jobs on the HAT-Tracking database such that the BizTalk Tracking database does not grow too large. The growth of this database can degrade performance because reaching the full capacity of the database imposes a limit on the rate of data insertion. This is especially true when especially when one BizTalk Tracking database supports multiple MessageBox databases.

Use following extensive values for the “DTA Purge and Archive (BizTalkDTADb)” job for better performance

declare @dtLastBackup datetime

set @dtLastBackup = GetUTCDate()

exec dtasp_PurgeTrackingDatabase 1, 0, 1, @dtLastBackup

  • Scale up the servers hosting the MessageBox and BizTalk Tracking databases if they are the bottleneck. You can scale up the hardware by adding CPUs, adding memory, upgrading to faster CPUs, and using high-speed dedicated disks.
  • Splitting the TempDB files across multiple files may resolve performance issues related to I/O operations. As a general guideline, create one file data file per processor and use the same size for all files created.
  • Change the database auto-grow settings to a fixed value such as 100-150MB. By default the database growth is configured to 10%, which can lead to delays when growing larger databases.
  • SQL Server memory should be set to a fixed value by setting both Min Server Memory and Max Server Memory to the same value. In general, allocate 75% of physical memory to SQL Server and leave 25% for the rest of the operating system and any applications. If this is a dedicated SQL Server, you can decrease the amount reserved for the operating system to a minimum of 1GB.
  • Make sure the BizTalk Database backup jobs are configured properly.

0 comments

Ride Hard or Stay Home!!!

Sometimes our lives are really suck and very hard to survive due to our job, responsibilities or due to our own madness or stupidity. We may need strong guts to face the reality and overcome adversity.

No matter what we should never give up and I called it as “Ride Hard or Stay Home!!!”


Cheers..

0 comments

Forward Outlook Emails to your Pager or to your Public Email Account

Last few weeks I was searching the ways of forwarding Outlook incoming emails to my mobile or to a public email account. After writing few code blocks and testing various SMTP related methodologies the best option I found so far is the Outlook Macro option.

Using Outlook Macro option you can write a custom VB code as the way you want and trigger it by using Outlook events such as when a new email received, send, etc. You can easily forward your Outlook emails to any number of public email addresses, your mobile Pager address in a totally customized way. You will receive the forwarded email to your public mail account or to your mobile as same way you send through the Outlook.

Here is the code I have written and try it with your own risk.

  1. MS Outlook 2007 > Tools > Macro > Visual Basic Editor
  2. Copy & paste below code
  3. Edit and provide your public email address within the code, Save and Run it.
  4. This will trigger when you receive a new email address and forward the content to given mail account

'This event trigger each time when a new email received to your Outlook

Private Sub Application_NewMail()

Dim sSubject As String

Dim sBody As String

Dim flag As String

' Open the default Inbox Folder.

Set objItemInbox = Application.GetNamespace("MAPI"). _

GetDefaultFolder(olFolderInbox).Items.GetLast

' Open the custom mail box folder that you have created.

Set objItemLA = Session.Application.GetNamespace("MAPI"). _

Folders("Personal Folders").Folders("LA Support").Items.GetLast

flag = 1

' If unread mail exist in your inbox, retrieve the 1st unread mail and assign details to the variables.

If objItemInbox.UnRead Then

sSubject = objItemInbox.Subject & " From: " & _

objItemInbox.SenderName

sBody = objItemInbox.Body

' If unread mail exist in your custom mail box, retrieve the 1st unread mail and assign details to the variables.

ElseIf objItemLA.UnRead Then

sSubject = objItemLA.Subject & " From: " & _

objItemLA.SenderName

sBody = objItemLA.Body

Else

flag = 0

End If

If flag = 1 Then

' Create a new outlook email Object using received email contents

Set objMailItem = Application.CreateItem(olMailItem)

With objMailItem

.Subject = sSubject

.Body = sBody

'Add your preferred public email account

.Recipients.Add "YourPublicEmail@gmail.com"

.Send

End With

Set objMailItem = Nothing

End If

End Sub

0 comments

Some days are better than other days..

It is true that some days are better than other days.

Its like:-
When I turn ON the radio, most rocking songs are playing & there are sensational..
When I am driving, I could see lots of stunning girls everywhere and that's phenomenal..
When I start working, I feel no dull and receive appreciation for a spectacular performance..
Also I am definitely enjoying with my fabulous breakfast for sure..

I am just wondering how we could make everyday like that awesome. At least most of the days.

0 comments

Patching MSMQ Vulnerabilities

I was working with WCF Technologies in one of my past projets and some of the Microsoft bugs really freak me out. Mainly, we have a WCF channel with NetMsmqBinding and MSMQ channel is authenticated through Certificates in a bi-directional way. In certain instances, we have identified that received MSMQ messages become poisoned due to no reason.

This is an identified Microsoft bug in MSMQ 3.0 version in following OS, Service Packs and patches; however it is hard to find a link explaining all affected OS and Service Packs.

Symptom:-

When we have a WCF Channel with MSMQ Binding, and we receive some messages from another source using an external certificate, the MSMQ 3.0 is rejecting those received messages by considering it as a poisoned message.

Root Cause:-

This is a known issue in the MSMQ 3.0 version in following OS+ Service Packs+ Patches.

Microsoft Windows Server 2003 + Service Pack 2 + KB 971032 (KB971032 cause this issue)

Microsoft Windows XP + Service Pack 2 + KB 971032 (KB971032 cause this issue)

Microsoft Windows XP + Service Pack 3 (Service Pack 3 cause this issue)

Resolution:-

Affected SW

Resolution

Microsoft Windows Server 2003 + Service Pack 2 + KB 971032

Install following hot-fix to resolve this issue

http://support.microsoft.com/kb/2028919

Microsoft Windows XP + Service Pack 2 + KB 971032

Upgrade to Service Pack 3 and install following hot-fix to resolve this issue

http://support.microsoft.com/kb/959682

Microsoft Windows XP + Service Pack 3

Install following hot-fix to resolve this issue

http://support.microsoft.com/kb/959682
0 comments
Subscribe to: Posts (Atom)