One of the primary purposes behind creating the BizTalk Terminator tool was to allow users the ability to easily resolve many of the common database integrity issues identified by the BizTalk MsgBoxViewer tool. Also, with BizTalk 2010, the BizTalk product group added a new SQL Agent Job to the product called Monitor BizTalk Server that identifies many of these same database integrity issues.  The purpose of this post is to illustrate how BizTalk Terminator can be used to resolve database integrity issues identified by MBV or the BTS2010 Monitor BizTalk Server job.

Looking for info on where to download Terminator?  Go here
Terminator Not Cleaning Up Caching Items?  Go here



Resolving issues identified by the BTS2010 Monitor BizTalk Server Job
One of the new features introduced in BizTalk 2010 is the Monitor BizTalk Server SQL Agent job.  This job was added to the product to provide an out-of-the-box way to identify database integrity issues that we previously had to use MBV or manual SQL scripts to identify.  If this job identifies a database integrity issue, the job itself will fail with a custom error message that describes the problem that was identified.  A database admin or a monitoring tool like SCOM can take a look at SQL Agent to identify when a problem has occurred.  See the BizTalk Server Monitoring section at http://msdn.microsoft.com/en-us/library/aa577454 (BTS.70).aspx for more information about this job.
Terminator provides the easiest way to resolve most of these issues once they have occurred.  Use the steps below to resolve issues identified by  the Monitor BizTalk Server Job.
 1.       Before running Terminator, always make sure you have a BizTalk Backup of your databases, all the BTS hosts have been stopped, BTS SQL Agent jobs have been disabled.
 2.       Use Table 1 below to determine the Terminator View and Resolution Tasks as well as the Terminator Task Category for the tasks associated with the issue identified by the BizTalk Monitor Job
 3.     Open Terminator to enter the initial screen. 
 4.     Specify the SQL server and database information for the BizTalk Management Database.
 5.     Check the checkboxes to acknowledge that you have completed step #1 above and click Connect.
 6.       Once connected, click the appropriate button depending on the Terminator Task Category
 7.       Select the appropriate task from the list, click the Parameters tab and populate any parameter that needs to be set and click the Execute button.
  Table 1 provides a summary of all the common database integrity issues identified by the Monitor job and the associated task in Terminator that is used to resolve or view the issue.  Note that some issues do not have any resolution options and will require troubleshooting by Microsoft Support.

Issue Identified by Monitor BizTalk Server JobTerminator Resolution TaskTerminator View Task
Messages without any referencesRepair Refcounts for All Messages
(in Manage task category)
No view task - rely on the Monitor job to identify these
Messages without reference countsRepair Refcounts for All Messages
(in Manage task category)
View Count of Messages Without RefCounts
Messages with reference count less than 0Repair Refcounts for All Messages
(in Manage task category)
No view task - rely on the Monitor job to identify these
Message references without spool rowsNo resolution task - contact MS supportNo view task - rely on the Monitor job or MBV to identify these
Message references without instancesDELETE Orphaned Messages in All Hosts
DELETE Orphaned Messages in Specific Host
(both in Delete task category)
View Count of Orphaned Messages by Host
Instance state without instancesNo resolution task - contact MS supportNo view task - rely on the Monitor job or MBV to identify these
Instance subscriptions without corresponding instancesDELETE Orphaned Instance Subscriptions(in Delete task category)View Count of Orphaned Instance Subscriptions
Orphaned DTA service instancesRepair Orphaned DTA Service Instances
 (in Manage task category)
View Count of Orphaned DTA Service Instances
Orphaned DTA service instance exceptionsDELETE Orphaned DTA Service Instance Exceptions
(in Delete task category)
View Count of Orphaned DTA Service Instance Exceptions 
TDDS is not running on any host instance when global tracking option is enabledNo resolution task within terminator - BizTalk Administrator needs to make sure one of the hosts has Tracking enabled and there is a running host instance for that hostNo view task - rely on the Monitor job or MBV to identify these
   Table 1.  BTS Monitor job-identified Issues and Associated Terminator Tasks  





Resolving issues identified by BizTalk MsgBoxViewer 
First, MBV is always being improved and updated so make sure you are always running the latest version. Below are some relevant MBV links:
- MBV Blog - http://blogs.technet.com/jpierauc/
- MBV FAQ - http://blogs.technet.com/b/jpierauc/archive/2008/07/22/msgboxviewer-mbv-q-a.aspx
- MBV Download location - http://blogs.technet.com/jpierauc/pages/msgboxviewer.asp x
 There are 2 ways to use Terminator to resolve issues identified by MBV.  This document will outline both options in detail:
 1.       MBV Integration – With a minimum of MBV version 10.13, MBV will output MBVCleanupScripts.xml which Terminator can consume to automatically provide the user with a list of Tasks that can be used to resolve identified issues.  Not all issues identified by MBV and addressable with Terminator are supported by MBV integration.
 2.       Manual Task Selection – The user can look at the MBV html output and manually select the appropriate tasks in Terminator
Table 2 provides a summary of all the common database integrity issues identified by MBV and the associated task in Terminator that used to resolve or view the issue.  Note that some issues require manual integration and some do not have any integration options and will require troubleshooting by Microsoft Support.


Issue Identified by MBV
Resolution Options
Terminator Resolution Task
Terminator View Task
Root Cause
Orphaned RFRs
MBV Integration or Manual Task Selection
DELETE Orphaned RFRs in all Hosts
DELETE Orphaned RFRs in Specific Host
 (both in Delete task category)
View Count of RFRs By Host
This is due to a known bug and there is a hotfix available.  See KB 941690 for details.
Orphaned Cache Instances
MBV Integration or Manual Task Selection
Terminate Caching Instances
 (in Delete task category)
View Count of Cache Messages in All Host Queues
 View Count of Cache Instances in All Hosts
This is due to a known bug and there is a hotfix available.  See KBs 944426 & 936536 for details.
Orphaned DTA Service Instances
MBV Integration or Manual Task Selection
Repair Orphaned DTA Service Instances
 (in Manage task category)
View Count of Orphaned DTA Service Instances
This is due to a known bug and there is a hotfix available.  See KB 978796 for details.  This can also be due to running a cleanup script against trackingdata_x_x or instances table.
Orphaned DTA Service Instance Exceptions
MBV Integration or Manual Task Selection
DELETE Orphaned DTA Service Instance Exceptions
(in Delete task category)
View Count of Orphaned DTA Service Instance Exceptions 
No known root cause
Message Refs w/out Spool rows
None
No cleanup task – contact MS support.
No view task – contact MS support.
No known root cause
Messages w/out RefCounts
MBV Integration or Manual Task Selection
Repair Refcounts for All Messages
(in Manage task category)
View Count of Messages Without RefCounts
This is due to an issue in BTS reference counting logic.
Message Refs w/out Instances
MBV Integration or Manual Task Selection
DELETE Orphaned Messages in All Hosts
DELETE Orphaned Messages in Specific Host
(both in Delete task category)
View Count of Orphaned Messages by Host
Only known root cause are Orphaned RFRs but we have seen other types of message refs w/o instances and there is no know root cause for those
Subscriptions w/out Instances
Manual Task Selection
DELETE Orphaned Instance Subscriptions
(in Delete task category)
View Count of Orphaned Instance Subscriptions
No known root cause
InstanceState w/out Instances
None
No cleanup task – contact MS support.
No view task – contact MS support.
No known root cause
Missing Control Messages
MBV Integration or Manual Task Selection
Repair Control Messages
(in Manage task category)
View Missing Control Messages
No known root cause
Large number of Suspended Messages
Manual Task Selection Only
Resume Instances or Terminate Instances and select Host, Class, and Status as appropriate.
(Resume in Manage task category and Terminate in Delete task category)
Count Instance (and Save Messages)
Could be due to various causes that lead instances to get suspended.  BTS Admin will need to look at the error message associated with the suspended items to investigate root cause.
Large MarkLog Table
MBV Integration or Manual Task Selection
PURGE Marklog Table
(in Delete task category)
View Table Row Count (No SQL Statistics)
Due BTS backup job’s backup on mark logic.  Mark is created across all DBs involved in backup so restores can be done in a transactionally consistent state. 
 Table 2.  MBV-identified Issues and Associated Terminator Tasks  

MBV Integration
This feature allows the user to resolve many of the common database integrity issues MBV identifies without having to analyze the MBV output or determine which task in Terminator is associated with which issue.  MBV 10.13 or above is required for this functionality and will produce an MBVCleanupScripts.xml file used by Terminator for MBV integration.
How to use Terminator's MBV mode for MBV Integration:
1.       Run MBV and verify that an MBVCleanupScripts.xml file has been created in the output folder.
2.       Before running Terminator, always make sure you have a BizTalk Backup of your databases, all the BTS hosts have been stopped, BTS SQL Agent jobs have been disabled.
3.     Open Terminator to enter the initial screen (Figure 1). 
4.     Specify the SQL server and database information for the BizTalk Management Database.
5.     Check the checkboxes to acknowledge that you have completed step #2 above and click Connect.
6.       Once connected, click the Import MBV button (Figure 2) and browse to the MBVCleanupScripts.xml file created in step #1.
7.       Notice that the dropdown list says "Select a MBV task from this drop down" (Figure 3) and clicking the dropdown provides a list of all tasks that have been selected to resolve all the issues that MBV identified (and support MBV integration mode).
8.       Execute each task from the list to resolve the issues identified by MBV.  Note that the Parameters tab must be clicked to enable the Execute button.  Make sure all params are populated correctly before executing a task.  The user can re-run MBV or use the Terminator View tasks mentioned in Table 2 to verify that the issues have been resolved.

Figure 1.  Initial Terminator Screen
Figure 2.  Import MBV
Figure 3.  MBV-identified task list




Important considerations for MBV Integration:
 ·         MBV only creates an MBVCleanupScripts.xml file if it identifies an issue that is addressable with Terminator using MBV Integration.  
 ·         MBVCleanupScripts.xml will not be created if MBV only identifies issues that require manual task selection to resolve and do not support MBV Integration.
 ·         Once the MBV file has been imported, use the MBV Task Category button and other Task Category buttons to go in and out of MBV mode
 ·         There are currently 8 Terminator tasks that can be pre-selected by MBV Integration: 
 1.        DELETE Orphaned RFRs in all Hosts
 2.        Terminate Caching Instances 
 3.        Repair Orphaned DTA Instances
 4.        DELETE Orphaned DTA ServiceInstance Exceptions
 5.        Repair Refcounts for All Messages
 6.        DELETE Orphaned Messages in All Hosts
 7.        Repair Control Messages
 8.        PURGE Marklog Table
  
Manual Task Selection
In some cases, the user prefers to analyze the MBV output html and select the Terminator tasks in a manual fashion. 
Manual Task Selection Steps:
 1.       Before running Terminator, always make sure you have a BizTalk Backup of your databases, all the BTS hosts have been stopped, BTS SQL Agent jobs have been disabled.
 2.       Use the below sections of this document to identify an issue mentioned in the Warnings and Summary Report section of the MBV output html and note down the Terminator View and Resolution Tasks as well as the Terminator Task Category.
 3.     Open Terminator to enter the initial screen. 
 4.     Specify the SQL server and database information for the BizTalk Management Database.
 5.     Check the checkboxes to acknowledge that you have completed step #1 above and click Connect.
 6.       Once connected, click the appropriate button depending on the Terminator Task Category
 7.       Select the appropriate task from the list, click the Parameters tab and populate any parameter that needs to be set and click the Execute button.










 Orphaned RFRs in Critical Warnings section of Warning and Summary Report:
  
MsgBox Database Integrity <>  
Total Orphaned 'Routing Failures' messages10991 (Possible Known Issue in BizTalk 2006) !!  KB 941690    Get More Info on 'TERMINATOR' Tool to clean or repair this issue ,    
  
Click Query Report button to see further details:
  
MsgBox Dbs : MsgBox Integrity Details (SQL Query, 4 Rules)
     - Messages count with no associated instances grouped by classID
MsgBox DbHostNameuidClassIDcount
SERVER – BizTalkMsgBoxDbMy32BitHosteaf8eea9- 366a-4cde-8dd3-57a4c39bf8e5 (Routing Failures Class)10991
SERVER - BizTalkMsgBoxDbMy32BitHost59f295b0- 3123-416e-966b-a2c6d65ff8e6 (Messaging Class)10020
SERVER – BizTalkMsgBoxDbMy32BitHostbb3a1470- f5c4-47c3-b71f-eaabc260fbd0 (Caching Class)10033
3 Rows   
  
 Terminator View Task:
 · View Count of RFRs By Host (in View category) 
Terminator Resolution Tasks:
 · DELETE Orphaned RFRs in all Hosts (in Delete category)
 · DELETE Orphaned RFRs in specific Host (in Delete category)
   
 Orphaned Cache Instances in Critical Warnings section of Warning and Summary Report:
  
Other MsgBox Checks <>  
Total active 'Cache msg' for MsgBox MSGBOX DB 1 (MASTER) "BizTalkMsgBoxDb" on SERVER5009 (they do NOT appear in HAT or BTS MMC and large number can impact MsgBox perfs) !!  kb 944426 - Orphaned cache instances may be built in the Instances table    You experience performance issues with BizTalk Server 2006 and throttling messages are logged in the performance log file      ,        
  
Click Query Report button to see further details:
  
MsgBox Dbs : Active Cached Msg Ref in Spool (SQL Query, 2 Rules)
     - MSGBOX DB 1 (MASTER) "BizTalkMsgBoxDb" on SERVER
Host Qcount
BizTalkServerApplicationQ0
My32BitHostQ5009
CreateTestHostQ0
TestJoBailQ0
4 Rows 
  
Terminator View Task:
 ·   View Count of Cache Messages in all Hosts Queues (in View category). 
 ·   View Count of Cache Instances in all Hosts (in View category).  Note that while MBV shows the number of orphaned cache messages, this Terminator task shows the number of cache instances.  One cache instances can have thousands of cache messages associated with it.

Terminator Resolution Task:
 ·   Terminate Caching Instances (in Delete category).  Note that while MBV shows the number of orphaned cache messages, Terminator terminates cache instances.  One cache instances can have thousands of cache messages associated with it.
  
 Orphaned DTA Service Instances in Critical Warnings section of Warning and Summary Report:
  
DTA Tables <>  
DTA Orphaned Instances (Incompleted Instances in DTA but not in Msgbox)10989 (Large number can impact DTA Size and so perfs) !!  Get More Info on 'TERMINATOR' Tool to clean or repair this issue 
  
Click Query Report button to see further details:
  
DTA Db : Orphaned Svc Instances (SQL Query, 2 Rules)
Instances Start DateOrphan Instances Count
01/15/20092
02/24/20091566
04/08/20099416
10/06/20081
10/07/20083
11/05/20081
6 Rows 
  
Terminator View Task:
 ·   View Count of Orphaned DTA Service Instances (in View category) 
Terminator Resolution Task:
 ·   Repair Orphaned DTA Service Instances (in Manage category)
   
Orphaned DTA Service Instance Exceptions in Non Critical Warnings section of Warning and Summary Report:
  
Other Checks <>  
"ServiceInstanceExceptions" DTA tableSome entries do not have a instance in dta_ServiceInstances : 10100 !  Get More Info on 'TERMINATOR' Tool to clean or repair this issue 
   
Click Query Report button to see further details:
  
DTA Db : Orphaned serviceInstanceExceptions (SQL Query, 1 Rule)
Count
10100
1 Rows
  
Terminator View Task:
 ·   View Count of Orphaned DTA Service Instance Exceptions (in View category)  
Terminator Resolution Task:
 ·   DELETE Orphaned DTA Service Instance Exceptions (in Delete category)
   

 Messages w/out RefCounts in Non Critical Warnings section of Warning and Summary Report:
  
MsgBox Database Integrity <>  
Total Messages without Ref Count11370 (Large number can impact MsgBox Dbs size and so perfs) !  Get More Info on 'TERMINATOR' Tool to clean or repair this issue 
  
Click Query Report button to see further details:
  
MSGBOX DB 1 (MASTER) "BizTalkMsgBoxDb" on SERVERMessages w/out RefCounts11370
   
Terminator View Task:
 ·   View Count of Messages Without RefCounts (in View category)
   
Terminator Resolution Task:
 ·   Repair Refcounts for All Messages (in Manage category)
   
 Message Refs w/out Instances in Non Critical Warnings section of Warning and Summary Report:
  
MsgBox Database Integrity <>  
Total Orphaned Messages not belonging to MSMQT or Routing Failure Svc Class20053 (Large number can impact MsgBox Dbs size and so perfs) !  Get More Info on 'TERMINATOR' Tool to clean or repair this issue 
  
Click Query Report button to see further details.  There are 20053 messages without instances not including the 10991 orphaned RFRs:
  
MsgBox Dbs : MsgBox Integrity Details (SQL Query, 4 Rules)
     - Messages count with no associated instances grouped by classID
MsgBox DbHostNameuidClassIDcount
SERVER - BizTalkMsgBoxDbMy32BitHosteaf8eea9- 366a-4cde-8dd3-57a4c39bf8e5 (Routing Failures Class)10991
SERVER – BizTalkMsgBoxDbMy32BitHost59f295b0- 3123-416e-966b-a2c6d65ff8e6 (Messaging Class)10020
SERVER – BizTalkMsgBoxDbMy32BitHostbb3a1470- f5c4-47c3-b71f-eaabc260fbd0 (Caching Class)10033
  
Go to MsgBox Dbs : MsgBox Integrity (SQL Query, 3 Rules) to see more details.  This a total count including orphaned RFRs and orphaned MSMQT messages:
  
MSGBOX DB 1 (MASTER) "BizTalkMsgBoxDb3" on SERVERMessage Refs w/out Instances31044
     
Terminator View Task:
 ·   View Count of Orphaned Messages by Host (in View category)  
Terminator Resolution Task:
 ·   DELETE Orphaned Messages in All Hosts (in Delete category).  Note that this task will also clean up orphaned RFRs and orphaned MSMQT messages.
 ·   DELETE Orphaned Messages in Specific Host (in Delete category).  Note that this task will also clean up orphaned RFRs and orphaned MSMQT messages.
  
 Missing Control Messages in Critical Warnings section of Warning and Summary Report:
  
 MsgBox Database Integrity  <>  
'Control Messages' CheckMissing Restart Msg in Spool table (can prevent some svc instances to be resumed, terminated or suspended) !!  Get More Info on 'TERMINATOR' Tool to clean or repair this issue 
'Control Messages' CheckMissing Suspend Msg in Spool table (can prevent some svc instances to be resumed, terminated or suspended) !!  Get More Info on 'TERMINATOR' Tool to clean or repair this issue 
'Control Messages' CheckMissing Terminate Msg in Spool table (can prevent some svc instances to be resumed, terminated or suspended) !!  Get More Info on 'TERMINATOR' Tool to clean or repair this issue 
'Control Messages' CheckMissing ResumeInDebugMode Msg in Spool table (can prevent some svc instances to be resumed, terminated or suspended) !!  Get More Info on 'TERMINATOR' Tool to clean or repair this issue 
  
Terminator View Task:
 ·View Missing Control Messages (in View category)
  
Terminator Resolution Task:
 ·   Repair Control Messages (in Manage category). 
  
 Large Number of Suspended Messages in Non Critical Warnings section of Warning and Summary Report:
  
MsgBox Queues <>  
Total Suspended Messages for BizTalkMsgBoxDb348926 - Large number can impact MsgBox db Perfs !  BizTalk Server : Monitoring and Troubleshooting    BizTalk Operation Guide - p388      ,        
  
Click Query Report button to see further details:

MsgBox Dbs : BizTalk Queues (SQL Query, 4 Rules)
     - MSGBOX DB 1 (MASTER) "BizTalkMsgBoxDb3" on SERVER
QueueCountResumables
My32BitHostQ10224 
My32BitHostQ_Suspended4892648926
InstanceStateMessageReferences_My32BitHost0 
TestJoBailQ0 
TestJoBailQ_Suspended00
InstanceStateMessageReferences_TestJoBail0 
CreateTestHostQ0 
CreateTestHostQ_Suspended00
InstanceStateMessageReferences_CreateTestHost0 
BizTalkServerApplicationQ0 
BizTalkServerApplicationQ_Suspended00
InstanceStateMessageReferences_BizTalkServerApplication0 
12 Rows  
  

Terminator View Task:
 ·   Count Instances (and Save Messages) (in View category)
  
Terminator Resolution Task:
 ·   Resume Instances (in Manage category).  User must verify suspended items need to be resumed before executing.
 ·   Terminate Instances (in Delete category).  User must verify suspended items need to be terminated before executing.
  
 Large MarkLog Table in Non Critical Warnings section of Warning and Summary Report:
  
MsgBox Tables <>  
MarkLog Table Rows in BizTalkMsgBoxDb3150000 starts to be a large number - Known issue caused by BackupJob !  Contact CSS to get 'TERMINATOR' Tool to clean or repair this 
  


Terminator View Task:
 ·View Table Row Count (No SQL Statistics)  (in View category).  Find MarkLog table in the output list.  
Terminator Resolution Task:
 ·PURGE Marklog Table (in View category)