Error Alerting and Troubleshooting

Rick Wattras -

Error alerts

Building on top of the built-in Mirth functionality

By default, Mirth has some Alert functionality built-in that can watch certain channels and trigger an action when it notices a failure. This is covered more in-depth in a separate portion of this guide, in the "Alerts" section of Navigating Mirth

However, we devised a way to have a little more control over error alerting while still using the Alerts framework as the foundation for the triggers themselves. We use the "Channel" Protocol in the Actions panel to have Mirth trigger when it sees an alert and then send the content of that error to a specific channel we setup for later processing.

Here are the steps you can take to create this kind of Alert and build a channel to handle it:

  1. Create a new channel in Mirth, named Error Alerter or something relevant to its intended use. Do not deploy the channel yet, but make sure it's enabled.
  2. On the Summary tab, click 'Set Data Types' and set all Inbound and Outbound properties to "Raw".
  3. On the Source tab, set the Source Connector Type to a Channel Reader. No need to do too much else with it for now.
  4. Go to the Alerts page in Mirth and create a new Alert. You can name it whatever you like but something simple like 'Alerts' usually suffices.
  5. At the top left in the "Errors (select all that apply)" section, select the 'Any' checkbox to tell the alerter to trigger on any type of error.
  6. The alerter should be enabled by default, but make sure the Enabled checkbox is checked anyway.
  7. On the right under Channels, select each of the channels that you wish to monitor for errors and click Enable.
    1. Be careful not to enable alerting for the same channel that you will be sending alerts to - this can lead to a vicious loop of failed messages being continuously routed to the alerter channel until manually stopped or potentially until the database runs out of space from trying to store all of the repeated messages.
  8. On the bottom left under Actions, set the Protocol to 'Channel' and the Recipient to the error alerts channel you created earlier (ie. 'Error Alerter').
  9. In the template box, you can decide how you want to pass the error information along but we find a JSON-like format with the following structure to be the most useful and easiest to parse later:

    {
    "channel": "${channelName}",
    "date": "${date}",
    "errorMessage": "${errorMessage}",
    "errorType": "${errorType}"
    }
    • That gets most of the relevant information that you'll need later.
    • If you do customize this another way, note that you'll have to deviate from some of the setup steps later in order to handle your custom message format.
  10. That's it to the Alert itself. Click save and if you're prompted with a warning about configuring SMTP setting, just click OK and continue on.
  11. Head back to the 'Error Alerter' channel you created and go to the Source tab and click Edit Transform
  12. Create a new step. If you used the suggested JSON from before, paste in the following code

    var msgJSON = JSON.parse(msg.toString());
    var channel = msgJSON.channel;
    var date = msgJSON.date;
    var errorMessage = msgJSON.errorMessage;
    var errorType = msgJSON.errorType;
    
    channelMap.put("channel", channel);
    channelMap.put("errorDate", date);
    channelMap.put("errorMessage", errorMessage);
    channelMap.put("errorType", errorType);
    • This grabs the error info from the Alert and makes it available for use in the channel destination.

 

Now you have Mirth watching certain channels and when it notices an error it kicks out a succinct JSON object containing all of the relevant information in a parseable format. You can setup the destinations on the Error Alerter channel to send to wherever you want; some common uses are an email sender and/or Slack integration, both of which we'll cover in a later section below.

 

Handling Stack Trace errors

If you find that the errors being triggered aren't fitting snugly into a JSON format (and thus are ERRORing in the dashboard of your Error Alerter channel), which can be due to how Mirth will sometimes spit out the full JavaScript stack trace if an exception was thrown, then you may want to use this as your transformer instead:

var errorJsonString = msg.toString();
var msgJSON;
try {
    msgJSON = JSON.parse(errorJsonString);
} catch (err) {
    var errMsgInd = errorJsonString.indexOf("errorMessage");
    var errTypeInd = errorJsonString.indexOf("errorType");
    var errSub = errorJsonString.substring(errMsgInd + 15, errTypeInd - 3);

    errorJsonString = errorJsonString.replace(errSub, "\"Full stack error; check channel dashboard for full details\"");

    // remove control characters
    errorJsonString = errorJsonString.replace(/[\x00-\x1F\x7F-\x9F]/g, "");

    msgJSON = JSON.parse(errorJsonString);
}

var channel = msgJSON.channel;
var date = msgJSON.date;
var errorMessage = msgJSON.errorMessage;
var errorType = msgJSON.errorType;

channelMap.put("channel", channel);
channelMap.put("errorDate", date);
channelMap.put("errorMessage", errorMessage);
channelMap.put("errorType", errorType);
  • This allows it to check for a stack trace error and replace it with a message alerting the recipient to go and manually check the error message in the dashboard. It works by first trying to parse the error JSON and if that fails due to the full stack formatting, then it extracts out the error text and replaces it with a generic message. Then it can parse the JSON.

Custom Error Text

It's easy to replace error text with something more descriptive. Here's an example of how you might inject your own wording by adding this code on top of the code in the transformer:

// Handle a benign error by looking for some phrasing in the error message
if (errorJsonString.indexOf("Benign Error") > -1) {
	
	// Phrase you want to use
	var errSub = "A benign error was thrown but can be ignored";

	// Get original full error text via JSON object names as substring
	var errorMessageJsonIndex = errorJsonString.indexOf("errorMessage");
	var errorTypeJsonIndex = errorJsonString.indexOf("errorType");
	var origSub = errorJsonString.substring(errorMessageJsonIndex + 14, errorTypeJsonIndex - 2);

	// Teplace original error text with new error message
	errorJsonString = errorJsonString.replace(origSub, "\"" + errSub + "\",");

	// Remove control characters to make sure that it will fit into the JSON
	errorJsonString = errorJsonString.replace(/[\x00-\x1F\x7F-\x9F]/g, "");
}

 

Muting Specific Errors

To mute specific error messages, just add a destination filter which checks the error message for specific text. For example: 

// 'socket closed' = channel was re-deployed, no need to alert
if ( $('errorMessage') == "Socket closed") {
	return false;
} else {
	return true;
}

 

Alerting to email and Slack

Some common Error Alerter destinations are Email senders and Slack integration. Here are some example JavaScript Writer destinations:

  • Email alerter, using the Mailgun API:

    importPackage(Packages.org.apache.http.client);
    importPackage(Packages.org.apache.http.client.methods);
    importPackage(Packages.org.apache.http.impl.client);
    importPackage(Packages.org.apache.http.message);
    importPackage(Packages.org.apache.http.client.entity);
    importPackage(Packages.org.apache.http.util);
    
    var httpclient = new DefaultHttpClient();
    var httpPost = new HttpPost("<insert your mailgun API URL here>");
    httpPost.addHeader("Authorization", "Basic '<insert your token here>");
    var nvps = java.util.Arrays.asList([new BasicNameValuePair("from", "<insert your Mailgun address here>"), 
    new BasicNameValuePair("to",  "<email1>"), 
    new BasicNameValuePair("cc",  "<email2>"), 
    new BasicNameValuePair("subject",  "Mirth Error"),
    new BasicNameValuePair("text",  "An error has occurred on your Mirth instance:\n\tChannel = " + $('channel') +"\n\tError message = " + $('errorMessage') + "\n\tDate = " + $('errorDate') + "\n\tType = " + $('errorType') + "\n\nPlease login to investigate if necessary.")]);
    
    httpPost.setEntity(new UrlEncodedFormEntity(nvps));
    var response = httpclient.execute(httpPost);
    
    try {
        var statusCode = response.getStatusLine().getStatusCode();
        var entity = response.getEntity();
        var responseString = EntityUtils.toString(entity, "UTF-8");
    
    } finally {
        response.close();
    }
     
  • Slack alerter:

    importPackage(Packages.org.apache.http.client);
    importPackage(Packages.org.apache.http.client.methods);
    importPackage(Packages.org.apache.http.impl.client);
    importPackage(Packages.org.apache.http.message);
    importPackage(Packages.org.apache.http.client.entity);
    importPackage(Packages.org.apache.http.entity);
    importPackage(Packages.org.apache.http.util);
    
    java.lang.System.setProperty("https.protocols", "TLSv1.2");
    var httpclient = new DefaultHttpClient();
    
    //
    // configurable API call info below
    //
    var apiToken = ""; // add API token here
    
    // slack channel id
    var channelID = ""; // add channel ID here
    
    // slack message
    var messageText = "A Mirth message has failed with the following details:\n\tChannel = " + $('channel') + "\n\tDate = " + $('errorDate') + "\n\tError Message = " +  $('errorMessage') + "\n\tError Type = " + $('errorType') + "\n\nPlease login to investigate if necessary.";
    var messageTextEncoded = encodeURIComponent(messageText);
    
    // slack bot name
    var botUserName = "<Insert name of the Slackbot that will post the messages>";
    
    // slack bot icon (optional)
    var iconUrl = "<insert URL for image to use as the Slackbot icon, if you want";
    var iconUrlEncoded = encodeURIComponent(iconUrl); 
    
    /////////////////////////
    
    // compile the POST call
    var postCall = "https://slack.com/api/chat.postMessage?token=" + apiToken + "&channel=" + channelID + "&text=" + messageTextEncoded + "&username=" + botUserName + "&icon_url=" + iconUrlEncoded + "&pretty=1";
    var httpPost = new HttpPost(postCall);
    
    // execute API call
    var resp = httpclient.execute(httpPost);
    try {
        var statusCode = resp.getStatusLine().getStatusCode();
        var entity = resp.getEntity();
        var responseString = EntityUtils.toString(entity, "UTF-8");
        channelMap.put("responseString", responseString);
    } 
    finally {
        resp.close();
    }
     

Troubleshooting

Once an error has occurred and your alerter has provided with the information about which channel it happened on and the error message itself, there's the matter of troubleshooting the error in order to resolve the issue. If the error message points to something obvious, such as a Read Time Out from a destination or a non-200 from an HTTP endpoint, then that's pretty straightforward. However if you're using your own JavaScript transform or are utilizing a JavaScript Reader or Writer, then looking at the stack trace provided by Mirth may not immediately let you know what the root issue is. Here are some suggestions for using the tools available to you within Mirth to help you determine why an error is occurring:

  • Check the actual error message in the dashboard by selecting the message with the ERROR status and clicking the Errors tab. Here you should see the actual error that triggered the alert, which may provide more information than what was sent by the alerter trigger.
  • Use Channel Map variables to save off values at certain points in your code to evaluate whether they are what you expect.
    • Similarly, using logger.debug("message") can be helpful to print out messages directly to the logs, but this is only useful if your logs aren't already busy enough that the entries won't get lost amongst the others.
    •  There's not really the concept of "breakpoints" in Mirth, but you could manually stop the code at a certain point if you inject a throw("error") statement in. That way you can stop the program at a certain point, then go to the failed message in the Dashboard and view the Channel Mapped variables with their current values at the time that the processing stopped.

 

Here is an example "full stack trace" error caused by a typo in the Transform:

If you are presented with a stack trace error like the one above, be aware that the line numbers are usually not very helpful as they represent the code after it's been compiled by Mirth into a separate program that contains all of the running logic. They will at least tell you which line is the problem in the code chunk that they provide, as seen above where line 2808 referencing a purposefully incorrect variable called "obxSegtypo". A similar error can occur if you're referencing a Code Template function that either isn't an active dependency for the channel or has a typo in it.

Other common errors when it comes to JavaScript writers are some of the following:

  • "Cannot set property ___ of undefined to ___" = This is common to constructing a JSON object and attempting to set the value for a subfield who's parent doesn't exist. Creating the parent object first, usually with some default value, should resolve the issue. You can also add "null checks" to see if the parent does exist before attempting to add any child objects if you like.
  • "BasicIndexOutOfBoundsException: index=___, size=___" = If you're referring to an array of values but overstep and attempt to pull from a slot that is greater than the size of the array, Mirth will throw an error like this. Other than making sure that your code is valid in how it's evaluating array values, you could also move away from using indexes and just use a for each(value in array) loop.