Application Monitoring

There are several things to consider when monitoring an application. This is not meant as an exhaustive list but here are few items:

• application state
• resource utilization
• proactively report application errors
• module/component status
• external dependency status (databases, services, etc.)

The monitor should be proactive and not reactive. It should provide early enough warning before something goes wrong. This includes monitoring dependencies and the application itself. There are tools that monitor logs and report when it detects various errors in the log file. Although this is better than nothing, it’s too reactive. If you’re lucky enough to have nagios in-house, this is better. However, in most cases, they are configured only for Production and, if lucky, QA. However, for other environments, it may be an overkill to use nagios.

Health Monitoring

The purpose of my health monitor tool is to provide a simple framework to quickly setup and actively check the overall health of any application.

Health monitoring tool is built on the following open source frameworks:

• quartz-scheduler – used to schedule recurring checks to verify your site, components/modules, and dependencies
• apache httpcomponents – used to retrieve resources (JSON/HTML) from HTTP/HTTPS servers
• grizzly NIO framework – used as embedded web container to host status page (see above screenshot)
• jackson – used to parse JSON resources and configuration
• freemarker – used for web/email template
• javamail – used to send failure notification

It can proactively check your application and its dependencies with predefined service checks:

• com.jstrgames.monitor.svc.impl.HttpService – make non-SSL web call
• com.jstrgames.monitor.svc.impl.HttpsService – make SSL web call
• com.jstrgames.monitor.svc.impl.SimpleJmxService – make simple remote JMX calls
• com.jstrgames.monitor.svc.impl.SocketService – make a client socket call

It can then verify these services with predefined rules:

• com.jstrgames.monitor.rule.HttpResponseCode – check if expected status code returned
• com.jstrgames.monitor.rule.HttpResponseBody – check if body contains expected values
• com.jstrgames.monitor.rule.JsonResponse – check if specified JSON key has expected value
• com.jstrgames.monitor.rule.SimpleJmxResult – check if specified JMX attribute has expected value

All can be configured on a simple JSON format configuration file.

Example: Checking HTTPS site

The use case is to verify that the text “Google Search” and “I’m Feeling Lucky” appears on the google.com page every 5 minutes:

[{"servicename": "HTTPS Example (SSL)",
  "classname": "com.jstrgames.monitor.svc.impl.HttpsService",
  "schedule": "0 0/5 * * * ?",
  "hostname": "www.google.com",
  "port": 443,
  "uri": "/",
  "rules": [{
    "classname": "com.jstrgames.monitor.rule.HttpResponseCode",
    "condition": "equals",
    "expected": 200
  },{
    "classname": "com.jstrgames.monitor.rule.HttpResponseBody",
    "condition": "contains",
    "expected": "Google Search"
  },{
    "classname": "com.jstrgames.monitor.rule.HttpResponseBody",
    "condition": "contains",
    "expected": "I'm Feeling Lucky"
  }]
}]

Here, class com.jstrgames.monitor.svc.impl.HttpsService is used check www.google.com at port 443. Once the resource has been retrieved, 3 rules are executed:

• verify http response code is 200
• verify http response body contains the phrase “Google Search”
• verify http response body contains the phrase “I’m Feeling Lucky”

If any of the above checks fails, service is considered to be in “FAIL” status. If it fails to connect to said service, it is in “ERROR” status. It will also send out a notification and user will see a similar content in their inbox, granted, not as pretty.

For more details, please go to my github repository. I will be updating its wiki page soon.

The key success factor is to ensure the user only performs minimal amount of steps. Too many steps and we end up with another chore for the user. If I could keep it down to 2 clicks, this tool will work. My first use case is simple: user start a new activity. This would be performed by right-clicking on a system tray icon then selecting the activity they’re working on (see right screenshot). Application will then be responsible for create a new audit entry with activity name and start date. When a different activity is clicked, application will be responsible for closing out prior activity and create a new audit activity. My second use case is to display a report that can be used to fill out the actual time sheet.

Retrieving the Project List
I needed a way to retrieve all projects/activities from time sheet application to be displayed as a menu item. At my office, time sheets are filled out online. Since you can’t access my intranet, I’ve built a mockup (see screenshot below). One major difference is the single sign on (SSO) authentication and authorization mechanism. Other than that, for the most part, this mockup is close to the time sheet application I have at the office.

As you can see, it’s just a simple HTML page with a TABLE element containing the data I wanted on the second column. Below is how I retrieve the HTML page.

HttpWebRequest req = (HttpWebRequest) WebRequest.Create(Settings.Default.TIMESHEET_URL);
HttpWebResponse res = (HttpWebResponse) req.GetResponse();
using (StreamReader reader = new StreamReader(res.GetResponseStream()))
{
  return reader.ReadToEnd();
}

Next, I needed a way to parse this HTML page. To achieve this, I’ve decided to use Html Agility Pack which allows me to target the portion of the TD elements using XPATH.

HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(GetTimesheetHtml());

// cache project list
this._projectList = new LinkedList();
HtmlNodeCollection items = doc.DocumentNode.SelectNodes(Settings.Default.PROJECT_NAME_XPATH);

bool isFirst = Settings.Default.SKIP_FIRST;
foreach (HtmlNode node in items)
{
  if (isFirst)
  {
    isFirst = false;
    continue;
  }

  string innerText = WebUtility.HtmlDecode(node.InnerText).Trim();
  if (string.IsNullOrEmpty(innerText)) continue;

  this._projectList.AddLast(innerText);
}

Displaying the Project List
A standard .NET Windows application would launch a .NET Form. In our case, we won’t launch a form at start of this application. Instead, we will create a custom ApplicationContext with a NotifyIcon to be displayed on system tray. This icon will be used to display a menu of projects/activities (see above screenshot). Each menu item will be associated with a mouse click event. When a project/activity menu is clicked, this application will create a new audit record that will be stored in an MS Access database.

private void InitializeContext()
{
  this._container = new Container();
  this._notifyIcon = new NotifyIcon(this._container)
  {
    ContextMenuStrip = new ContextMenuStrip(),
    Icon = Properties.Resources.clock,
    Text = "Time Tracker",
    Visible = true
  };
  this._notifyIcon.ContextMenuStrip.Opening += this.ContextMenuStrip_Opening;
}

protected void ContextMenuStrip_Opening(object Sender, CancelEventArgs e)
{
  e.Cancel = false;
  this._notifyIcon.ContextMenuStrip.Items.Clear();
  LinkedList list = WebManager.Instance.GetProjects(false);
  foreach (string item in list)
  {
    this._notifyIcon.ContextMenuStrip.Items.Add(
      item,
      item.Equals(EntryManager.Instance.ActiveActivity) ? Properties.Resources.checkmark : null,
      this.ContextMenuStrip_Clicked);
  }
  this._notifyIcon.ContextMenuStrip.Items.Add(
  EntryManager.INACTIVE_PROJECT,
  EntryManager.INACTIVE_PROJECT.Equals(EntryManager.Instance.ActiveActivity) ? 
    Properties.Resources.checkmark : null, this.ContextMenuStrip_Clicked);

  this._notifyIcon.ContextMenuStrip.Items.Add(new ToolStripSeparator());
  this._notifyIcon.ContextMenuStrip.Items.Add("Refresh Project List", null, this.ContextMenuStrip_Clicked);
  this._notifyIcon.ContextMenuStrip.Items.Add("View Report", null, this.ContextMenuStrip_Clicked);

  this._notifyIcon.ContextMenuStrip.Items.Add(new ToolStripSeparator());
  this._notifyIcon.ContextMenuStrip.Items.Add("Exit", null, this.ContextMenuStrip_Clicked);
}

Display the Report
In addition to displaying projects/activities, we want to add “View Report” to the context menu. This will display the activity report for the week and eliminate the guessing game at the end of the week.

Copy of the VS2010 Project can be found here.