Key Performance Counters for Windows Server using Performance Monitor

Every windows server ships with a lot of free tools, to help monitor and better understand what’s happening on the system. One of the most use full tools is called “Performance Monitor”, don’t let the outdated UI fool you. Performance Monitor is an extremely useful tool that allows you to log, collect, and visualize all your performance related data.

Before you go increase your VM size on your cloud provider, or switch to SSDs. Be sure to take a look at Performance Monitor, they could surprise you.

The data you collect can be broken down into four main groups:

  • CPU
    • Spawning too many threads
    • Doing too much work at one time
    • Background job that could be eating up all the CPU cycles
    • Too many Garbage collector calls
  • Memory
    • Creating too much garbage
    • Memory leaks
    • Trying to load too much data into memory at one time
    • Not streaming in data
  • Disk
    • Writing to disk all the time
    • Using too much ram at one time, so the OS starts to use Disk
    • Serving too much static content for your site (use a CDN)
  • Network
    • More traffic then your one machine has bandwidth for
    • Other applications on machine saturating connection

Any hiccup in any one of this areas of the machine could have a cascading effect on performance degradation, since each one could act as a bottle neck to the others.

On to the actual metrics you want to watch for, grouped by type:

CPU

% Processor Time:

  • What is it ?
    • Total time the processor was busy processing.
  • Why should I care ?
    • Gives a very general measure of how busy the processor can get.
  • How does it help me ?
    • If this counter is always very high, then you need to use some of the counters down below

% Privileged Time:

  • What is it ?
    • Total time the processor spent executing in kernel mode.
  • Why should I care ?
    • This measure takes into account only the kernel related operations that processor does, like memory management
  • How does it help me ?
    • If this counter is above 20% you have a driver, or hardware issue.

% User Time:

  • What is it ?
    • Total time the processor spent executing in any user application code.
  • Why should I care ?
    • Give you an idea of how much work your application code forces the processor to do.
  • How does it help me ?
    • If this percentage is too high it may be conflicting with privileged processor time. You always want to have some buffer, between “user time” and “privileged time” so the system can run smoothly.

Queue Length:

  • What is it ?
    • Number of threads waiting for a core to become available.
  • Why should I care ?
    • Give you an idea of how much work your machine is trying to do, at any given time.
  • How does it help me ?
    • Divide this number by the core count of machine. If the value is greater than 3, there is too much CPU pressure on the machine, and created a back long.

Process (*) \ Thread Count:

  • What is it ?
    • Amount of threads currently active in this process.
  • Why should I care ?
    • More threads means more CPU utilization.
  • How does it help me ?
    • Could be an indication that your application is spinning up too many threads at once.

Disk

Average Disk Queue Length

  • What is it ?
    • A simplified definition is how many disk operations (read & writes) were queued.
  • Why should I care ?
    • Gives you an idea of how saturated your disk is.
  • How does it help me ?
    • If the queue length is over 2 for a prolonged periods of time, then it could be an indication of a disk bottleneck.

% Disk Idle Time

  • What is it ?
    • How much time your disk spends doing nothing.
  • Why should I care ?
    • Gives you a picture of when the disk is free.
  • How does it help me ?
    • If the “Disk Idle Time” is low then it is doing a lot of work, which is okay. But only if the “Average Disk Queue Length” is below 2. Could give you an indication that the disk isn’t busy when it should be busy.

Avg Disk sec/Read & Avg Disk sec/Write

  • What is it ?
    • Latency of your disks.
  • Why should I care ?
    • Gives you an indication of how much lag there is to do anything with the disk.
  • How does it help me ?
    • Could indicate a hardware issue if the latency is too high.

Memory

Available Mega bytes

  • What is it ?
    • Amount of physical Memory available to processes.
  • Why should I care ?
    • Lets you know if your running out of memory.
  • How does it help me ?
    • Could indicate a memory leak if it continues to decrease

Pages / Sec

  • What is it ?
    • Rate at which pages are read directly from disk (slows down the whole system)
  • Why should I care ?
    • Occurs when there are page faults which cause system wide delays.
  • How does it help me ?
    • The higher this number gets, the more the system is running out of memory. The more it runs out of memory the more page faults will occur.

Pool Nonpaged Bytes

  • What is it ?
    • Area of memory for objects that cannot be written to disk
  • Why should I care ?
    • Eats up your available memory (ram)
  • How does it help me ?
    • If it becomes greater then 80% could lead to system halting, Nonpaged Pool Depletion Issue (Event Id 2019)

Pool Paged Bytes

  • What is it ?
    • Area of memory for objects that can be written to disk, when not being used.
  • Why should I care ?
    • The more object you have in this area the more expensive it will be to retrieve them.
  • How does it help me ?
    • The bigger this value gets the longer it takes to retrieve objects from memory.

Process (*) \ Private Bytes

  • What is it ?
    • The current size in bytes of the memory this process has allocated, that cannot be shared with other running processes.
  • Why should I care ?
    • Tells you how much memory your process takes up.
  • How does it help me ?
    • If this value gets consistently bigger over time, it could indicate the your application has a memory leak.

Network

Output Queue Length

  • What is it ?
    • The length of the output packet queue.
  • Why should I care ?
    • The higher this number the large the backlog of queue packets. The longer it takes to send out data
  • How does it help me ?
    • Helps indicate a network bottle neck. If greater then one, the systems network is nearing capacity.

Resources used:

How I went about choosing a Deep Learning Framework

The following is a excerpt that was made, as part of my final capstone project.

Introduction

The hardware and software section will be primarily exploring the two key parts in the development of neural networks. Currently the two competing software libraries for the development of neural networks are PyTorch and Tensor Flow. And the two competing hardware platforms to train models is between AMD and Nvidia [6]. In this section I will explore the benefits and disadvantages of each.

Deep Learning Software & Hardware Selection

When looking into developing our model I identified the 2 key choices, software selection and hardware selection. I identified framework selection as a key choice since, it would act as the key building block in constructing the model, and effect how fast I could train them. Where as hardware selection was important since it would be the primary limiting factor in how fast I could train the model, and how complex I could make the model.

Software Selection

Due to the exponential expansion of machine learning (ML) research and computing power seen over the last decade. There has also been an explosion of new types of software infrastructure to harness it. This software has come from both academic and commercial sources. The need for this infrastructure arises from the fact that there needs to be a bridge betIen theory and application. When I looked at what Ire the most popular frameworks, I found it was a mix of strictly academic and commercial driven software. The four main frameworks Ire Caffe, Theano, Caffe2 + PyTorch, and Tensor Flow (TF).

When I went about choosing a framework, I considered three different factors, community, language, and performance. Community was one the biggest factors, since I had no real production experience in doing any sort of large scale ML modeling and deployment. The only framework that fulfilled this need was Google’s Tensor Flow. It had been released in 2015 and had been made available to the open source community. Leading to many academic researchers to contribute and influence its development. Which has resulted in many other companies using it in their production deep learning pipelines. The combination of both software developers and scientists using it has led to a lot of community driven development. This has lead to making it easier to use and deploy. A side effect of this large amount of adoption is the generation of detailed documentation. Written by the community, large amount of personal, and company blogs, detailing how they used TF to accomplish their goals. The only real competitor at the time of writing it this is Facebook’s Caffe 2 + PyTorch Libraries which was just open sourced early this year.

The other factor was the language interface it would use. I wanted an easy to use interface, with which to build out the model. When I looked at what was available, I found that all of the popular frameworks were written in C++ and CUDA, but had a easy to use Python based interface. The only framework out of the four mentioned above, that only had C++ based interface was Caffe.

The most important part of framework selection was the performance aspect. Most if not all ML research and production use cases happen on Nvidia GPU hardware. This is due to Nvidia’s development of their CUDA programming framework for use with their GPUs. It makes parallel programming for their GPUs incredibly easy. This parallelization is what lets the complex matrix operations be computed with incredible speed. There were only two frameworks out of the four I mentioned, that used the latest version of CUDA in its code base. Which were TF and Caffe 2 + PyTorch, however Caffe 2 + PyTorch was not as robust as Tensor Flow in supporting the different versions of CUDA.

In the end I choose to go with TF since it had a better community and CUDA support. I did not choose to go with its nearest competitor, since it was not as well documented, and its community was just starting to grow. Whereas TF has been thoroughly documented and has had large deployments outside of Google (such as at places like LinkedIn, Intel, IBM, and UBER). Another major selling point for TF is the fact that, it is free, continually getting new releases, and has become an industry standard tool.

Deep Learning Software Frame Works
Name Caffe Theano Caffe 2 + PyTorch Tensor Flow
Computational Graph Representation No Yes Yes Yes
Release Date 2013 2009 2017 + 2016 2015
Implementation language C++ Python & C C++ C++, JS, Swift
Wrapper languages N/A Python Python, C++ C, C++, Java, GO, Rust, Haskell, C#, Python
Mobile Enabled NO NO YES YES
Corporate Backing UC Berkeley University of Montreal Facebook Google
CUDA enabled NO YES YES YES
Multi GPU Support NO NO YES YES
Exportable Model YES NO YES & NO YES
Library of pretrained models YES NO YES YES
Unique Features Don’t need to code to define Network First to use CUDA and Computational Graph in Memory Uses the original developers of Caffe and Theano frameworks

 

VISDOM – Error function Visualization Tool

 

PoIrs Facebook ML

Tensor Board – Network Visualization and Optimization Tool

 

Developed by Google Deep Brain

 

 

PoIrs Google ML

Under Active Development No No Yes Yes
 

 

NOTE

The reason as to why PyTorch and Caffe 2 are always mentioned together is because they are meant to be used together. PyTorch is much more focused on research and flexibility. Where as Caffe 2 is more focused on production deployment and inference speed. Facebook’s researchers use PyTorch to prototype models, then translate the model into Caffe 2, using their model transfer tool known as ONIX.

Table 1 A summary of all information of note that I collected during my research

My Girl Friend

Today is Valentines day and I want to talk about a person that has always been with me, my computer.

Just by reading the line above you maybe thinking I am a Nerd. This is more a fact than a truth. Computers have been around in my life since I was 5 years old, and I love them. Ask me if I would rather have a super computer vs a Ferrari, and I would always pick the super computer.

It wasn’t till I started making my own money did I start building my computer. Contrary to popular belief building your own computer is extremely easy these days. You can think of it like high tech Lego that is super expensive. Quality components these days can handle a fall or something being dropped on them, very easily.

Building computers for guys like me, is basically how it is for gear heads and their cars. I originally started with an okay computer (i5 & GTX 1070) and moved on to do more … a lot more. I finished my latest upgrade yesterday I got a new case, mother board, and water cooler that will allow me to over clock. Its some thing new that I want to get into 🙂

As it stands here are my specs:

CPU: i7 – 7700k – Corsair – H115i water cooler

GPU: GTX 1080 Ti – Asetek – Custom water cooler

Mother Board: ATX – Z270 GAMING PRO CARBON

Hard Drives: – 1 X 256GB SSD / 1 X 1TB HHD / 1 X 3TB HHD

This is the best of the best as of late, this is what I wanted since I was 12 🙂