The Froogle DBA – Building CMS based on Extended Properties

Originally Posted 8/24/2014

Before I dive into the next component in the Froogle DBA’s toolbox, I wanted to give you all an idea of what’s to come.  In the original Froogle DBA plan on which my SQL Saturday presentation was built there are three component involved in automatically documenting the managed environment in Central Management Servers (CMS).  I already went over Extended Properties and showed you how to add them to a database.  I spoke a little about the advantage of using Extended Properties to document your environment.  I even talked about using CMS to query the Extended Properties to answer some common questions.

While I was procrastinating from working on my presentation it dawned on me that not everyone will want to run the other piece of the Froogle DBA trinity, Management Data Warehouse (MDW).  A lot of people will already have monitoring tools in place, and MDW is probably the hardest piece to scale and manage.  So I thought, how could people take advantage of tighter integration between CMS and Extended Properties without having to run MDW.  So I spent some time today an through together a PowerShell script that can be used to populate groups in CMS based on Extended Properties you have defined on your databases.

The idea is that you will have a group inside of CMS that lists all the Instances of SQL Server you manage.  This script will query all the Extended Properties defined at the database level on these instances.  It will then look for those that start with a specific Prefix (the default is “CMS “).  It will then build a group for each of these properties under the root of the CMS, removing the prefix,  So, for example the Extended Property “CMS Application” would create a group in CMS called “Application”.  It will then build a sub-group for each unique value defined for that Extended Property.  Then lastly it will register all the instances of SQL Server that had that value under the sub-group.  For example, if you have a database on Instance1 and another on Instance2 with the Extended Property “CMS Application” set with the value of “Application 1” you would see the results in the following picture.

2161216_orig

Now I need to fully disclose that this is a script that I have not run in production.  So, please verify this in a test environment before turning it loose on production.  And, I would be very interested on any feedback you might have on the script.  Did you find it useful?  Did you need to modify it to fit in your environment?  Have ideas for enhancements?

Well, I guess it’s time to reveal the script.

#=========================================================================
# Filename:     PopulateCMSFolders.ps1
# Created on:   20140824
# Created by:   John Greeson
#
# Description:  This script will query all the databases belonging to
#               instances in the "Instance List" group and look for 
#               Extended Properties starting with "CMS ".  It will then 
#               build supporting folders in CMS based on those variables if
#               they are missing.  It will empty all the folders it found
#               and then repopulate them with the list of subfolders and 
#               instances if found in the query of Extended Properties.
#
# Requirements: Inorder for this script to run the CMS Server needs to be
#               registered in SSMS by the user running the script on the 
#               machine it will run on.  This means if you want to automate 
#               the task you will need to determine what windows account
#               the automated task will run as, log onto the machine it 
#               will run as as that user, and register the CMS Server in 
#               SQL Server Management Studio.
#
#  The MIT License (MIT)
#
#  Copyright (c) 2014 John Greeson
#
#  Permission is hereby granted, free of charge, to any person obtaining a copy
#  of this software and associated documentation files (the "Software"), to deal
#  in the Software without restriction, including without limitation the rights
#  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
#  copies of the Software, and to permit persons to whom the Software is
#  furnished to do so, subject to the following conditions:
#
#  The above copyright notice and this permission notice shall be included in
#  all copies or substantial portions of the Software.
#
#  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
#  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
#  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
#  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
#  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
#  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
#  THE SOFTWARE. 
#
#=========================================================================


#=========================================================================
#                          VARIABLES TO CHANGE
#=========================================================================

# Central Management Server
$CMSInstance = "SQLMON";

# Master List of Instances
$MasterInstanceList = "Instance List";

# Extended Property Prefix
$Prefix = "CMS ";

#=========================================================================
#                                THE MAGIC
#=========================================================================

Import-Module SQLPS -DisableNameChecking;

# Create data table to store Extended Properties
$table = New-Object system.Data.DataTable “ExtendedProperties”
;
# Add Columns to data table
$col1 = New-Object system.Data.DataColumn Instance,([string]);
$col2 = New-Object system.Data.DataColumn Connection,([string]);
$col3 = New-Object system.Data.DataColumn Database,([string]);
$col4 = New-Object system.Data.DataColumn Name,([string]);
$col5 = New-Object system.Data.DataColumn Value,([string]);

$table.Columns.Add($col1);
$table.Columns.Add($col2);
$table.Columns.Add($col3);
$table.Columns.Add($col4);
$table.Columns.Add($col5);

# Set path to the CMS list containing the master list of Instances
$pathToListOfInstances = "SQLSERVER:\SQLRegistration\Central Management Server Group\" + $CMSInstance + "\" + $MasterInstanceList;

# Get list of instances
$instanceList = get-childitem $pathToListOfInstances -recurse;

# Gather All Extended Properties
foreach($instance in $instanceList)
{
    $instanceName = $instance.Name;
    $instanceConnection = $instance.ServerName;

    $server = new-object ("Microsoft.SqlServer.Management.Smo.Server") $instanceConnection;

    foreach($database in $server.Databases)
    {
        foreach($property in $database.ExtendedProperties)
        {
            $row = $table.NewRow();
            $row.Instance = $instanceName;
            $row.Connection = $instanceConnection;
            $row.Database = $database.Name;
            $row.Name = $property.Name;
            $row.Value = $property.Value;
            $table.Rows.Add($row);
        }
    }
}

# Set path to the CMS root
$pathToCMSRoot = "SQLSERVER:\SQLRegistration\Central Management Server Group\" + $CMSInstance;

# Conditionally populate groups
$rootGroups = $table | where Name -like $($Prefix + "*") | Group-Object -Property Name;
foreach($rootGroup in $rootGroups)
{
    $groupName = $rootGroup.Name;

    # Strip off the tag from the Extended Property
    $firstLevelGroup = $groupName.Substring($Prefix.Length);

    # Define path to the first level group
    $groupPath = $pathToCMSRoot + "\" + $firstLevelGroup;
 
    if(!(Test-Path $groupPath))
    {
        mkdir $groupPath;
    }

    # Empty out groups we manage
    Remove-Item $($groupPath + "\*") -Recurse;

    # Create Subgroups
    $subGroups = $table | where Name -eq $groupName | Group-Object -Property Value;
    foreach($subGroup in $subGroups)
    {
        # Define path to the first level group

        $groupPath = $pathToCMSRoot + "\" + $firstLevelGroup + "\" + $subGroup.Name;

        mkdir $groupPath;

        # Add instances to subgroup
        $instances = $table | where Name -eq $groupName | where Value -eq $subGroup.Name | sort-object -Property Instance -Unique;

        foreach($inst in $instances)
        {
           $con = $inst.Connection;
           New-Item -Name $inst.Instance -path $groupPath -ItemType Registration -Value ("Server=$con; integrated security=true");
        }
    }
}

The Froogle DBA – Central Management Servers

Originally published 8/22/2014

Before I start talking about Central Management Servers let me take a moment to better explain what I mean about managing SQL Server at scale.  In the first blog post in this series I failed to mention what I meant by scale.  Scale means a lot of different things.  In this case, scale is meant to the overall size of an environment.  Most often this means having a very large number of databases and/or instances managed per DBA.  

The core of the Froogle DBA’s ability to manage Microsoft SQL Server at scale is Central Management Servers (CMS).  A Central Management Server is an instance of SQL Server that maintains server groups that contain the connection information for one or more instances of SQL Server.  CMS was introduced in SQL Server 2008, so the instance of SQL Server serving as the Central Management Server has to be running SQL Server 2008 or newer.  But you can use CMS to manage instance at least as far back as SQL Server 2000.  Instructions on how to register a Central Management Server can be found here, or you can consult the images below.

Right click on “Central Management Servers” on the Registered Servers tab and select “Register Central Management Server…”
Enter the connection information for the SQL Server 2008 or newer instance you want to use as your Central Management Server and click "Save".
Enter the connection information for the SQL Server 2008 or newer instance you want to use as your Central Management Server and click “Save”.
Now your Central Management Server will show up in the list under Central Management Servers. Note: All DBAs on your team will need to do this to see the servers and groups you register.
Now your Central Management Server will show up in the list under Central Management Servers. Note: All DBAs on your team will need to do this to see the servers and groups you register.

The step above will need to be done on all client machines you wish to share Central Management with.  In the case of the Froogle DBA, this means they will want all the members of their team to register the same CMS.

Once you have your Central Management Server registered, the fun really begins.  Now you can create a central version of the truth as to what SQL Instances your team manages.  You would be surprised the number of places I have been where the simple question of “Which instances are we responsible for?” can’t be answered.  I recommend just having a group under your CMS named “Instance List” which you simply place all your managed instances in.  First, this allows you to quickly see all the instances you are responsible for, and secondly we will use this  list later on when we start integrating other SQL Server components with CMS.  In order to create a group in CMS you can follow the instructionshere, or consult the images below.

Right click on the CMS Server name (or a group to create a subgroup) and select "New Server Group...".
Right click on the CMS Server name (or a group to create a subgroup) and select “New Server Group…”.
Fill in the group name and description and select "OK".
Fill in the group name and description and select “OK”.

Now you can register the SQL instances you manage under the “Instance List” group you just created.  If you are like me the first instance you are going to want to create is the Instance serving as your Central Management Server.  This will fail, as the next series of images shows, with the following error .

“You cannot add a shared registered server with the same name as the Configuration Server.
(Microsoft SQL Server, Error: 35012)”

Right click on the group you want to add an instance to and select "New Server Registration...".
Right click on the group you want to add an instance to and select “New Server Registration…”.
Enter your CMS server information and select "Save".
Enter your CMS server information and select “Save”.
The error message you get.
The error message you get.

In order to register the CMS server in CMS you are going to need to register it by IP address.  You will find a lot of advise on the internet to use the loopback interface (127.0.0.1) to register the CMS server.  This is great if you want everyone to have to log into the CMS server to use Management Studio.  If you take this advice and connect to a CMS server and run a query on the CMS server everything will look like you are querying the CMS server, but you will really be querying the default instance of SQL Server on the host Management Studio is running on.  Instead, you are better off ensuring that your CMS server has a static ip address (or pin the ip via DHCP).  If you just can’t get a static ip address for your CMS server then you can register it this same way, but in place of the ip address you are going to need to know the port number SQL Server is running on and then enter ServerName,PortNumber (i.e., SQLMON,1433) for the Server Name in the New Server Registration window.  An example setting it up using the ip address in the New Server Registration window is shown in the next images.

Enter ipconfig from the command line in a command window on your CMS server. If you can't log into your CMS server you can ping the server by running "ping CMSServerName" replacing CMSServerName with the name of your CMS server. Record the ip address assigned to your CMS server.
Enter ipconfig from the command line in a command window on your CMS server. If you can’t log into your CMS server you can ping the server by running “ping CMSServerName” replacing CMSServerName with the name of your CMS server. Record the ip address assigned to your CMS server.
Enter the ip address in the "Server Name" box towards the top of the New Server Registration window. Then to display a more friendly name change the "Registered Server Name" box towards the bottom of the window to be the name of your CMS server. Click save.
Enter the ip address in the “Server Name” box towards the top of the New Server Registration window. Then to display a more friendly name change the “Registered Server Name” box towards the bottom of the window to be the name of your CMS server. Click save.
You now have your CMS server registered inside of CMS and it will work regardless of what machine you connect to CMS from.
You now have your CMS server registered inside of CMS and it will work regardless of what machine you connect to CMS from.

Registering the rest of the instances in your environment should be pretty straight forward.  I have run into situations where I needed to register instances where DNS would not resolve the SQL Server.  If you run into this situation you can use the same technique as above (assuming that you have a static ip address for the instance to add).  If the instance is running on a non default port, you may need to add the port number (i.e.,  192.168.1.5,1234 with the ip addresss then a comma then the port number).

Once you have servers registered in CMS, you can now run scripts in Management Studio on either individual servers or the group as a whole.  You can also evaluate policy at the group level and perform a lot more tricks.  For now this is enough information for the Froogle DBA.  If you don’t plan to follow the Froogle DBA plan then you may want to add other groups and register servers manually.  Some example of the groups that I have found useful are grouping servers on the following:

  • Version of SQL Server
  • Edition of SQL Server
  • Applications
  • Environment (Development, Test, QA, Integration, Production, etc..)
  • Features (SQL Server features such as Mirroring, Availability Groups, Clustering, etc..)

But ultimately the groups you create and use are the ones that are meaningful and useful to you and your team.  There is a lot more about Central Management Servers to go over with for Froogle DBAs, but this is enough to get you started.  We will revisit CMS as we go over other tools in the Froogle DBA’s toolbox.