Gathering Extended Properties into your Management Data Warehouse database

Originally Posted 8/27/2014

In the first post on the Management Data Warehouse we looked at how to set it up and start collecting default data using Data Collection.  In this post, we will see how to create a custom collector to gather all Extended Properties at the database level on our Instances configured for Data Collection.   This shows how easy it is to create a custom collection set and collect information about our SQL Server instances.

Before we dive into the process, lets discuss a little more about what we are trying to accomplish.  If you have been following along with the Froogle DBA posts, then you know that I recommend you document your database inside the databases themselves using Extended Properties.  By doing so, you can use Central Management Servers to report on your environment.  But sometimes it is just easier and faster to query this information from a single source.  Since we are already leveraging Management Data Warehouse why don’t we use it to consolidate this information from across our environment.  We are going to gather all Extended Properties set on databases, once a day and store them in the Management Data Warehouse database.  We are going to decide we are not going to keep a lot of history on this information, so we will just keep one weeks worth of data.

With this information let’s start working on building our collector.  To start with we need to understand that the way data collection works is that a collector is going to run on each of our instances and report back to the Management Data Warehouse.  This is a distributed collection model and means we will eventually need to “install” this collector on each of our instances.  The implication of this is we will only need to worry about writing a collector to gather information for one instance and then deploy that collector to all of our instances.  We can also further simplify the process of creating the collector by using the data collection’s ability to run a script against all user database on an instance.  With this we only need to write a query to gather the database level Extended Properties for a given database.  The query we will use is shown below.


WITH NULLRECORD AS (
SELECT 0 AS class
)
SELECT
sys.extended_properties.name
, sys.extended_properties.value
FROM NULLRECORD
LEFT JOIN sys.extended_properties ON NULLRECORD.class = sys.extended_properties.class;
The reason we perform the left join is to ensure that we get at least one row per database.  The default collectors that come with SQL Server are gathering data at the instance level and not the database level.  We could use the query below, but we would not be able to answer the question in MDW of which databases are missing Extended Properties.

SELECT
sys.extended_properties.name
, sys.extended_properties.value
FROM sys.extended_properties
WHERE sys.extended_properties.class = 0;

So now that we have our query let’s dive into the process of creating a Data Collection Set.  A Data Collection Set is made up of a Collection Schedule, and a series of Collection Items.  We will have just the one collection item which will gather the Extended Properties.  If we had other things we would like to collect once a day it is best practice to add them as additional items to a single Collection Set vs. having individual Collection Sets for each.  This greatly improves the scalability of Management Data Warehouse.  So let’s address each of these one at a time.  We’ll tackle them in reverse order starting with the Collection Set.

How we build the Collection Set is determined by the Data Collector Type we wish to use.  There are four Data Collector Types provided by SQL Server; Generic T-SQL Query, Generic SQL Trace, Performance Counters, and Query
Activity.  Since the information we wish to gather is based on a T-SQL query we will be using the Generic T-SQL Query Collector Type.  Each collector type has a different set of information it needs which is provided in an XML segment to the Data Collection engine.

In this case we need to tell it the Query to run, the Table to load the data into, and what database(s) to run the query
against.  You can look at the Generic T-SQL Query Collector Type link for more specifics, but the XML segment we want to use is as follows.

<ns:TSQLQueryCollector xmlns:ns=”DataCollectorType”>
<Query>
<Value>
WITH NULLRECORD AS (
SELECT 0 AS class
)
SELECT
sys.extended_properties.name
, sys.extended_properties.value
FROM NULLRECORD
LEFT JOIN sys.extended_properties ON NULLRECORD.class = sys.extended_properties.class
</Value>
<OutputTable>ExtendedProperties</OutputTable>
</Query>
<Databases
UseUserDatabases=”true”/>
</ns:TSQLQueryCollector>

This segment says to run the query against all user databases (<Databases UseUserDatabases=”true” />) and store the results in a table in the custom_snapshots schema in the Management Data Warehouse called “ExtendedProperties” (<OutputTable>ExtendedProperties</OutputTable>).  We will need to wrap this XML in a stored procedure call later, but before we put this together we need to address the issue of scheduling.

SQL Server ships with 6 schedules for Data Collection Sets.  They are all designed to run multiple times a day.  There are collection schedules available for collecting at 5, 10, 15 and 30 minutes.  There are also collection schedules available to collect at 1 and 6 hours.  We could just use the 6 hour collection and collect multiple times a day.  But since we don’t have a business need to do so we will take advantage of the fact that we can create our own schedules. All of the schedules created for data collection all start with “CollectorSchedule_”.  To avoid confusion (and the chance that SQL Server adds a daily collection schedule in the future) we will avoid using that convention.  We’ll use “Collector_Schedule_” instead.

In order to create a new schedule we need to use the sp_add_schedule stored procedure in the msdb databases.  If you are interested in all the options available you can refer to the information in BOL.  But for our purposes to create a schedule to run daily at 01:00 we would use the following script.  We will intentionally not include the time of day in the name of the schedule, because as our environment grows (or if you have a large environment already) we may need to stagger the times that instances report into Management Data Warehouse to improve performance.

USE msdb;

IF NOT EXISTS(
SELECT schedule_uid
FROM [msdb].[dbo].[sysschedules]
WHERE name = N’Collector_Schedule_Daily’
)
EXEC sp_add_schedule @schedule_name =  ‘Collector_Schedule_Daily’,
@enabled = 1
, @freq_type = 4
, @freq_interval = 1
, @freq_subday_type = 1
, @freq_recurrence_factor = 1
, @active_start_time = 010000
, @owner_login_name = ‘sa’;

So now that we have the xml segment for the collector and the schedule, lets put it all together into a single script which will create the schedule if needed, drop the collection set if it exists, create the collection set, and then finally start up the collection set.  This is the script that we can use with Central Management Server to get all of our Instances collecting Extended Properties and reporting them to the Management Data Warehouse.

USE msdb;

DECLARE @collectorname VARCHAR(128);
SET @collectorname = ‘Database Extended Properties’;

IF NOT EXISTS(
SELECT schedule_uid
FROM [msdb].[dbo].[sysschedules]
WHERE name = N’Collector_Schedule_Daily’
)
EXEC sp_add_schedule @schedule_name =  N’Collector_Schedule_Daily’,
@enabled = 1
, @freq_type = 4
, @freq_interval = 1
, @freq_subday_type = 1
, @freq_recurrence_factor = 1
, @active_start_time = 010000
, @owner_login_name = ‘sa’;

IF EXISTS(
SELECT name
FROM [syscollector_collection_items_internal]
WHERE name = @collectorname

)
EXEC sp_syscollector_delete_collection_set @name = @collectorname;

DECLARE @collector_type UNIQUEIDENTIFIER;
SELECT @collector_type = collector_type_uid
FROM [msdb].[dbo].[syscollector_collector_types]
WHERE name = N’Generic T-SQL Query Collector Type’;

DECLARE @schedule UNIQUEIDENTIFIER;
SELECT @schedule = schedule_uid
FROM [msdb].[dbo].[sysschedules]
WHERE name = N’Collector_Schedule_Daily’;

DECLARE @collectionsetid INT;
DECLARE @collectionsetuid UNIQUEIDENTIFIER;
EXEC [msdb].[dbo].[sp_syscollector_create_collection_set]
@name=@collectorname,
@collection_mode=1, — Non-cached mode
@description=N’Records Database Extended Properties’,
@days_until_expiration=7,  — 1 week
@schedule_uid=@schedule,
@collection_set_id=@collectionsetid OUTPUT,
@collection_set_uid=@collectionsetuid OUTPUT;

DECLARE @parameters XML;
SET @parameters=‘<ns:TSQLQueryCollector xmlns:ns=”DataCollectorType”>
<Query>
<Value>
WITH NULLRECORD AS (
SELECT 0 AS class
)
SELECT
sys.extended_properties.name
, sys.extended_properties.value
FROM NULLRECORD
LEFT JOIN sys.extended_properties ON NULLRECORD.class = sys.extended_properties.class
</Value>
<OutputTable>ExtendedProperties</OutputTable>
</Query>
<Databases
UseUserDatabases=”true”/>
</ns:TSQLQueryCollector> ‘
;

DECLARE @collectionitem INT;
EXEC [msdb].[dbo].[sp_syscollector_create_collection_item]
@name=@collectorname,
@parameters=@parameters,
@collection_item_id=@collectionitem OUTPUT,
@collection_set_id=@collectionsetid,
@collector_type_uid=@collector_type;

EXEC sp_syscollector_start_collection_set @name = @collectorname;

So now we have a collector we can use to collect Database Extended Properties for all our user databases.  Once we deploy this to all of our SQL Server instances we will have a central repository of this information we can use.  There are many other collectors we could write to gather information on our environment.  We will save those for another time.

I hope you have been enjoying the Froogle DBA series.  I would love to hear any feedback you might have.

The Froogle DBA – Building CMS based on Extended Properties

Originally Posted 8/24/2014

Before I dive into the next component in the Froogle DBA’s toolbox, I wanted to give you all an idea of what’s to come.  In the original Froogle DBA plan on which my SQL Saturday presentation was built there are three component involved in automatically documenting the managed environment in Central Management Servers (CMS).  I already went over Extended Properties and showed you how to add them to a database.  I spoke a little about the advantage of using Extended Properties to document your environment.  I even talked about using CMS to query the Extended Properties to answer some common questions.

While I was procrastinating from working on my presentation it dawned on me that not everyone will want to run the other piece of the Froogle DBA trinity, Management Data Warehouse (MDW).  A lot of people will already have monitoring tools in place, and MDW is probably the hardest piece to scale and manage.  So I thought, how could people take advantage of tighter integration between CMS and Extended Properties without having to run MDW.  So I spent some time today an through together a PowerShell script that can be used to populate groups in CMS based on Extended Properties you have defined on your databases.

The idea is that you will have a group inside of CMS that lists all the Instances of SQL Server you manage.  This script will query all the Extended Properties defined at the database level on these instances.  It will then look for those that start with a specific Prefix (the default is “CMS “).  It will then build a group for each of these properties under the root of the CMS, removing the prefix,  So, for example the Extended Property “CMS Application” would create a group in CMS called “Application”.  It will then build a sub-group for each unique value defined for that Extended Property.  Then lastly it will register all the instances of SQL Server that had that value under the sub-group.  For example, if you have a database on Instance1 and another on Instance2 with the Extended Property “CMS Application” set with the value of “Application 1” you would see the results in the following picture.

2161216_orig

Now I need to fully disclose that this is a script that I have not run in production.  So, please verify this in a test environment before turning it loose on production.  And, I would be very interested on any feedback you might have on the script.  Did you find it useful?  Did you need to modify it to fit in your environment?  Have ideas for enhancements?

Well, I guess it’s time to reveal the script.

#=========================================================================
# Filename:     PopulateCMSFolders.ps1
# Created on:   20140824
# Created by:   John Greeson
#
# Description:  This script will query all the databases belonging to
#               instances in the "Instance List" group and look for 
#               Extended Properties starting with "CMS ".  It will then 
#               build supporting folders in CMS based on those variables if
#               they are missing.  It will empty all the folders it found
#               and then repopulate them with the list of subfolders and 
#               instances if found in the query of Extended Properties.
#
# Requirements: Inorder for this script to run the CMS Server needs to be
#               registered in SSMS by the user running the script on the 
#               machine it will run on.  This means if you want to automate 
#               the task you will need to determine what windows account
#               the automated task will run as, log onto the machine it 
#               will run as as that user, and register the CMS Server in 
#               SQL Server Management Studio.
#
#  The MIT License (MIT)
#
#  Copyright (c) 2014 John Greeson
#
#  Permission is hereby granted, free of charge, to any person obtaining a copy
#  of this software and associated documentation files (the "Software"), to deal
#  in the Software without restriction, including without limitation the rights
#  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
#  copies of the Software, and to permit persons to whom the Software is
#  furnished to do so, subject to the following conditions:
#
#  The above copyright notice and this permission notice shall be included in
#  all copies or substantial portions of the Software.
#
#  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
#  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
#  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
#  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
#  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
#  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
#  THE SOFTWARE. 
#
#=========================================================================


#=========================================================================
#                          VARIABLES TO CHANGE
#=========================================================================

# Central Management Server
$CMSInstance = "SQLMON";

# Master List of Instances
$MasterInstanceList = "Instance List";

# Extended Property Prefix
$Prefix = "CMS ";

#=========================================================================
#                                THE MAGIC
#=========================================================================

Import-Module SQLPS -DisableNameChecking;

# Create data table to store Extended Properties
$table = New-Object system.Data.DataTable “ExtendedProperties”
;
# Add Columns to data table
$col1 = New-Object system.Data.DataColumn Instance,([string]);
$col2 = New-Object system.Data.DataColumn Connection,([string]);
$col3 = New-Object system.Data.DataColumn Database,([string]);
$col4 = New-Object system.Data.DataColumn Name,([string]);
$col5 = New-Object system.Data.DataColumn Value,([string]);

$table.Columns.Add($col1);
$table.Columns.Add($col2);
$table.Columns.Add($col3);
$table.Columns.Add($col4);
$table.Columns.Add($col5);

# Set path to the CMS list containing the master list of Instances
$pathToListOfInstances = "SQLSERVER:\SQLRegistration\Central Management Server Group\" + $CMSInstance + "\" + $MasterInstanceList;

# Get list of instances
$instanceList = get-childitem $pathToListOfInstances -recurse;

# Gather All Extended Properties
foreach($instance in $instanceList)
{
    $instanceName = $instance.Name;
    $instanceConnection = $instance.ServerName;

    $server = new-object ("Microsoft.SqlServer.Management.Smo.Server") $instanceConnection;

    foreach($database in $server.Databases)
    {
        foreach($property in $database.ExtendedProperties)
        {
            $row = $table.NewRow();
            $row.Instance = $instanceName;
            $row.Connection = $instanceConnection;
            $row.Database = $database.Name;
            $row.Name = $property.Name;
            $row.Value = $property.Value;
            $table.Rows.Add($row);
        }
    }
}

# Set path to the CMS root
$pathToCMSRoot = "SQLSERVER:\SQLRegistration\Central Management Server Group\" + $CMSInstance;

# Conditionally populate groups
$rootGroups = $table | where Name -like $($Prefix + "*") | Group-Object -Property Name;
foreach($rootGroup in $rootGroups)
{
    $groupName = $rootGroup.Name;

    # Strip off the tag from the Extended Property
    $firstLevelGroup = $groupName.Substring($Prefix.Length);

    # Define path to the first level group
    $groupPath = $pathToCMSRoot + "\" + $firstLevelGroup;
 
    if(!(Test-Path $groupPath))
    {
        mkdir $groupPath;
    }

    # Empty out groups we manage
    Remove-Item $($groupPath + "\*") -Recurse;

    # Create Subgroups
    $subGroups = $table | where Name -eq $groupName | Group-Object -Property Value;
    foreach($subGroup in $subGroups)
    {
        # Define path to the first level group

        $groupPath = $pathToCMSRoot + "\" + $firstLevelGroup + "\" + $subGroup.Name;

        mkdir $groupPath;

        # Add instances to subgroup
        $instances = $table | where Name -eq $groupName | where Value -eq $subGroup.Name | sort-object -Property Instance -Unique;

        foreach($inst in $instances)
        {
           $con = $inst.Connection;
           New-Item -Name $inst.Instance -path $groupPath -ItemType Registration -Value ("Server=$con; integrated security=true");
        }
    }
}

The Froogle DBA – Extended Properties

Originally posted 8/23/2014

In the previous blog post we introduced Central Management Servers (CMS) and went over how to set one up and create groups and add servers.  Central Management Servers provide the Froogle DBA the ability to centrally manage their environment and share a common list of instances with other DBAs on their team.  The power of CMS really begins to grow however when you have it set up so you can apply scripts and evaluate policy to groupings of like instances.  For example, if we are asked to tell us what applications are still running on SQL Server 2005 instances it would be great to have a group in CMS defined in CMS with just SQL Server 2005 instances in them.  We could then query the group to return the list of database names on all SQL Server 2005 servers rather quickly.

But how do we figure out what application these database support.  One way would be to keep track of the mapping of databases to applications in either a spreadsheet or a table in a databases.  But there is another more scalable option, Extended Properties.  Extended Properties are key value pairs that can be stored inside of a databases and associated with the database itself or any of the objects inside the database.  You can see the extended properties in Management Studio by right clicking on an object and selecting properties.  In the property window there will be a tab for Extended Properties.  The following images show how you can see the Extended Properties for a database.

Right click on the database and select "Properties".
Right click on the database and select “Properties”.
On the left side click on the "Extended Properties" tab.
On the left side click on the “Extended Properties” tab.
On the Extended Properties tab you can see the key value pairs that have been defined for the object (or not defined as in this case).
On the Extended Properties tab you can see the key value pairs that have been defined for the object (or not defined as in this case).

To add an extended property to the database it is as easy as typing the key in the first empty row under “Name”.  Then if you want to assign a value to the property just enter that next to it under “Value”.  The following images show how to add an Extended Property.

Enter the name of the Extended Property under "Name".
Enter the name of the Extended Property under “Name”.
If you wish to assign the property a value you can do so now by entering it next to the name of the Extended Property under "Value".
If you wish to assign the property a value you can do so now by entering it next to the name of the Extended Property under “Value”.

But why would we want to use these Extended Properties?  In order for us to scale our support for SQL Server systems we have to do our best to make the environment as self documenting as possible.  So, as in the examples above, you can create an Extended Property in each of your databases that keeps track of the application(s) supported by the databases.  This means that if the database moves from instance to instance you always know what the database is supposed to be supporting.  This also allows you to answer the question proposed above definitely with a single query without having to translate databases to applications using a spreadsheet or a lookup table.

Below is an image of some Extended Properties I have found beneficial over the years.  We have discussed how you can use CMS to query Extended Properties to answer questions, but we can also improve Central Management Servers with Extended Properties.  You will notice in the image that there are two Extended Properties on this database prefixed with “CMS”.  We will show in a later blog post how we can use these Extended Properties to build dynamics groups inside of CMS.  But that is for another day.

9525218_orig