There will be a proper series of posts about Kerberos coming along (it’s a big subject with a lot of words to write – I teach a two day course if you are really interested), but in the interim here’s a pictorial guide to the delegation you will need to set up.

The key points are:

• Web site account delegates to itself, to any other web site accounts and to SSRS account over HTTP
• SSRS account delegates to all report data sources’ accounts over MSOLAPSvc.3 and MSSQLSvc
• Claims to Windows Token service account delegates to all PerformancePoint, Excel, Visio and InfoPath services data sources’ accounts over MSOLAPSvc.3 and MSSQLSvc
• PerformancePoint, Excel, Visio and InfoPath services all delegate to all their required data sources’ accounts over MSOLAPSvc.3 and MSSQLSvc

setspn -T (forest) -F -Q (service)/(server)

Both the service and the SPN can use wildcards so the following command will give a list of all the SQL Server SPNs in the entire forest “my forest”.

setspn -T myforest -F -Q MSSQLSvc/*

The user is asked to look at a list of RDL files and guess which one is the top level report, and which are subreports which are only intended to be consumed inside a ‘container’ report or are used as the target of hyperlink actions inside another report.  But it’s easy to hide reports which are not intended for users to run individually by making use of SharePoint’s View features.  This example has screenshots from SharePoint 2010 but you can also do the same thing in MOSS 2007.

The goal is not to physically move the subreports, but to make them hidden from the list a user is presented with by categorising them in a managed way.  So, from the Library ribbon, click “Create Column”.

In the next dialog, fill in the following values:

• Column Name: "Report Type"
• Type of information: "Choice"
• Require that this column contains information: "Yes"
• Enforce unique values: "No"
• Choices: "Report", "Subreport"
• Display choices using: "Drop-Down Menu" or "Radio Buttons"
• Allow ‘Fill-in’ choices: "No"
• Default value: "Choice", "Report"
• Add to default view: Checked

You should end up with a page something like this:

After you create the column, you will be returned to the library. As long as you checked the “Add to default view” box when you created the column, you should now have an additional column headed “Report Type”, with no values for any of the reports.  So, edit the properties of each report and set the value for the new “Report Type” property to either “Report” or “Subreport”.

Once you have completed categorising your reports and subreports, create a new view by clicking the “Create View” button in the library ribbon.

Select “Standard view”, then create the view with the following settings:

• View name: "Reports only"
• Make this the default view: Checked
• View audience: "Create a public view"

You can add in any additional columns or sorts as you wish, then in the “Filter” section, enter the values you need:

After creating the view, you will be taken back to the report library but now you are only seeing the top level reports.

There is a useful bit of regular expression code available from Microsoft at http://msdn.microsoft.com/en-us/library/01escwtf.aspx so all I had to do was work it into a script component.  Here’s how.

Add a script component to the data flow and set the properties you’ll need.  The input column will be the email address field you want to check, then add an output column with the DataType Boolean.

In the script itself, you need to add the regular expressions library to the script’s using statements:

using System;
using System.Text.RegularExpressions;

Immediately after this, add a new class to actually parse the email address and use the regex to determine its validity:

public class RegexUtilities
{
    public static bool IsValidEmail(string strIn)
    {
        // Return true if strIn is in valid e-mail format.
        return Regex.IsMatch(strIn,
               @"^(?("")("".+?""@)|(([0-9a-zA-Z]((\.(?!\.))|[-!#\$%&'\*\+/=\?\^`\{\}\|~\w])*)(?<=[0-9a-zA-Z])@))" +
               @"(?(\[)(\[(\d{1,3}\.){3}\d{1,3}\])|(([0-9a-zA-Z][-\w]*[0-9a-zA-Z]\.)+[a-zA-Z]{2,6}))$");
    }
}

Finally, inside the ProcessInputRow add some code to execute the validation class on each row:

    public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
        string emailAddress;
        emailAddress = Row.EMAILADDRESS;
                
        bool valid = false;

        if (RegexUtilities.IsValidEmail(emailAddress))
            valid = true;
        else
            valid = false;

        Row.ValidEmailAddress = valid;

    }

The field ValidEmailAddress will now be on the dataflow.

The sales fact table has an associated dimension which describes the sale record.  There is a surrogate key relationship between the fact and dimension tables but there’s also the business key (SaleNumber) from the source system.

Why is this bad?

• There will be a one-to-one relationship between the fact table and the dimension, which means the dimension is potentially going to be huge.  Joining between fact and this dimension will be slow at query time.
• The dimension is very poorly built for Analysis Services.  If Status only has three possible values but those three are being repeated on every line instead of normalised into their own dimension, SSAS build times will be bigger than they should be.
• If you have facts that get updated in the source, you are going to have to manage this in two places in the data warehouse.  A type 2 SCD with millions of rows in it like this one is going to be painfully slow to populate on your ETL runs.
• Some of the attributes should have their own dimensions – status, payment method, order method, carrier – or maybe a junk dimension.  This will compress the dimension to the unique values of each of these fields rather than making it as big as the fact table itself.
• The CarriageWeight field is meaningless as an attribute because no-one would ever want to analyse orders by their carriage weight.  So it should be included on the fact table, though in an SSAS cube its AggregationType would be set to None.
• Delivery address fields could go in several places:
  • Into a delivery address dimension
  • Onto the fact table if they aren’t used for analysis purposes
  • Into the customer dimension if each customer only has one delivery address

So – dimensions which describe facts in a one-to-one way should be avoided.

So it’s more important than ever to make sure you set a defined time period for which your ETL is loading records, with a start and end time.  This period needs to be respected for every extract in a run, so for example you need to make sure you extract order headers and order lines which cover exactly the same time period.

Here’s the simplest but most reliable way I have been able to find to do this.

The first step in the SSIS defines a new load and sets the date/time range which the load covers.  Subsequent steps use these time periods when selecting source records.

(In a “real” solution I’d expect the first step to be done in your ‘master’ package and the results passed to subsequent child packages.)

The Execute SQL step does two things: creates a new load and sets the time window it is working with, then places the information in package variables.  The variables required are a load ID, a start date/time and a finish date/time:

In the meta-database which logs and controls the ETL runs, there is a simple table which mirrors these three values.

CREATE TABLE [dbo].[LoadWindowControl](
    [LoadID] [int] identity NOT NULL,
    [LoadFrom] [datetime] NOT NULL,
    [LoadTo] [datetime] NOT NULL
) ON [PRIMARY];
GO

The SQL task calls a stored procedure:

CREATE PROCEDURE [dbo].[uspSetLoadWindow]
    @LoadID int output,
    @LoadFrom datetime output,
    @LoadTo datetime output
AS
BEGIN

    SET NOCOUNT ON;

    declare @StartTime datetime;
    select @StartTime = dateadd(s, 1, coalesce(max(LoadTo), 0)) from LoadWindowControl;
    
    declare @EndTime datetime;
    set @EndTime = convert(datetime, convert(varchar(30), CURRENT_TIMESTAMP, 120));
    
    insert into LoadWindowControl (LoadFrom, LoadTo)
    values (@StartTime, @EndTime)
    
    select @LoadID = @@IDENTITY, @LoadFrom = @StartTime, @LoadTo = @EndTime
END;
GO

This is very simple – just adding a new record into the list of loads with an end date of “now” and a start date of the last end plus one second.  All three of these values are passed back as output parameters so the SSIS package can put them into variables.  Calling the proc from SSIS is easy:

And the parameter mapping inserts the right value to the right variable:

The first time this package runs, the table gets a record like this:

To check these values are being passed into the SSIS variables, here’s a quick look at the Locals window for anyone confused by these dates, remember I’m in the UK so the format is dd/mm/yyyy):

With these parameter values now populated them you can use them as part of the SQL command in each source.  For example:

For each package you can pass these variables through as configurations, or you could just pass through the load ID and retrieve the related dates using an Execute SQL task.

Can Analysis Services even deal with this type of cube size?  What kind of hardware do you need in order to run it?  How do you tune a cube in the terabyte range? Just how quick can your query response times be?

In this case, the cube contains just under 14 billion (13, 680, 000, 000 to be exact) fact table records.  These can be drilled down to a time level of minute.  The cube users need to be able to drill from year down to minute, and across dimensions of client, fact location and geography.  And we need to be getting ‘instant’ response using PerformancePoint or Excel for free ad hoc analysis.  This all adds up to needing a very highly tuned cube.  It took about three days to reach a point I was beginning to be happy with the performance of the cube.  The only way to approach it was to think of a very large cube as simply a regular cube but with more data in it and follow a standard tuning procedure, then add more if required.

So, the approach is:

1. Sort the dimensions out.  When you have this many fact records, the cube needs dimensions set up as efficiently as possible.
2. Remove many-to-many structures if possible; optimise them if you really can’t live without them. (This was probably the biggest single optimisation.)
3. Sort the partitions out. Partition on an attribute (or set) that will often be used in queries at the level you are partitioning. Use the slice property. Don’t use any attribute that’s used in a many-to-many relationship as a partition slice.
4. Aggregate efficiently and appropriately.

Dimension Optimisation

I think this is probably the foundation of cube tuning.  All the aggregations in the world won’t really solve your tuning problems if they are having to work with badly built dimensions.

Here are my rules:

• Don’t include any attributes that won’t be needed just because they are in the data warehouse dimension.
  For example, things like addresses are almost never needed.  By including these you are simply increasing the number of cells your cube has to contain – therefore the query space and the amount of work the engine has to do when loading up the cube from disk.
• If there are columns you really must include but will be rarely queried and don’t form part of an attribute hierarchy, turn off Attribute Hierarchy Enabled and make them a property of the key.  They will still be available in Excel or whatever for any users that desperately need them.
• Keep every attribute’s key small.  You may need an attribute key which has more than one column – if so use the minimum columns you can.  For example, just because your date hierarchy goes Year –> Quarter –> Month, the key for the month attribute doesn’t need to do the same.  Just Year and Month would do.
  For text attributes with lots of different values, seriously consider creating a numeric key column for them in the source dimension.
• Design hierarchies properly and set up the attribute relationships correctly. Always put attributes into hierarchies if you possibly can – it might look to the end user that they get the right result by using attributes rather than hierarchies but it is nowhere near as efficient.
• If an attribute is used in a hierarchy, hide it as an attribute.  Just to enforce the hierarchies you’ve designed on your users.  Even in a date dimension don’t let the year/quarter/month/week/etc attributes remain visible.  Put them into logical hierarchies and then hide the attribute.  Create ‘Month of Year’ type attributes and make those visible instead – and make sure you have both a numeric key and a text name for these types of attribute.
• Make relationships ‘rigid’ if you can (dates will be a good candidate for example), but don’t compromise your ability to perform incremental refreshes just to do this.

Here’s an extreme example.  When the cube was first put together, there was a dimension that looked like this:

Almost everything there is never going to be used in any type of query.  In fact only one attribute is needed so the dimension should look like this:

Not only has the majority of attributes been deleted, but the remaining one has a rigid relationship.

Many to Many Dimensions

Effectively, a many-to-many is to cube performance what Kryptonite is to Superman. The additional amount of work required each time the many-to-many dimension is queried is considerable, and the amount of aggregations required to improve their performance will explode the size of your cube.  If you really must have a many-to-many, keep the intermediate measure group as small as possible by removing repeating rows (i.e. every row in it should be distinct).

One pitfall – if you use a many-to-many dimension as part of your partitioning strategy, you cannot set the slice property (it always produces an error on processing) of the partition.  Even worse, the engine won’t be able to determine which partitions contain the data it needs for query results (i.e. automatically slice) so every partition in the cube will get touched on every query.  These are probably facets of the same problem.

There’s a white paper on tuning many-to-many dimensions at http://blogs.msdn.com/b/sqlcat/archive/2007/12/21/new-best-practices-articles-published-analysis-services-many-to-many-dimensions-query-performance-optimization-techniques.aspx.  But my advice would be – avoid them at all costs on a cube this size because you are going to have so much data to query that their performance penalty is going to be be massive.

Partition Design

Every tip I’ve read on the Internet tells you that no partition should have more than twenty million records in it.  But as the screenshot below shows, my cube has eight or nine times that number per partition.

Why not make the partitions smaller? Several reasons:

• The measure group is already partitioned by month (a very common query unit), it’s just that each month has a lot of records. Partitioning by the next smallest time unit (day) would mean we’d need approx 30 times as many partitions which would be:
• Difficult to manage
• A lot of individual files for the cube to need to read data from
• A problem for aggregation since the largest meaningful aggregate would be one day (remember aggregations are by partition)
• Partitioning by another attribute isn’t practical because unlike dates, most other attributes would probably have a variable number of members (imagine partitioning by client for example) so you would have to dynamically add and remove partitions.
• The cube partition scheme exactly maps to the data warehouse’s fact table partitioning scheme.  So loading a partition has a nice performance boost – the RDMBS just has to supply the entire contents of one database partition for each partition we process in the cube.  We actually have no indexes at all on the fact tables, which certainly speeds up ETL inserts.
• Ultimately, the cube performs absolutely perfectly even with these very large partitions.

With the partitions designed and the queries added, make sure you set the Slice property on every partition.  Ignore the common myth that you do not need to set it on MOLAP partitions – it takes a second to set it, it provides a nice safeguard against errors in your partitioning statements, and it guarantees the engine will hit just the required partitions at query time.

Aggregations

On a cube this large, you really will need very good aggregations – aggregations that are specifically designed to support exactly the queries being executed.  This is because there is so much granularity level data that any query which can’t use an aggregation is going to be incredibly slow.  Experience with this terabyte cube showed the best approach was:

• Find one frequently used attribute and set its aggregation usage to Full. The best candidate here is Month because:
• That matches the partition strategy
• There are a relatively small number of members (only 60 if there are five years’ of data) so the aggregation designer should respect your setting
• Month is a good attribute for almost all queries.  You usually query based on a date range (this month, all months in this year, etc.)
• Every other attribute in every dimension is just left as Default.
• Use BIDS Helper to update all the counts in the cube. In a cube this size, be prepared to leave it setting counts overnight!
• Although the aggregation designer still won’t produce particularly useful aggregations, I ran it anyway on all the measure groups.  Deciding how many aggregations you want it to produce is a tightrope walk – too few and it’s pointless, too many and you can start seeing gigabytes of aggregations being designed.  The rules I followed eventually were:
• Use the ‘Until I click stop’ setting
• Watch the size and number of aggregations being generated.  Aim for under 150 aggregations, or less than three or four gigabytes.

Usage Based Optimisations

This is the real place you are going to win with aggregations.  Using the designer, or designing your own (unless you really understand what you are doing) is just playing at it.

Once you’ve got your cube built, get query logging enabled straight away.  This is easy:

• Create a database to hold the logs
• Enable logging in the server’s properties and set the name of the table you will hold the logs in.

You want to use the logs in a structured way, so make sure you are doing this in development – don’t wait until the cube is in production and you start getting complaints from users about speed.  Users will be running all sorts of random queries which will be getting logged and affecting the optimisation recommendations.  That’s fine but at this stage you want to remain in control of everything that’s affecting your cube’s design.

With an empty query log (hint: it’s a database table so you can always TRUNCATE it to clear it between tuning runs, but SSAS will empty it for you each time you deploy a change to the cube), navigate to your dashboard and open every page in it. Run each of your reports. I.e. do one of every thing that will be a regular activity.  This will enable the optimisation wizard to put together a set of aggregations that support everything you have built.

At this point, I ran the usage based optimisation wizard against each measure group, accepted its suggestions and merged them into the existing aggregation designs.

Over time, of course, this cube would have further aggregations added based on real-world usage by users.

Results

How quick is the cube after all this work?  The following video should give an idea:

Promise: no smoke and mirrors, frames cut out, or anything like that.

And the hardware?  This is on a DL680 (4 x quad) with 64gb or memory.  Nothing special.

This post covers the following scenarios:

• Clustered SQL Server
• Clustered Analysis Services
• Scale out and load balanced Reporting Services
• Reporting Services integrated with SharePoint

I’ve been building just such an architecture in a lab environment using SQL Server 2008R2 and SharePoint 2010 but the steps below should be applicable to previous versions as well.

The steps involved are:

• Install SQL and SSAS clusters and set up DNS
• Install SSRS for scale out deployment and set up network load balancer DNS
• Setting SPNs
• Trusting computers and service accounts for delegation
• Configuring Reporting Services
• Configuring SharePoint
• Configuring Reporting Services/SharePoint integration

SQL and SSAS Clusters

The good news is you really don’t need to do anything that you wouldn’t normally do when setting up a cluster for either of these two services.  Create the cluster, install the services on the nodes, set up MSDTC, then create a DNS name that points at the cluster address.

Make sure each node in the cluster uses the same service account. For example, create a domain account called svc-SQL and use it as the service account for SQL on every node of the SQL cluster.  This is required, because the SPN to be set up later can only use one account for the cluster as a whole.  Do the same for Analysis Services.

SSRS Cluster

Here again there’s nothing out of the ordinary; the order I’d do it in is:

• Provision the servers, configure the NLB to load balance across them, and create a DNS entry for the RS service on the NLB
• Install Reporting Services but don’t configure it
• Install the SharePoint binaries on each server and  join them to the farm (without making them a Web Front End)
• Run the RS configuration tool on the first server, create a new database in SharePoint mode, then complete the rest of the configuration (report server web service, etc.)
• Run the RS configuration on each other server, pointing them to the same Reports database.
• After configuring all the RS servers, add all the servers to the scale-out deployment using the appropriate tab on the configuration tool.  You only need to do this on one of the servers.

Again, you need to make sure the webservice is running under the same account on all servers.

Service Principle Names

First real challenge.  There are some key points to bear in mind:

• You create SPNs for the cluster name, not for the individual nodes in the cluster. Don’t create SPNs for the individual nodes as this seems to break Kerberos.
• You must specify the port number, even if you are using the default ports for SQL Server and Analysis Services.
• You must create SPNs using the fully qualified domain name of the servers, but creating them for the NetBIOS name seems to be optional – your mileage may vary on this.

The SPNs you will need are for:

• SQL cluster
• SSAS cluster
• RS cluster
• SharePoint Central Admin
• SharePoint web sites
• SharePoint applications such as PerformancePoint if you are using them

The commands will follow this template:

setspn –S MSSQLSvc/[SQL cluster].[domain]:1433 [service account]
setspn –S MSOLAPSvc.3/[SSAS cluster].[domain]:2383 [service account]
setspn –S HTTP/[RS NLB].[domain]:80 [service account]
setspn –S HTTP/[central admin server or NLB].[domain]:[port] [service account]
setspn –S HTTP/[web front end or NLB].[domain]:[port] [service account]

Let’s build some examples of that.  In this scenario, the SQL cluster is called SQL, the SSAS cluster is called AS, the SSRS load balancer is called RSNLB, the SharePoint Central Admin box is SP01, and the SharePoint web front end load balancer is called WFENLB. The domain is mycompany.local and all services are on their default port, other than Central Admin which is on port 8080.  You’d need to run the following commands.

setspn –S MSSQLSvc/SQL.mycompany.local:1433 svc-SQL
setspn –S MSOLAPSvc.3/AS.mycompany.local:2383 svc-AS
setspn –S HTTP/RSNLB.mycompany.local:80 svc-RS
setspn –S HTTP/SP01.mycompany.local:8090 svc-SPFarm
setspn –S HTTP/WFENLB.mycompany.com:80 svc-SPPortal

Something to bear in mind: if you have created an alternative URL for your SharePoint site (such as http://intranet), you need to create the SPN for that name instead of the WFE load balancer name.

Trusting for Delegation

On the DC, ensure that each of the following are trusted for Kerberos delegation:

• SharePoint Central Admin server
• SharePoint web servers
• Reporting Services servers
• SharePoint Central Admin web application pool account
• SharePoint web site application pool account
• Reporting Services web app service account

You can’t trust accounts until an SPN has been set up – the ‘Delegation’ tab doesn’t even appear in their account’s properties – so you have to complete this step after the SPNs have been created.

You might find you need to trust the SQL and SSAS boxes and service accounts.  You also might find you need to trust the virtual network names for their clusters.

Configure Reporting Services

On each of the SSRS servers, locate the file rsreportserver.config and edit it:

• Find the element <AuthenticationTypes> and ensure it contains the element <RSWindowsNegotiate/>.  Remove the element <RSWindowsNTLM/> if it exists.
• Find the element <UrlRoot> and insert the web address of the SharePoint site (in the example above the correct value would be http://WFENLB or http://intranet).
• Find the element <Hostname> and add the name of the SSRS cluster (in the example above the correct value would be RSNLB).  The <Hostname> element may not exist, so you will need to create it – it goes inside the <Service> element, the same as the <UrlRoot> element.  You would create a line such as <Hostname>RSNLB</Hostname>.

You must repeat for every SSRS server otherwise only some of the servers will actually work and your users will get ‘intermittent’ errors.

SharePoint

A trip to Central Admin should show that the RS servers have already been added to the farm.  Make sure the Central Admin and all other site collections are set to use Windows Authentication in all their zones which you intend to use SSRS in.

SharePoint and Reporting Services Integration

SharePoint 2010 automatically installs the SSRS add in as part of the pre-requisites.  For MOSS 2007, download the latest version of the add in and ensure your entire SP farm is service-packed.

Integration is then ready to be configured from Central Admin. In SP 2010 the config panel is at General Application Settings –> Reporting Services Integration.  The values you need are:

• Report Server Web Service URL: the address of the SSRS web service as accessed through the network load balancer, for example http://RSNLB/reportserver
• Authentication Mode: Windows Authentication
• Credentials: The username and password of a domain account which is in the local Administrators group on the SSRS servers.  If there isn’t a single account that’s in this group on every one of your SSRS boxes, use an account which is in the Administrators group for the first SSSR box in the deployment.  You need to specify the username in the format [domain]\[user]

After the first server is integrated, you are given the chance to add each of the other SSRS servers in the scale-out deployment to the integration.  You must complete this for each server you have in the deployment.

Finally, you can click ‘Set server defaults’ to change any of the the properties of the integration you want to alter.

Points to Bear in Mind

Not working?  Here’s what I check in my setup:

• Kerberos needs TCP and UDP ports 88 open in order to work.  Is your firewall blocking that port?  Try disabling the Windows firewall to check.
• Are the clients using Internet Explorer 6 or above?  Is the browser seeing that the site is in the local intranet zone? Does the zone allow integrated Windows authentication?
• Go back over the SPNs.  One mistake in your setspn command will have broken Kerberos completely.
• Check all the accounts and computers are trusted.
• Try navigating direct to the SSRS web service (i.e. not by opening a report in SharePoint’s interface).  If you can run a report through http://rsnlb/reportserver but get an error message when doing it through SharePoint, the probable cause is with a SharePoint SPN or delegation permission not with Reporting Services.

Database setup is exactly the same, and the two variables for environment and configuiration database connection string are set up the same, but the package needs just a script task.  The only property for the script which needs to be set is MaximumErrorCount which should be set to zero.

This script connects to the configuration database, selects all the variables that are valid for the current package and environment and applies them.  In the script code add another using for System.Data.SQLClient then add the following:

        public void Main()
        {
            // Create a variable dispenser and a variables collection
            VariableDispenser disp = this.Dts.VariableDispenser;
            Variables vars = default(Variables);

            String connString = null;
            String environmentName = null;
            String packageName = null;

            try
            {
                disp.LockForRead("User::strConfigDatabaseConnString");
                disp.LockForRead("User::strEnvironment");
                disp.LockForRead("System::PackageName");
                disp.GetVariables(ref vars);

                connString = vars["User::strConfigDatabaseConnString"].Value.ToString();
                environmentName = vars["User::strEnvironment"].Value.ToString();
                packageName = vars["System::PackageName"].Value.ToString();
                try
                {
                    // Connect to the configuration database
                    SqlConnection conn = new SqlConnection(connString);
                    conn.Open();

                    // Retrieve configuration values
                    String sql = String.Concat("select VarName, VarValue, VarType from ConfigVars where EnvironmentName = '",
                        @environmentName, "' and (PackageName = 'Sys' or PackageName = '", @packageName, "')");
                    SqlCommand command = new SqlCommand(sql, conn);

                    SqlDataReader reader = command.ExecuteReader();

                    // Loop through the reader
                    while (reader.Read())
                    {
                        String varName = reader["VarName"].ToString();
                        String varValue = reader["VarValue"].ToString();
                        String varType = reader["VarType"].ToString();

                        // Set the value for the variable
                        SetVariable(varName, varValue, varType);
                    }

                    // Tidy up
                    reader.Close();
                    reader.Dispose();
                    command.Dispose();
                    conn.Close();
                    conn.Dispose();

                }
                catch
                {
                    Dts.Events.FireWarning(0, "Set configuration variable values", "Unable to retrieve configuration values from database.", "", 0);
                }
            }
            catch
            {
                Dts.Events.FireWarning(0, "Set configuration variable values", "Unable to get configuration database connection string.", "", 0);
            }

            Dts.TaskResult = (int)ScriptResults.Success;
        }

        public void SetVariable(string varName, string varValue, string varType)
        {
            VariableDispenser disp = this.Dts.VariableDispenser;

            Variables vars = default(Variables);
            try
            {
                // Lock the variable we are going to provide a value for
                disp.LockOneForWrite(varName, ref vars);

                // Convert the value to the right type for the variable and assign it
                if (varType.ToLower() == "string")
                {
                    vars[0].Value = varValue;
                }
                else if (varType.ToLower() == "int32")
                {
                    vars[0].Value = Convert.ToInt32(varValue);
                }
                else if (varType.ToLower() == "datetime")
                {
                    // Add more strings to this array depending on the format of dates you have used in the config database
                    String[] sFormats = { "yyyy-mm-dd" };
                    DateTime dt = DateTime.ParseExact(varValue, sFormats, null, System.Globalization.DateTimeStyles.None);
                    vars[0].Value = dt;
                }
                else if (varType.ToLower() == "boolean")
                {
                    vars[0].Value = Convert.ToBoolean(varValue);
                }

                Boolean bReFire = true;
                Dts.Events.FireProgress(String.Concat("Set ", varName, " value to '", varValue, "'."), 100, 0, 0, "Set configuration variable values", ref bReFire);

            }
            catch (DtsRuntimeException ex)
            {
                Dts.Events.FireWarning(0, "Set configuration variable values", ex.Message, "", 0);
            }
        }

Very simple and very flexible.

Using Configurations is the normal way to set up your SSIS packages with the right values needed to make it behave correctly, but when you deploy your SSIS from development to test or live, you need to change those configurations.  There’s no easy mechanism for this.  Here’s a more flexible alternative which doesn’t use Integration Services’ built-in Configuration system, but lets you write your own mechanism.  It takes a bit of setting up but you only need to do it once – then you just reuse the template package.

The basic principle is that for every property which needs to be configured, you create a variable and use that variable as the property value in an expression.  For example, the connection manager for your source has an expression for ‘Connection String’.  So you create a variable named ‘@strSourceConnectionString’ and place that into the ConnectionString expression for the connection.  Then, the value for that variable is read from a database at runtime.  Crucially, you tune the SQL statement that retrieves variable values so that it retrieves the correct ones for your current environment.  This is what improves it over a standard Configuration.

The method:

Firstly, create a database table which will hold your variable values.  I would put this in the ETL management/audit database.  The columns you will need will be for the variable name, the variable value, the variable type, the environment and the package name (if you have variables with the same name but different values in each package).

These columns contain:

• The name of the variable to be configured – must be the same as the variable in SSIS.
• The value to be assigned to the variable.
• The data type of the variable in SSIS (because you are going to need to convert the string value to the right data type before it can be assigned).
• The environment for which this variable should be given this value.  You can put whatever you like in this column depending on how many environments you have and what they’re called.
• The name of the package for which this variable should be given this value.  This would need to match the name of the SSIS package exactly because it’s going to be searched for using the System::PackageName variable.  You can also use a value such as ‘SYS’ or ‘ALL’ to indicate the same value for the variable is used in all packages.

Examples might be:

You can see that strStringVariable has three entries – for the package VariableTest it has different values for Dev and Prod environments, and for the package VariableTest2 it has a different value.  The other variables apply to ‘Sys’ packages which I’m going to use to mean all packages.

With this structure in place, we can move to SSIS.  Your package will need to know which environment you are currently in and where to find all the configuration variables; for this I think the best way is to create two variables for Environment and ConfigConnectionString, and save this as an XML configuration.  When you deploy the SSIS to a different environment, just copy the .dtsconfig file to the same location (I create a new folder on the root of C:) on the new server.  Then you only ever have to change one config value in one XML file for the environment to retrieve all the right configurations.

Now create an ADO.NET connection manager and set its ConnectionString expression to the variable for the config database.

Create three string variables named strVarName, strVarValue and strVarType.  Also create the variables which you are going to add values to, remembering they need to be named exactly the same as the value in the VarName column in your config database.

The next thing to do is retrieve all the variable values from the config database and assign them to the variables in the package.  This could be done with a single script (here’s how) but here I am demonstrating grabbing the values then assigning them in a loop – it requires a bit less understanding of C# scripting in SSIS.

Add an Execute SQL Task, a ForEach Loop and a Script Task inside the loop.

The SQL task uses the config database connection, and its SQL statement is something like:

select VarName, VarValue,VarType from Configs
where EnvironmentName = @envName and (PackageName in (@pacName, ‘Sys’))

Map the @envName parameter to your Environment variable and the @pacName to the System::PackageName variable.  Set the ResultSet property to FullResultSet and place the result set into an object variable named objVariables.

Properties for the ForEach loop are:

• Enumerator: Foreach ADO Enumerator
• ADO object source vasriable: User::objVariables
• Variable Mappings: User::strVarName, User::strVarValue, User::strVarType

On the loop and the script, set the MaximumErrorCount property to zero.  This is because you will get errors if the variable you are trying to assign a value to doesn’t actually exist in the package – but you don’t want to crash the whole package because of this.

The basis of the script code you need is:

        public void Main()
        {

            // Create a variable dispenser and a variables collection
            VariableDispenser disp = this.Dts.VariableDispenser;
            Variables vars = default(Variables);
           
            string varName = null;
            string varValue = null;
            string varType = null;

            try
            {
                // Lock and assign variables
                disp.LockForRead(“User::strVarName”);
                disp.LockForRead(“User::strVarValue”);
                disp.LockForRead(“User::strVarType”);
                disp.GetVariables(ref vars);
                varName = String.Concat(“User::”, vars["User::strVarName"].Value.ToString());
                varValue = vars["User::strVarValue"].Value.ToString();
                varType = vars["User::strVarType"].Value.ToString();

                vars.Unlock();
            }
            catch
            {
                Dts.Events.FireWarning(0, “Set configuration variable values”, “Unable to lock variables.”, “”, 0);
            }

            // Call method which assigns values to SSIS variables
            SetVariable(varName, varValue, varType);

            // Finish
            Dts.TaskResult = (int)ScriptResults.Success;
        }

        public void SetVariable(string varName, string varValue, string varType)
        {
            VariableDispenser disp = this.Dts.VariableDispenser;

            Variables vars = default(Variables);
            try
            {
                // Lock the variable we are going to provide a value for
                disp.LockOneForWrite(varName, ref vars);

                // Convert the value to the right type for the variable and assign it
                if (varType.ToLower() == “string”)
                {
                    vars[0].Value = varValue;
                }
                else if (varType.ToLower() == “int32″)
                {
                    vars[0].Value = Convert.ToInt32(varValue);
                }
                else if (varType.ToLower() == “datetime”)
                {
                    // Add more strings to this array depending on the format of dates you have used in the config database
                    String[] sFormats = {“yyyy-mm-dd”};
                    DateTime dt = DateTime.ParseExact(varValue, sFormats, null, System.Globalization.DateTimeStyles.None);
                    vars[0].Value = dt;
                }

                Boolean bReFire = true;
                Dts.Events.FireProgress(String.Concat(“Set “, varName, ” value to ‘”, varValue, “‘.”), 100, 0, 0, “Set configuration variable values”, ref bReFire);

            }
            catch (DtsRuntimeException ex)
            {
                Dts.Events.FireWarning(0, “Set configuration variable values”, ex.Message, “”, 0);
            }

            // Unlock variables
            vars.Unlock();
        }
    }

Note that I have used a VariableDispenser where required, so you don’t need to add any variables to the ReadOnlyVariables or ReadWriteVariables.  This script only covers string, integer and datetime variables – if you’ve got other types you’ll need to add additional condition handlers to the if…elseif block to do the conversions required.

What happens when you run the package?  If you set the strEnvironmentName variable to ‘Dev’, you get this:

If you set the value to ‘Prod’ you get different values:

So, everything is completely portable between environments.

Let me know (Twitter: @DuncanSutcliffe) if this works for you or if it doesn’t.