In Part 1 I hadn’t decided if I was going to use Web Deploy as the base of this blog series or the PowerShell scripts I already had in production. I decided to give Web Deploy a chance. In the end I didn’t regret it, but I must admit it was not straight forward. Hopefully this post will make it a breeze for you

THE commands

The two commands below solves all the requirements I listed in my previous post. I spent a great deal of time learning and studying Web Deploy in order to come up with this, so I hope you find it useful. It might not look too hard now, but nothing is when you have the solution in front of you I’m always open for better ways if you find one though. If I could ask the Web Deploy team for a new feature, it would be to make the two commands below really simple.

The rest of this post will discuss these commands in detail. Text in blue can be replaced with your own values.

Create package:

msdeploy.exe 
	-verb:sync 
	-source:manifest=PackageManifest.xml
	-dest:package=Package.zip,encryptPassword="certificatePassword" 
	-enableLink:AppPoolExtension
	-disableLink:CertificateExtension 
	-disableLink:ContentExtension 
	-replace:objectName=httpCert,targetAttributeName=hash,replace=08bf3e051bd10cd8d89ac1a3ac431887886ed343
	-replace:objectName=logFile,targetAttributeName=directory,replace=C:\Web\Logs 
	-replace:objectName=virtualDirectory,targetAttributeName=physicalPath,match="C:\\WebDeployTemplateWebSites",replace="C:\Web" 
	-declareParam:name=HttpsBinding,kind=DestinationBinding,scope=webSiteName,match=.*:443: 
	-declareParam:name=HttpBinding,kind=DestinationBinding,scope=webSiteName,match=.*:80:
	-declareParam:name=AppPoolUsername,kind=DeploymentObjectAttribute,scope=processModel,match=processModel/@userName 
	-declareParam:name=AppPoolPassword,kind=DeploymentObjectAttribute,scope=processModel,match=processModel/@password

Install package:

msdeploy.exe 
	-verb:sync 
	-source:package=Package.zip,encryptPassword="certificatePassword" 
	-dest:manifest=InstallManifest.xml,computerName=FQDN,username="domain\user",password="domainUserPassword" 
	-setParam:name=HttpsBinding,value=10.0.0.21:443: 
	-setParam:name=HttpBinding,value=10.0.0.21:80:
	-setParam:name=AppPoolUsername,value=MyDomain\AppPoolServiceAccount
	-setParam:name=AppPoolPassword,value=MySecureAppPoolPassword

Note that the above commands are one-liners. I only structured them the way I did to make them readable.

Web Deploy Command Options

Before diving into the inner workings of these commands, we need to know a bit more about the tool. Web Deploy have a lot of functionality and is extremely powerful. Here’s the command line syntax:

	msdeploy.exe -verb:<verbName>
	-source:<provider>[=<pathToProviderObject>
		[,<providerSetting>=<providerSettingValue>]]
	[-dest:<provider>[=<pathToProviderObject>
        	         [,<providerSetting>=<providerSettingValue>]]
	]
	[-<MSDeployOperationSetting> ...]

Doesn’t look too scary right? There is more to it though…

Verbs
The following verbs exist:

• delete
• dump
• getDependencies
• getParameters
• getSystemInfo
• sync

Providers
Let’s look at which providers it has:

• Web Deploy appHostConfig Provider
• Web Deploy appHostSchema Provider
• Web Deploy appPoolConfig Provider
• Web Deploy archiveDir Provider
• Web Deploy auto Provider
• Web Deploy cert Provider
• Web Deploy comObject32 Provider
• Web Deploy comObject64 Provider
• Web Deploy contentPath Provider
• Web Deploy createApp Provider
• Web Deploy dbFullSql Provider
• Web Deploy dbMySql Provider
• Web Deploy dbSqlite Provider
• Web Deploy dirPath Provider
• Web Deploy fcgiExtConfig Provider
• Web Deploy filePath Provider
• Web Deploy gacAssembly Provider
• Web Deploy iisApp Provider
• Web Deploy machineConfig32 Provider
• Web Deploy machineConfig64 Provider
• Web Deploy manifest Provider
• Web Deploy metaKey Provider
• Web Deploy package Provider
• Web Deploy recycleApp Provider
• Web Deploy regKey Provider
• Web Deploy regValue Provider
• Web Deploy rootWebConfig32 Provider
• Web Deploy rootWebConfig64 Provider
• Web Deploy runCommand Provider
• Web Deploy setAcl Provider
• Web Deploy urlScanConfig Provider
• Web Deploy webServer Provider
• Web Deploy webServer60 Provider

These are of course only the built-in providers, then you can create your own or use 3rd party ones if you like. A quick scan through this list shows that it does a lot more that just web stuff, like COM objects, registry settings, certificates, gac, databases.. the lot.

Provider Settings
The providers again have a set of common provider settings:

• authType
• computerName
• encryptPassword
• getCredentials
• ignoreErrors
• includeAcls
• password
• storeCredentials
• tempAgent
• userName
• wmsvc

MSDeployOperationSetting
From the documentation:

Web Deploy operation settings are non-provider specific command-line flags. They modify all of a Web Deploy operation.

• allowUntrusted
• declareParam
• declareParamFile
• dest
• disableLink
• disableRule
• disableSkipDirective
• enableLink
• enableRule
• enableSkipDirective
• postSync
• preSync
• removeParam
• replace
• retryAttempts
• retryInterval
• setParam
• setParamFile
• showSecure
• skip
• source
• useCheckSum
• verb
• verbose
• whatif
• xml
• xpath

Manifest Provider
We now have an overview of the command line syntax of the tool, but there are a few other important aspects. The first one being the manifest provider. Most likely you want to use more than one provider in your command, and that’s exactly what this provider does. Here’s an example from the documentation:

<sitemanifest>
   <appHostConfig path="mySite" />
   <gacAssembly path="System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
   <comObject path="Microsoft.ApplicationHost.AdminManager" />
   <contentPath path="c:\source" />
   <regKey path="HKLM\Software\ODBC" />
</sitemanifest>

I would actually preferred having these on the command line instead. When I trigger my commands from a build, I’m inserting variables into the command anyways and I would be better of with the ONE command and not have to maintain the xml files as well. Actually I would like to have both options. In the command I would suggest something like this (which would adhere to the existing conventions):

-source:appHostConfig=mySite,gacAssembly=System.Web…,[…]=[…]

Link Extensions
Another one is the concept of link extensions, which can be enabled/disabled by using the enableLink and disableLink operations. One example drawn from the documentation is:

…if you specify -disableLink:ContentExtension on the command line, you can prevent content from being included in a sync operation. This enables you to synchronize two Web servers without moving any content.

Web Deploy Rules
Web Deploy rules exists to disable or enable built-in or custom rules (using e.g. the enableRule and disableRule operations) for the sync verb. This is however (naturally) only true if the isDefault attribute is set to true in the rule definitions. See here to find the built-in rules. By creating the file Msdeploy.exe.configsettings in %program files%\IIS\Microsoft Web Deploy folder, you can add custom rules.

Packages and Archives
From the docs:

The Web Deploy package and archiveDir features let you create a snapshot backup of your Web site or Web server into a .zip file or archive directory. In addition, the parameterization and manifest features let you customize the archive or package that you create. You can then use your package files to deploy Web sites and Web servers to other computers or computer locations.

Setting up Web Deploy for remote access

To be able to access a server remotely, Web Deploy must be given access to the server. Several options are available, but I’ll focus on the Web Deployment Agent Service, which requires administrator privileges to use.

The Agent Service requires Web Deploy to be installed on the target server. You can find the detailed installation instructions here: http://technet.microsoft.com/en-us/library/dd569030(WS.10).aspx

Basic Usage

Let’s look at some vanilla examples for deploying web stuff remotely:

msdeploy
  -verb:sync
  -source:contentpath=c:\data,
  -dest:contentpath=c:\data,
      computerName=Server2,
      username=admin,
      password=pass

This copies your local C:\data directory to the server Server2, and provide a username and password to get access to the remote server. This is done by using the sync verb and the contentpath provider. So we can push files, which is nice cause we don’t have to use FTP, BITS or similar.

Deployment Requirements 

We want to do a lot more than just copy files. Here’s a list extracted from my previous post of what I want to deploy to remote servers:

• File content
• Virtual Directories and Applications settings
• AppPools settings
• Certificates
• Bindings
• Log location settings
• Virtual Directory file location settings

Packaging structure

My commands can be used to produce one package per web site multiplied by the number of environments you have (e.g. Dev, Test and Prod). So if you have 4 web sites and 3 environments, you create 12 packages.

To better understand why this is, here’s a list of the settings that are specific for each environment and common across all servers in one specific environment:

• File content
• SSL Certificate password
• Certificate hash/thumbprint
• Web site log directory location
• Virtual directory root locations

For server specific differences we create the package so that it accepts parameters for:

• User and password for the service account under which the worker process of the Application Pools runs
• The https binding (if used)
• The http binding (if used)

If we did not define the last three parameters, we would have to multiply the packages with the amount of servers in the different environments as well. 4 web sites, 3 environments and 2 servers in each environment = 24 packages. You don’t want to go there…

Another option would be to define all possible changes as parameters, leaving you with only one package. I tried that. I could not figure out how to accomplish all of these using params, so I analyzed what I needed and used a little of both. It works, but could be better.

As an example of the above, a package for the web site blog.torresdal.net could be created for e.g. Dev with these settings:

• File content = E:\MyBuildOutput\blog.torresdal.net
• Certificate password = MySecureSSLPassword
• Certificate hash=08bf3e051bd10cd8d89ac1a3ac431887886ed343
• Web site log dir = E:\Logs\blog.torresdal.net
• Virtual Directory locations = E:\Web

…and used these parameters when deploying:

• Username for AppPool = MyDomain\AppPoolServiceAccount
• Password for AppPool = MySecureAppPoolPassword
• Https binding = 10.0.0.21:443:
• Http binding = 10.0.0.21:80:

Dissecting The Create Package Command

I believe you now have an idea of what the commands above do, but lets pull it apart and describe each piece by itself.

The package manifest file
First we’re going to look at the manifest file which contains all source providers used to create the package:

<sitemanifest>
   <appHostConfig path="blog.torresdal.net" />
   <cert path="my\08bf3e051bd10cd8d89ac1a3ac431887886ed343 " />
   <dirPath path="E:\MyBuildDrop\LatestVersion\Dev\blog.torresdal.net" />
</sitemanifest>

This is the way that Web Deploy allows you to use more than one provider. Here’s a short description of each provider and what it does:

The create Package command
Now lets look at each section of the command:

Here’s a description of the parameter declarations allowing us to pass in params when deployment the package (these are also part of the same command):

To shorten the above command a tiny bit you can define the parameters in an xml file instead, like this:

<parameters>
   <parameter name="HttpsBinding" description="Web Site Binding for SSL">
      <parameterEntry kind="DestinationBinding" scope="sikker.frende.no" match=".*:443:" />
   </parameter>
   <parameter name="HttpBinding" description="Web Site Binding for http">
      <parameterEntry kind="DestinationBinding" scope="sikker.frende.no" match=".*:443:" />
   </parameter>
   <parameter name="AppPoolUsername" description="Account username for this application pool">
      <parameterEntry kind="DeploymentObjectAttribute" scope="processModel" match="processModel/@userName"/>
   </parameter>
   <parameter name="AppPoolPassword" description="Account password for this application pool">
      <parameterEntry kind="DeploymentObjectAttribute" scope="processModel" match="processModel/@password"/>
   </parameter>
</parameters>

You will then replace the –declareParam operation settings above with this:

-declareParamFile:MyParamFile.xml

However, if you’re going to execute this command from TFS or some other build tool, you’re better off leaving them inside the command. Less moving parts.

Dissecting The Deploy Package Command

We also need a manifest file when installing the package. This is ALMOST identical to the manifest we used when creating the package:

<sitemanifest>
   <appHostConfig path="blog.torresdal.net" />
   <cert path="my\08bf3e051bd10cd8d89ac1a3ac431887886ed343 " />
   <dirPath path="E:\Web" />
</sitemanifest>

The only difference between the two is in the dirPath provider. For the package we used files from E:\MyBuildDrop\LatestVersion\Dev\blog.torresdal.net, but when deploying we PUT files to E:\Web.

The deployment package command
Each section in detail:

The Build/Deploy Process Using Web Deploy

Here’s the exact same deployment process overview as in my previous post, only now with Web Deploy. As a result the process got two steps shorter

What’s Next?

In my next post I’ll either show you how to control the load balancer during deployment or integration with TFS. Both will be covered in the end.

Update: Part 2 is now available, covering Web Deploy (a.k.a. msdeploy)

Getting code deployed should be as easy as open up a web site (only taking slightly longer ). Devs or IT people should not spend manual labor (with the possibility of mishaps) on getting files from one place to another, making changes to IIS (or whatever you’re using), restarting servers, copy/changing web.config files etc. That’s the job for scripts and automation tools. Not to mention the cost savings of not needing IT people to do deployment. I bet you there’s no one manual step in the process of deployment that cannot be automated. Saying that, you have to consider how many times you deploy per day/week/month or year before going for full automation. However, if you’re serious about being Agile/Lean, you can’t do without a auto deployment scheme.

In the coming blog posts I’ll walk you through the steps I went through to automate our deployment, and hopefully you’ll find it interesting and even suggest improvements.

Overview
Below is an overview of the environments we’re deploying to:

One Load Balancer (LB) in each environment, two web servers in Dev and Test, and 3 in Prod. The actual numbers might or might not be true , but that doesn’t really matter. In addition there’s SQL Servers, but I will not cover that here.

Why have a LB in Dev? Reason number one is to catch any possible LB issues in Dev before going to Test and Prod, and have the exact same environment in Dev as in Prod. It’s also useful to try out new stuff, like having the LB do caching etc.

Deployment Frequency
For Dev we auto deploy every night (part of nightly build) and at will during day. For Test, 2-4 times per week and to Prod 1-2 times every 2nd week. That was yesterday! Today we can do it when the sun comes out of the clouds (not often in Bergen), every time I refill my coffee cup or whenever we feel like. The point being: we are no longer constrained by how often we can deploy.

Why All These Environments?
You can read about that here, but for us:

• Dev is where we try out things without physically hurting users, but still being in a real server environment avoiding the “works on my machine” issue.
• Test is as close to Prod as we can get (at external hosting provider, different network, firewalls etc) and where we make sure things run smoothly before going to Prod.
• Prod is Prod

Tech Details
Here’s the tech stuff we use which might be relevant:

• All servers are running Windows Server 2008 R2
• Web servers are running on IIS 7.5 (since we’re on R2)
• Application Request Routing in IIS is used as Load Balancer and runs on 2008 R2 Server Core (if you like, check out my previous post about setting up and configuring ARR)
• TFS 2010 for builds
• Team City for CI

Also note that we have access to the actual subnet where Test and Prod lives. This does however not mean we have access to all servers and features in all environments, it just means we can be given access to certain things not recommended through external firewalls, like PowerShell Remoting. This is where your environments might be different from ours.

Why Is Deployment Difficult?
First of all because every environment is different and there are no really good tools to automate the whole process. The challenge is to find the right tools to solve the problems your organization is facing, and have the tools work for you to get to the final goal.

What’s The Challenges?
For us it was about:

1. How can we safely move files from a build server to Dev, Test and Prod?
2. How can we automate the process of taking a node out of an LB cluster?
3. How can we safely execute an upgrade on a server in Dev, Test or Prod and get feedback of progress, errors, and abort and roll back on failure?
4. How can we remotely make changes to IIS?
5. How can we avoid all manual tasks? (like adding a virtual directory in IIS or copy a web.config file)

Safety and automation is two keywords that sticks out. Where safe means no-one else than the intended persons or services should be able to perform the specified tasks. Automation meaning no manual operations should ever be needed in either Dev, Test or Prod except from IT maintenance like hardware upgrades, windows update etc.

What tool options have we?

Copy files:

• Secure FTP in IIS 7 on a non public/available IP
• or PowerShell with BITS
• or WebDeploy

Taking LB nodes offline/online:

• Use PowerShell Remoting to execute PowerShell scripts on ARR server
• or the Web Farm Framework

Safely execute an upgrade:

• Use PowerShell Remoting to execute PowerShell scripts on web servers
• or WebDeploy

Avoid manual tasks:

• Script all tasks, so they can be repeated

The Build/Deploy Process     

What About MSI’s?
If you read my blog you know I’ve done quite a bit of MS Installer stuff and WiX in particular. MSI’s are perfect for deploying to multiple places where you have no control. The drawback is that most developers don’t know how to customize MSI’s and often end up with a versioning problem and leaving lots of old stuff behind on the server after upgrades. If you have people skilled in Windows Installer, please feel free to use MSI, but I personally find XCopy to be very easy and is what I recommend if you’re not an ISV. With MSI’s you still have to install them remotely, which could be done with WMI or PowerShell.

Notes on WebDeploy
I’m currently looking at using Web Deploy to simplify/reduce the amount of scripts needed. WebDeploy would replace the FTP and deploy steps, but first impression is that it’s too generic, making it really hard to do simple things without spending quite a bit of time learning the tool, it’s underlying package schema and IIS schemas. Hopefully one day Web Deploy will be the only tool I’ll need to execute the whole deployment process.

What’s Coming?
In future blog posts I’ll walk you through step-by-step how to accomplish the above solution. While I’m writing this I’m not 100% sure if it will be a solution using PowerShell (which I have in production) or a slightly modified version using Web Deploy. It all depends on which one is easiest and which has the potential of being maintained by other people than me in the long run.

Hopefully this will give you the input you need to fully automate your deployment process as well.

A few months back I went on a tour with MSDN Live here in Norway talking about WIF and ADFS. These talks where recorded, but only in Norwegian, so I did a screen cast of the same talk in English. This will eventually be available on Channel9, but until then I’ve made it available here.

WIF together with AD FS 2.0 really made it easy to do federation with partner organizations which was exactly what we needed in the company I work for. This talk is however not limited to that scenario, so if your looking for good solutions around Single Sign-On or identity management in general I recommend you check this out.

The Community For MVC.NET will be running a Live Meeting with Phil Haack on August 26th @ 9am PST (4pm GMT / 6pm CET) on upcoming MVC 2 features.

Even though I don’t develop software in SharePoint, this blog post by Lee Richardson reflect my thoughts and worries about SharePoint development.

The essence of his post is in this paragraph:

But the power given to end users makes rigor and good design extremely hard for developers because the production machine you deployed to last month may look completely different now that you’re ready to redeploy. And Microsoft makes good design even harder with an inflexible API. Critical classes like SPSite (think SqlConnection) contain no public constructor, rendering them completely unmockable (unless you’re willing to spend $450 per developer for some TypeMock Isolator magic). And vital classes like SPList (think DataSet) are marked final, crippling your ability to make nice strongly typed entities in your architectures.

Lee proposes some techniques to use that you should read in his post, but it’s half baked and not good enough in my opinion. The things he suggests (which is usually taken for granted on any other project) is made really hard to do by MS. Should MS rethink the way SharePoint is architected for the sake of developers to ease maintenance, improve code quality and basically allow developers working with ShapePoint to be craftsmen? Or is the dynamics of the product given to the end user so good that it’s worth all the extra efforts?

A good friend of mine, Mark Nijhof, will be doing a presentation on FubuMVC at the next European Virtual Alt.NET meeting. Mark is actively involved with the alternative MVC framework for ASP.NET together with Chad Myers and Jeremy D. Miller. Mark will also do a Fubu presentation at NNUG Bergen 27th of May (invitation will be available soon).

If you want to know more about FubuMVC you can check out the interview I did with Chad on InfoQ and of course their public website. I encourage you to download Fubu and try it out, cause it has some interesting differences which I personally like compared to ASP.NET MVC.

Also, Jeremy Miller is coming to Bergen and NNUG in mid June just before NDC (thanks Mark for organizing this!), so I’m really looking forward to that. A lot of stuff happening at the local community in Bergen at the moment!