I’ve always tried to avoid bug tracking tools and have several times deleted the entire content in such tools (reported bugs) to great success. However, that’s not what I want to talk about here, but if this sounds weird to you go check out Gojko’s post on the subject.
What I want to talk about here is one issue that came up during my team’s retrospective meeting on Friday. One of the improvement points that came up was bugs. Not that there where too many or that we should find more, but that some bugs where fixed and then reappeared as bugs again on a later stage. The suggested solution was: Instead of having the tester pull the buggy story from test and back into dev, he would write a test (integration or unit) that would replicate the bug and check it into source control.
Since we’re doing Kanban, what would the effect of the above action be? It will effectively stop the “assembly line”. In practice CI will report a failing test and the team will stop what they are doing to fix the failing test.
This is of course something we’ll adopt for all bugs, not just bugs that gets reported twice. I’ll leave it up to you to figure out all the benefits (and drawbacks if any) as well as why this solves our problem at hand. I’m looking forward to see how this will work out in practice.
Update: Part 2 is now available, covering Web Deploy (a.k.a. msdeploy)
Getting code deployed should be as easy as open up a web site (only taking slightly longer :-)). Devs or IT people should not spend manual labor (with the possibility of mishaps) on getting files from one place to another, making changes to IIS (or whatever you’re using), restarting servers, copy/changing web.config files etc. That’s the job for scripts and automation tools. Not to mention the cost savings of not needing IT people to do deployment. I bet you there’s no one manual step in the process of deployment that cannot be automated. Saying that, you have to consider how many times you deploy per day/week/month or year before going for full automation. However, if you’re serious about being Agile/Lean, you can’t do without a auto deployment scheme.
In the coming blog posts I’ll walk you through the steps I went through to automate our deployment, and hopefully you’ll find it interesting and even suggest improvements.
Below is an overview of the environments we’re deploying to:
One Load Balancer (LB) in each environment, two web servers in Dev and Test, and 3 in Prod. The actual numbers might or might not be true ;-), but that doesn’t really matter. In addition there’s SQL Servers, but I will not cover that here.
Why have a LB in Dev? Reason number one is to catch any possible LB issues in Dev before going to Test and Prod, and have the exact same environment in Dev as in Prod. It’s also useful to try out new stuff, like having the LB do caching etc.
For Dev we auto deploy every night (part of nightly build) and at will during day. For Test, 2-4 times per week and to Prod 1-2 times every 2nd week. That was yesterday! :-) Today we can do it when the sun comes out of the clouds (not often in Bergen), every time I refill my coffee cup or whenever we feel like. The point being: we are no longer constrained by how often we can deploy.
Why All These Environments?
You can read about that here, but for us:
Here’s the tech stuff we use which might be relevant:
Also note that we have access to the actual subnet where Test and Prod lives. This does however not mean we have access to all servers and features in all environments, it just means we can be given access to certain things not recommended through external firewalls, like PowerShell Remoting. This is where your environments might be different from ours.
Consider Using a LB Even If You Don’t Need One For Performance Reasons
Load Balancers are useful for other things than load balancing. The biggest benefit (except from its core task), is that you can do upgrades and maintenance on servers without taking the whole site offline, by always leaving at least one server online.
Consider Turning Off IIS Recycling
Do you know that IIS automatically recycle your applications every 1740 minute, effectively restarting them? Are your web sites free from memory leaks or do you want to know if you have memory leaks? Why not turn off recycling? This is too big of a topic to cover here, but go Google: IIS7 recycle.
Consider Using Windows Server 2008 R2 Server CORE
This should get you slightly better performance, but for me it’s more about scripting. Most of the things that needs to be done on server core, must be performed from command line, forcing you to create scripts.
Why Is Deployment Difficult?
First of all because every environment is different and there are no really good tools to automate the whole process. The challenge is to find the right tools to solve the problems your organization is facing, and have the tools work for you to get to the final goal.
What’s The Challenges?
For us it was about:
Safety and automation is two keywords that sticks out. Where safe means no-one else than the intended persons or services should be able to perform the specified tasks. Automation meaning no manual operations should ever be needed in either Dev, Test or Prod except from IT maintenance like hardware upgrades, windows update etc.
What tool options have we?
- Secure FTP in IIS 7 on a non public/available IP
- or PowerShell with BITS
- or WebDeploy
Taking LB nodes offline/online:
- Use PowerShell Remoting to execute PowerShell scripts on ARR server
- or the Web Farm Framework
Safely execute an upgrade:
- Use PowerShell Remoting to execute PowerShell scripts on web servers
- or WebDeploy
Avoid manual tasks:
- Script all tasks, so they can be repeated
The Build/Deploy Process
What About MSI’s?
If you read my blog you know I’ve done quite a bit of MS Installer stuff and WiX in particular. MSI’s are perfect for deploying to multiple places where you have no control. The drawback is that most developers don’t know how to customize MSI’s and often end up with a versioning problem and leaving lots of old stuff behind on the server after upgrades. If you have people skilled in Windows Installer, please feel free to use MSI, but I personally find XCopy to be very easy and is what I recommend if you’re not an ISV. With MSI’s you still have to install them remotely, which could be done with WMI or PowerShell.
Notes on WebDeploy
I’m currently looking at using Web Deploy to simplify/reduce the amount of scripts needed. WebDeploy would replace the FTP and deploy steps, but first impression is that it’s too generic, making it really hard to do simple things without spending quite a bit of time learning the tool, it’s underlying package schema and IIS schemas. Hopefully one day Web Deploy will be the only tool I’ll need to execute the whole deployment process.
In future blog posts I’ll walk you through step-by-step how to accomplish the above solution. While I’m writing this I’m not 100% sure if it will be a solution using PowerShell (which I have in production) or a slightly modified version using Web Deploy. It all depends on which one is easiest and which has the potential of being maintained by other people than me in the long run.
Hopefully this will give you the input you need to fully automate your deployment process as well.
Going to QCon London? I am and a bunch of my friends are too! Just a heads up that the early bird is about to expire this Friday.
If you haven’t been to QCon before, I can really recommend it. What I find especially attractive by this conference is its various tracks that should satisfy the most demanding devs and architects, and of course not to mention the speaker list (which is still growing). Here’s a few selected speakers from my point of view:
And here’s the tracks:
Hope to see you in London March 10-12 (March 8-9 for tutorials).
A couple of days ago, my article about “The Current Direction Of Agile” was published on InfoQ. The article mainly represent my own learning curve within agile and lean inspired processes, hoping that my view would be interesting to others. I feel the recent Lean inspiration that agile has gotten the last couple of years, has pushed the agile community into a better place and been given tools to improve agile further.