How to reduce testing time and still improve quality

This is the conversation I've had the most frequently since joining Microsoft a few months ago.  The sides of the debate are framed like this:

People who have had experience in the world of online services will say that release time frames need to be short to remain competitive.  They'll cite agile methods with lightweight planning, and generally very light test plans.  People who have come from the packaged software (windows/office) side say that you end up cutting the corner on quality and by having longer releases can get more work at higher quality done.

Today I'll focus on the quality angle.  I argue that rapidly released software can be higher quality with a lower QA cost.  You can have your cake and eat it too.  To think through this, we need a measure of software quality.  That's an elusive problem, but for my purposes, we don't actually need to be perfect.  I propose:

  • Software Bugginess = Sum of cost of all bugs (that seems obvious)
  • Cost of a bug = Severity of bug * number of user sessions encountering it.
  • Severity is loosely defined as the amount of pain a bug causes to a user (and the part that is hard to quantify).
Now in the packaged software world, once the software is burned onto DVDs or pre-loaded on new computers for purchase, it's effectively frozen.  Oh yes, there are patches that get released, but those come with a fairly intrusive cost to users and are never 100% applied to all your customers.  So as a software development manager, to reduce the bugginess of your software you have to focus on getting rid of all high severity bugs.
 

Now I have no science to back this up, but I'm pretty sure most QA processes lead to a graph similar to the one at left.  Most of the bugs are ferreted out in the early phase of testing, but to getting rid of all high severity bugs becomes a decreasing return on investment.  So packaged software development managers must insist on a long test cycle to ensure high quality in their software.

Online services, however, have the beautiful property that you can update the software for all of your users by simply updating the server.  While that isn't always a trivial process, it is completely in the software development manager's control.  So now, instead of focusing on getting rid of every last high severity bug, we can focus on the other side of that equation; reducing the number of user sessions experiencing it.  That is a huge win because we can just do the first part of the QA curve where we find most of the bugs without paying the long price to get near 100% of the bugs out.  Further, there are lots of techniques for reducing the number of user sessions experiencing a bug, from rolling out the release slowly to a subset of servers, to doing daily bugfix releases to fix the bug you created yesterday.

But here's where most people screw it up.  People hear about agile and moving fast, and decide that they too must move fast to be competitive.  And they get excited about lowering their quality bar since it's easy to fix the bugs the next day.  And they hire only developers since all small companies should require developers to be testers (a subject of a future post).  And the developers rush out code they've barely tested at 7pm on Friday.  Then they go home, and are startled to get the SMS at 4am that their new code has broken.  So they resolve to fix it, and test it more carefully, and getting the fix and the test out takes until next Wednesday.
I see lots of places where they lower the quality bar, but don't take the next step of fixing problems quickly and generally release crap.  And thus was born the idea that agile development processes tend to be low quality.

But it doesn't have to be this way.  To release often and have a lower QA bar to release you must do several things:
  1. Have a simple and automated release process.  You plan to release often, but if it's cumbersome or manual, you will fail at this and spend too many resources doing releases.
  2. Have a simple and automated rollback process.  You're making a decision to roll out software that hasn't been rigorously tested.  You're definitely going to make mistakes and you'll need to fix them quickly.  Making sure that rollback works should be a part of your release to the staging environment before you release to production.
  3. Have a segment of your production site that you can release new software too without having to release to everyone.
  4. Have a source control environment that 100% replicates your production code, and is separate from your bleeding edge development environment.
  5. Plan to release regularly, and insist on fixing bugs the same day or next day they're discovered.
  6. Measure the number of days in production you've had for your various severities of bugs.  If you don't measure it, you will never fix it.
It's a culture and a mentality to do these things, but they're basics and really 101 level fundamentals on running a fast moving development organization.  But doing them, and focusing on the number of days in production for severe bugs, you can actually achieve higher quality systems than packaged software that has undergone a 6 month testing cycle.

Share this post!

Bookmark and Share

19 comments:

subha said...

thnaks for sharing boysdays page... it very useful to me..i also share my frds,..
AngularJS training in chennai | AngularJS training in anna nagar | AngularJS training in omr | AngularJS training in porur | AngularJS training in tambaram | AngularJS training in velachery

Anonymous said...

Malakai Watene ZelezniakSuperCoach analysis: Elijah Taylor (corked thigh) is out. Matt Eisenhuth moves to lock with Chris Lawrence back in the staring back on the edge. Alex Twal (shoulder) and Josh Aloiai (broken hand) return on the bench with Jacob Liddle dropping out.Sea Eagles: 1. (tags: Jordan Shoes For Sale Cheap, 2020 Jordan Release Dates, Ray Ban Sunglasses Outlet)

She continues, "I never had an exit plan, so for me coming up with one was difficult, but it was rewarding in the end. He and I were both extremely proud of how he exited. We talked about it a lot and we were extremely proud. Nottinghamshire, earlier this month, had signed left arm spinner Graeme White on a two year contract, and the club's director Mick Newell suggested it had factored in releasing Brown. "Jason was unfortunate to suffer injury at a time when he could have played a significant part in limited overs cricket but we feel that the time is right for him to move on given the signing of Graeme White," Newell said. "It has been a frustrating year for him but there was no lack of effort on his part and I'm sure he'll find another county. (tags: Cheap Real Yeezys, Discount Jordan Shoes Wholesale, Cheap Yeezys)

"I loved our effort, I loved the way we worked," Rangers defenseman Kevin Shattenkirk said. "I think guys were hungry and desperate ourselves, we just had a lot of mistakes that fed them. Whether they went in or not, we just kept giving them momentum and not allowing ourselves to gain it ourselves." (tags: Coach Outlet Store Online , Michael Kors Bags Sale, Coach Outlet Store, MK Outlet).

hamdam said...


Artikel Anda mendorong saya untuk melakukan penelitian di bidang ini


خرید بلیط هواپیما

KimberlypTownson said...

Coach Outlet Online
Michael Kors Bags Outlet
Cheap Real Yeezys
Cheap Yeezys
Michael Kors Purse Outlet
Coach Outlet Store Online
MK Outlet
Coach Bags Clearance
Coach Bags Outlet

Ali jan said...

Increases profit margins. The increase in efficiency boosts in sales and reduction in costs along with customer satisfaction and all the rest ultimately lead to positive change in the overall profit margins. Salesforce will help you to attain maximum value and experience of customer interaction while increasing productivity. Salesforce interview questions and answers

vé máy bay từ Nhật Bản về Việt Nam said...

Mua vé tại Aivivu, tham khảo

khi nào có chuyến bay từ singapore về việt nam

đặt vé máy bay nha trang sài gòn

giá vé máy bay đi hà nội

giá vé máy bay rẻ đi nha trang

vé máy bay hà nội đi quy nhơn

Maradona Jons said...

Along with it also has a live casino 24 hours a day. สมัคร ufa Bet online gambling website is available to play with all casinos, baccarat, slots, roulette, dice and many other card games. We have gathered it here. With many promotions Apply for UEFA Bet now, we are happy to serve you all. With a professional team ready to solve problems And serve all of you There is a team to support the service. And answer your questions 24 hours a day.

aaronnssd said...

I liked your work and, as a result, the manner you presented this content Software Development company in Noida Uttar Pradesh. It is a valuable paper for us. Thank you for sharing this storey with us.

amc stars said...

tanx for post
کتاب جنایت و مکافات
https://www.pinterest.com/pin/1043779651108380245/

amc stars said...

tanx for post
دانلود قسمت 1 اول سریال جیران
دانلود قسمت آخر سریال جیران
دانلود سریال جیران
دانلود قسمت 16 شانزدهم سریال خاتون
دانلود قسمت 15 پانزدهم سریال خاتون

Anonymous said...

mmorpg oyunlar
İnstagram Takipci Satin Al
TİKTOK JETON HİLESİ
Tiktok Jeton Hilesi
antalya saç ekimi
referans kimliği nedir
İnstagram takipçi satın al
Mt2 Pvp
instagram takipçi satın al

Anonymous said...

tuzla toshiba klima servisi
tuzla beko klima servisi
çekmeköy lg klima servisi
ataşehir lg klima servisi
çekmeköy alarko carrier klima servisi
ataşehir alarko carrier klima servisi
beykoz beko klima servisi
üsküdar beko klima servisi
pendik lg klima servisi

Back linker said...

EDIUS Pro Crack is the one of the best universal video editing tool that delivers the efficient choice to the industry leaders. Edius 9 Free Download Full Version With Crack

Samia Beel said...

Mixed in key crack is the best option for Djs for creating genuine music for their work. This amazing software helps to mixing up your favorite keys according to your track. Mixed In key Free Download

hacklink said...

Good content. You write beautiful things.
sportsbet
taksi
mrbahis
mrbahis
hacklink
korsan taksi
vbet
vbet
sportsbet

tipobet said...

Good text Write good content success. Thank you
kibris bahis siteleri
betmatik
bonus veren siteler
kralbet
mobil ödeme bahis
slot siteleri
poker siteleri
betpark

atilla said...

kütahya
nevşehir
erzincan
adana
bartın

FB8K6

Serkan said...

yurtdışı kargo
resimli magnet
instagram takipçi satın al
yurtdışı kargo
sms onay
dijital kartvizit
dijital kartvizit
https://nobetci-eczane.org/
L0L2D

ceren said...

https://saglamproxy.com
metin2 proxy
proxy satın al
knight online proxy
mobil proxy satın al
1İ7DNY

Post a Comment