How to reduce testing time and still improve quality

This is the conversation I've had the most frequently since joining Microsoft a few months ago.  The sides of the debate are framed like this:

People who have had experience in the world of online services will say that release time frames need to be short to remain competitive.  They'll cite agile methods with lightweight planning, and generally very light test plans.  People who have come from the packaged software (windows/office) side say that you end up cutting the corner on quality and by having longer releases can get more work at higher quality done.

Today I'll focus on the quality angle.  I argue that rapidly released software can be higher quality with a lower QA cost.  You can have your cake and eat it too.  To think through this, we need a measure of software quality.  That's an elusive problem, but for my purposes, we don't actually need to be perfect.  I propose:

  • Software Bugginess = Sum of cost of all bugs (that seems obvious)
  • Cost of a bug = Severity of bug * number of user sessions encountering it.
  • Severity is loosely defined as the amount of pain a bug causes to a user (and the part that is hard to quantify).
Now in the packaged software world, once the software is burned onto DVDs or pre-loaded on new computers for purchase, it's effectively frozen.  Oh yes, there are patches that get released, but those come with a fairly intrusive cost to users and are never 100% applied to all your customers.  So as a software development manager, to reduce the bugginess of your software you have to focus on getting rid of all high severity bugs.

Now I have no science to back this up, but I'm pretty sure most QA processes lead to a graph similar to the one at left.  Most of the bugs are ferreted out in the early phase of testing, but to getting rid of all high severity bugs becomes a decreasing return on investment.  So packaged software development managers must insist on a long test cycle to ensure high quality in their software.

Online services, however, have the beautiful property that you can update the software for all of your users by simply updating the server.  While that isn't always a trivial process, it is completely in the software development manager's control.  So now, instead of focusing on getting rid of every last high severity bug, we can focus on the other side of that equation; reducing the number of user sessions experiencing it.  That is a huge win because we can just do the first part of the QA curve where we find most of the bugs without paying the long price to get near 100% of the bugs out.  Further, there are lots of techniques for reducing the number of user sessions experiencing a bug, from rolling out the release slowly to a subset of servers, to doing daily bugfix releases to fix the bug you created yesterday.

But here's where most people screw it up.  People hear about agile and moving fast, and decide that they too must move fast to be competitive.  And they get excited about lowering their quality bar since it's easy to fix the bugs the next day.  And they hire only developers since all small companies should require developers to be testers (a subject of a future post).  And the developers rush out code they've barely tested at 7pm on Friday.  Then they go home, and are startled to get the SMS at 4am that their new code has broken.  So they resolve to fix it, and test it more carefully, and getting the fix and the test out takes until next Wednesday.
I see lots of places where they lower the quality bar, but don't take the next step of fixing problems quickly and generally release crap.  And thus was born the idea that agile development processes tend to be low quality.

But it doesn't have to be this way.  To release often and have a lower QA bar to release you must do several things:
  1. Have a simple and automated release process.  You plan to release often, but if it's cumbersome or manual, you will fail at this and spend too many resources doing releases.
  2. Have a simple and automated rollback process.  You're making a decision to roll out software that hasn't been rigorously tested.  You're definitely going to make mistakes and you'll need to fix them quickly.  Making sure that rollback works should be a part of your release to the staging environment before you release to production.
  3. Have a segment of your production site that you can release new software too without having to release to everyone.
  4. Have a source control environment that 100% replicates your production code, and is separate from your bleeding edge development environment.
  5. Plan to release regularly, and insist on fixing bugs the same day or next day they're discovered.
  6. Measure the number of days in production you've had for your various severities of bugs.  If you don't measure it, you will never fix it.
It's a culture and a mentality to do these things, but they're basics and really 101 level fundamentals on running a fast moving development organization.  But doing them, and focusing on the number of days in production for severe bugs, you can actually achieve higher quality systems than packaged software that has undergone a 6 month testing cycle.

Share this post!

Bookmark and Share


shreekavi said...

Amazing Post. keep update more information.
German Classes in Chennai
German Classes in Bangalore
German Classes in Coimbatore
German Classes in Madurai
German Language Course in Hyderabad
Best german classes in bangalore
German classes in marathahalli
German language classes in bangalore
Ethical Hacking Course in Bangalore
Software Testing Course in Bangalore

kalavani said...

The blog you shared is very good. I expect more information from you like this blog. Thankyou.
Python Training in bangalore
Python Course in Bangalore
Angularjs course Bangalore
Angularjs Training in Bangalore
Web Designing Course in bangalore
Web Development courses in bangalore
Salesforce Course in Bangalore
salesforce training in bangalore
Big Data Training in Bangalore
Hadoop Training in Bangalore

gem1musi said...

دانلود آهنگ جدید
دانلود آهنگ جدید
آهنگ شاد رقصیدنی
ریمیکس رادیو جوان
ریمیکس شاد قدیمی
آهنگ تولدت مبارک
آهنگ پرسپولیس
آهنگ رپ خارجی
آهنگ های علی یاسینی
آهنگ شاد عروسی

subha said...

thnaks for sharing boysdays page... it very useful to me..i also share my frds,..
AngularJS training in chennai | AngularJS training in anna nagar | AngularJS training in omr | AngularJS training in porur | AngularJS training in tambaram | AngularJS training in velachery

Unknown said...

Malakai Watene ZelezniakSuperCoach analysis: Elijah Taylor (corked thigh) is out. Matt Eisenhuth moves to lock with Chris Lawrence back in the staring back on the edge. Alex Twal (shoulder) and Josh Aloiai (broken hand) return on the bench with Jacob Liddle dropping out.Sea Eagles: 1. (tags: Jordan Shoes For Sale Cheap, 2020 Jordan Release Dates, Ray Ban Sunglasses Outlet)

She continues, "I never had an exit plan, so for me coming up with one was difficult, but it was rewarding in the end. He and I were both extremely proud of how he exited. We talked about it a lot and we were extremely proud. Nottinghamshire, earlier this month, had signed left arm spinner Graeme White on a two year contract, and the club's director Mick Newell suggested it had factored in releasing Brown. "Jason was unfortunate to suffer injury at a time when he could have played a significant part in limited overs cricket but we feel that the time is right for him to move on given the signing of Graeme White," Newell said. "It has been a frustrating year for him but there was no lack of effort on his part and I'm sure he'll find another county. (tags: Cheap Real Yeezys, Discount Jordan Shoes Wholesale, Cheap Yeezys)

"I loved our effort, I loved the way we worked," Rangers defenseman Kevin Shattenkirk said. "I think guys were hungry and desperate ourselves, we just had a lot of mistakes that fed them. Whether they went in or not, we just kept giving them momentum and not allowing ourselves to gain it ourselves." (tags: Coach Outlet Store Online , Michael Kors Bags Sale, Coach Outlet Store, MK Outlet).

hamdam said...

Artikel Anda mendorong saya untuk melakukan penelitian di bidang ini

خرید بلیط هواپیما

KimberlypTownson said...

Coach Outlet Online
Michael Kors Bags Outlet
Cheap Real Yeezys
Cheap Yeezys
Michael Kors Purse Outlet
Coach Outlet Store Online
MK Outlet
Coach Bags Clearance
Coach Bags Outlet

Ali jan said...

Increases profit margins. The increase in efficiency boosts in sales and reduction in costs along with customer satisfaction and all the rest ultimately lead to positive change in the overall profit margins. Salesforce will help you to attain maximum value and experience of customer interaction while increasing productivity. Salesforce interview questions and answers

mecolefqwarxp said...

You will agree with me that women are generally synonymous to fashion. And one of the fashion Coach Handbags Clearance that women adore so much when they want to look fashionable and thus look beautiful, is fine looking attractive and quality New Air Jordan Shoes. Thus, in as much as women love putting on quality fine looking Cheap Air Force Ones, research has confirmed that women just don't throw money around like their opposite counterpart, so quality fine looking MK Outlet they prefer, but they also prefer buying them cheap. An increasing number of people these days are finicky about the kind of Air Jordan Sale they wear. While almost everyone wants to wear the latest Michael Kors Factory Outlet and trendiest Cheap Yeezys not many are prepared to shell out exorbitant amounts of money for these. Therefore, if you want to know insider secrets to buy cheap Coach Factory Outlet Online read on to know more. (Article Source From Coach Outlet Clearance Sale)

vé máy bay từ Nhật Bản về Việt Nam said...

Mua vé tại Aivivu, tham khảo

khi nào có chuyến bay từ singapore về việt nam

đặt vé máy bay nha trang sài gòn

giá vé máy bay đi hà nội

giá vé máy bay rẻ đi nha trang

vé máy bay hà nội đi quy nhơn

thomas said...

MAJOR168 is open for football betting today. There are many big camps together BTi SBOBET IBCBET CMD365 if you are looking for a football betting website. Do not miss this site, there is football, there are all sports in the world. คาสิโนออนไลน์. Betting is available 24 hours a day with the best odds per pair in Thailand. Guaranteed automatic deposit and withdrawal system 10 seconds.

Live sports betting Online football betting Good price with every football match open for today online football betting SAGAME88 There are many big camps together, SBOBET IBCBET BTi CMD365, the only website complete in online football betting คาสิโนออนไลน์. There is every sport on the planet in here. With the automatic deposit and withdrawal system for 10 seconds, we have a live football system to watch every night.

We offer a wide variety of services. Called him the only player to finish with everything else does not have to go to the web preview ufabet as online. Online casinos Baccarat online Online betting games, Slot online, and with new technology, you can play ufabet via mobile phone today. Mobile Baccarat, play online via the website

Ufabet1688 of us again the way we are websites directly, not through a General Services, where customers will know it absolutely was extremely really no cheating possible on-site gambling online, it is ufabet1688 of us will hit prices.

Maradona Jons said...

Along with it also has a live casino 24 hours a day. สมัคร ufa Bet online gambling website is available to play with all casinos, baccarat, slots, roulette, dice and many other card games. We have gathered it here. With many promotions Apply for UEFA Bet now, we are happy to serve you all. With a professional team ready to solve problems And serve all of you There is a team to support the service. And answer your questions 24 hours a day.

Maradona Jons said...

Internet slots (Slot Online) is the introduction of a gambling machine. Slot machine As stated above Used to make electronic games known as web-based slots, as a result of the development era, many people have turned to gamble with each other by computers. Will bring slot games to make web based gambling games Via the world wide web network process Which players can play throughout the slot routine or even will have fun with Slots with the service provider's website Which internet slots games are available in the form of participating in rules. It's similar to playing on a slot machine. Both practical photos and sounds are equally thrilling as they go to lounge in the casino ever.


no deposit bonus forex 2021 - takipçi satın al - takipçi satın al - takipçi satın al - tiktok takipçi satın al - instagram beğeni satın al - instagram beğeni satın al - google haritalara yer ekleme - btcturk güvenilir mi - - - - - - google haritalara yer ekleme - altyapısız internet - mikrofiber havlu - - tiktok jeton hilesi - tiktok beğeni satın al - microsoft word ücretsiz indir - misli apk indir - binance güvenilir mi - takipçi satın al - mikrofiber havlu - uc satın al - takipçi satın al - takipçi satın al -

Unknown said...

takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
takipçi satın al
instagram takipçi satın al
instagram takipçi satın al
takipçi satın al
takipçi satın al
instagram takipçi satın al
instagram takipçi satın al
instagram takipçi satın al
instagram takipçi satın al
takipçi satın al
instagram takipçi satın al

Post a Comment