SoftwareTesting:SoftwareUpdateTesting

(write me)

I like to think in terms of risk, both the risk of something going wrong, and the cost if something does go wrong.

I can think of three separate systems involved in the overall software update process (and there might be more):

1. build system that generates the update and install files
2. servers that deliver update information and update files to client
3. client update process which queries for, receives, and applies the update files.

And I've heard about the following problems that occurred in the past:

• build system generated update file that removed everything
• server ran out of space, resulting in zero-length update file. Client looped trying to update.
• client-side update log file not created correctly
• update of legacy installation resulted in inconsistent state, causing odd problems

My proposed strategy for software update automated testing is as follows:

• add simple scripts to the build system to sanity-check the update files

The chance of generating bad update files is small, but the cost should that happen is huge.

--Davel 10:22, 24 Jan 2006 (PST)

Testing The Update Process

One thing which I suspect is time-consuming is actually testing the generated updates against all the builds they are supposed to apply to. Can we produce an automated system which, for various values of "old build", downloads and silently installs the full build, and then installs the old build and patches it to see if the bits are exactly the same?

This tests the update mechanism and URLs, and the integrity and correctness of the updates.

Would this obviate the need for a human to do the same process? If so, if you have 40 localisations and 3 platforms and 3 older versions, it might save a heck of a lot of time!

--Gerv

More Ideas

• (2006-02-27) IRC comments
  • davel: In the past, I asked for testing of the updater and/or update logic
  • davel: bsmedberg, you need to test 2 parts - update checking logic, and updater.
  • davel: bsmedberg, I think just snooping the url used to fetch update info is enough for first pass testing of the logic.
  • davel: bsmedberg, then create a set of nop (or limited-op) partial mar files, and run the updater by hand.
  • davel: bsmedberg, after that testing is complete, we can start on end-to-end testing. But I'd like to see the above tests done first to catch obvious bugs.