Testing Thoughts

May 27th, 2009

Testing has been on my mind recently. In fact I did an overview presentation and discussion for the company. I decided to do a “get everyone on the same page” type of presentation. Wasn’t what everyone expected, but I thought it was needed to be clear about terms and meanings. I now need to take it to the next need and discuss how the current tests match the different categories and where improvement can be targeted. Combine this with the lean software engineering concepts, and I should have a good start.

Now to the thoughts! I have been thinking that now only are there differents categories of tests; programmer, unit, regression, acceptance, etc. But there are levels within these categories. I am thinking these levels are like thresholds, where if you just wrote a new function or fixed a bug, you would run all tests. If you are just testing out a new platform or environment, maybe you would just run a level of tests that has caused problems in the past or tests that are known to identify targeted problems. Another level might be a configuration type, which tests different configurations within the same environment.

Of course, the task for me is to take an existing growing test base, which has been successful and propose a different way of doing things more efficiently. WITHOUT losing confidence in results.

An associate, who is the test engineer for the group, is doing a great job improving the existing testing environment! I hoping to attack the issues from the bottom, and get the individual tests or test groups to be available real-time. Maybe using a Tomcat server and ajax for web reporting of the results. This is where my idea of levels enter the picture!

I will report my experience soon,


Taking Software Engineering to a New Place

February 5th, 2009

Sometimes life intervenes and gives you a bonus. Since the last post I have started a new position, which will allow me to exercise my  knowledge of software engineering. Unfortunately it has taken awhile to get up to speed and return to this blog. I will continue with the theme of going through the Knowledge Areas, but will also intersperse what I am doing in my new position.

I am now working for The HDF Group. The folks here are top notch and the products they produce are excellent! So what can I contribute in software engineering that would make a difference? That’s the wonderful aspect of software engineering in general – there’s always room for improvement. What I have found is that they have developed significant software engineering practices. This is due in large part to education and the development environment. The opportunity for me, as a newcomer, is to document the practices and the principles behind them, gather all the odd papers and notes, and develop a reference system of these practices. Eventually, one learns how the operation works, as long as the folks with the knowledge are available. [Or you stumble through]

I also hope to work in the principles behind lean and agile software development. This includes my recent understanding of the need to seperate project management from software development, and rediscover systems engineering. I will soon be presenting a couple of lean software development seminars based heavily on Mary and Tom Poppendieck’s two books. I will upload them here as usual. Also, I hope to tailor a seminar based on Tom Gilb’s work to the development environment here.

Finally, I am already making plans to be at the Agile 2009 conference in Chicago this summer!


Software Configuration Management

September 24th, 2008

SWEBOK states :

Software Configuration Management (SCM) is the discipline of identifying the configuration of software at distinct points in time for the purpose of systematically controlling changes to the configuration and of maintaining the integrity and traceability of the configuration throughout the system life cycle.

With six subareas:

  • Management of SCM Process
  • Software Configuration Identification
  • Software Configuration Control
  • Software Configuration Status Accounting
  • Software Configuration Auditing
  • Software Release Management and Delivery

We are now into the support services of software engineering. This knowledge area deals with organizational management and processes which cover the entire life cycle of software products. According to IEEE/EIA Standard 12207: “The configuration management process is a process of applying administrative and technical procedures throughout the software lifecycle to; identify and define software items in a system; control modifications and releases of the items; record and report the status of the items and modification requests; ensure the completeness, consistency, and correctness of the items; and control storage, handling, and delivery of the items”.

SCM deals with how to control change efficiently and allow auditing of those changes.  Next time we will look at how lean and agile deals with theses issues.


Review of Agile 2008 Conference

August 25th, 2008

It’s been a few weeks since the conference, so let’s see what I thought of the Agile 2008 Conference in Toronto.

First, I think this will be my primary conference, although I would like to visit the Better Software Conference every now and then. I was pleasantly surprised to have enjoyed the pre-conference Research-In-Progress Workshop sessions. We listened to short presentations and were able to provide feedback to the presenters. The Keynote presenters were delightful and thought-provoking. And the biggest benefit was the quantity of presentations, which could cause you to wonder what you might have missed. I was pleased with all the sessions I attended, even when the speaker had difficulty speaking (nervous, foreign, etc.). I do try to make allowances for the speaker and try to concentrate on the subject being presented. The open spaces and social gatherings were wonderful chances to pursue other topics and user experiences. Of particular note is the week-long open development for a charitable organization, with the results presented at the closing banquet.

The major takeway thoughts I have are;

  • Software Engineering needs to be redfined
  • Project Development has mixed up Systems Engineering with Project Management
  • Agile principles have been developed from Lean principles.
  • Lean/Agile principles are as old as the Greeks
  • There needs to be Distinction between principles and methods
  • The community needs to promote Professionalism

What sessions did I enjoy?

  • Expanding Agile Horizons by Mary Poppendieck
  • Real Options by Olav Maassen and Chris Matts
  • Ancient Philosophers by Neal Ford
  • TDD Principles for Database Development by Dennis Hoyd and Sebastian Meine
  • What A CIO Wants to Know by Niel Nickolaissen
  • Collaboration Explained by Jean Tabaka
  • Keynote by Alan Cooper

BTW, check out the Pomodoro Technique. This technique for time management certainly enables the Agile principles. And check out the Jean Tabaka book or engage her as a consultant to learn how to manage a meeting!


Commentary on Software Maintenance

July 24th, 2008

If the maintenance of software is such a large part of any project, why is it treated as an afterthought. First let me state that with the frequent deliveries points with most all agile techniques, I believe that maintenance starts with at least the first delivery and with test driven development/programming it starts with the first test created. But for most of the software development era, maintenance happened after the original project development team went on to the next great project. It happened after the QA department allowed the product to ship to customers. It was regulated to the back room, or to the customer support group with the fresh out of school programmers.

Try doing a search on “software maintenance with agile or lean”, I found very little to support the old notion. The best quote I could find comes from Lean Software Development – Mary and Tom Poppendieck, page185; “Agile practices rule in software maintenance departments. In fact, these folks are wondering why it took the rest of the software development community so long to figure out how to develop production-ready software.”

In Software Engineering, Volume 1: The Development Process, 2nd edition, by the IEEE Computer Society Press, K.H. Bennett’s “Software Maintenance: A Tutorial” identifies four software maintenance type;

  • Corrective Maintenance – Maintenance performed to correct faults in hardware or software.
  • Adaptive Maintenance  – Software maintenance performed to make a computer program usable in a changed environment.
  • Perfective Maintenance – The improvements in performance or functionality of software; for example, in response to user suggestions and requests.
  • Preventive Maintenance – Designing a software system that is easy to maintain. The continuous upgrading of a system to enable the system to cope with current and future changes.

To me that seems a lot like some agile practices in building the original product. Maybe that is why my search was disappointing – maintenance is built-in! So the project length is the same as the life of the product.

How will the bean counters handle that?


Software Maintenance

June 9th, 2008

According to SWEBOK, this knowledge area addresses the phase of the life cycle that commences upon delivery. However, it acknowledges that maintenance activities can occur sooner. I think that agile further blurs this line, because of the frequent delivery cycles. SWEBOK breaks this area into four sub-areas;

  1. Software Maintenance Fundamentals – definitions and terminology, the nature of maintenance, the
    need for maintenance, the majority of maintenance costs, the
    evolution of software, and the categories of maintenance.
  2. Key Issues in Software Maintenance – the technical issues, the management
    issues, maintenance cost estimation, and software maintenance
  3. Maintenance Process – topics on the maintenance processes and maintenance
  4. Techniques for Maintenance – program comprehension, re-engineering, and
    reverse engineering.

The relevant IEEE standards for the definition of Software Maintenance is IEEE 1219 which states:

“the modification of a software product after delivery to correct faults, to
improve performance or other attributes, or to adapt the product to a modified environment.”

AND IEEE/EIA 12207 states:

“the process of a software product undergoing modification to code and associated documentation due
to a problem or the need for improvement. The objective is to modify the existing software product while
preserving its integrity.”

ISO/IEC 14764, the international standard is similar.

IEEE/EIA 12207 also identifies the primary activities of software maintenance;

  1. process implementation
  2. problem and modification analysis
  3. modification implementation
  4. maintenance review/acceptance
  5. migration
  6. retirement

Reading this knowledge area in SWEBOK, one finds that it consistently notes the misconception of maintenance as a minimal task in comparison to development. With notations to outside references, SWEBOK states that non-corrective actions account for over 80% of the effort. There is much more involved with software maintenance than just fixing “bugs”.

BTW, one of the technical issues identified by SWEBOK, is that of limited understanding. It states that 40-60% of the maintenance effort can be devoted to understanding the software to be modified.


How does Agile affect testing

April 15th, 2008

Testing was easy to identify in the waterfall method, it happened at the end just before shipping the product! Of course, if the project was over the estimated time for completion test was also the first to feel the pressure. Now with agile thinking, testing is involved throughout the process and doesn’t need to feel that end of project time crunch, right?

On the Sticky Minds site is a paper titled “An Uncomfortable Truth about Agile Testing” by Jeff Patton. He brings up the distinction between verification and validation.

  • Verification: the software conforms to specification, without failing
  • Validation: the software is fit for use

Jeff makes the point that “By building part of the software and demonstrating that it works, we’re able to complete one of the most difficult types of testing: validation.” So each iteration of the product, allows the user to test the validation of the product sooner and the team can make changes earlier so that the final product produces value to the user. The tester as an integrated member of the team constantly verifies that the code produced meets the requirements. This is where testing needs the most help, they could be testing the same type of functionality over and over. Automated tools might help, as well as computing resources, the tester as they are now become a critical member of the team.

Finally, what is the purpose of testing? To find errors! And everyone knows that the sooner you find them, the cheaper they are to fix.


Software Testing

March 21st, 2008

SWEBOK introduces this knowledge area with;

Testing is an activity performed for evaluating product quality, and for improving it, by identifying defects and problems.

SWEBOK defines it as follows:

Software testing consists of the dynamic verification of the behavior of a program on a finite set of test cases, suitably selected from the usually infinite executions domain, against the expected behavior.

In addition, SWEBOK provides further definition of four key issues in the above definition:

  • Dynamic: This term means that testing always implies executing the program on (valued) inputs. To be precise, the input value alone is not always sufficient to determine a test, since a complex, nondeterministic system might react to the same input with different behaviors, depending on the system state. In this KA, though, the term input will be maintained, with the implied convention that its meaning also includes a specified input state, in those cases in which it is needed. Different from testing and complementary to it are static techniques, as described in the Software Quality KA.

  • Finite: Even in simple programs, so many test cases are theoretically possible that exhaustive testing could require months or years to execute. This is why in practice the whole test set can generally be considered infinite. Testing always implies a trade-off between limited resources and schedules on the one hand and inherently unlimited test requirements on the other.

  • Selected: The many proposed test techniques differ essentially in how they select the test set, and software engineers must be aware that different selection criteria may yield vastly different degrees of effectiveness. How to identify the most suitable selection criterion under given conditions is a very complex problem; in practice, risk analysis techniques and test engineering expertise are applied.

  • Expected: It must be possible, although not always easy, to decide whether the observed outcomes of program execution are acceptable or not, otherwise the testing effort would be useless. The observed behavior may be checked against user expectations (commonly referred to as testing for validation), against a specification (testing for verification), or, finally, against the anticipated behavior from implicit requirements or reasonable expectations.

Finally, the SWEBOK identifies 5 subareas; Software Testing Fundamentals, Test Levels, Test Techniques, Test Related Measures, and Test Process.

Testing used to be an end of production – yes/no thing. Very much based on the production line thinking of the waterfall process. However, because of the nature of software production, it was actually a throw it back and forth over the programming wall process. Once the major show stopping bugs were fixed it was sent out to the customers. Agile has changed all that by promoting test driven development. Now it is considered standard operating procedure to involve testing from the very start. In fact, I had written about involving testers as part of pair programming. While this was a groundbreaking idea for me, later I found out that this was becoming a popular practice.

Of course, testers test the limits and boundaries of code, while developers use unit tests to create the code.


Agile Software Construction

February 9th, 2008

What does agile do for software construction?

Lets review what the agile principles are;

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

The principle that seems most applicable to construction is, Working Software. If developers use test-driven design, they will produce working software at each iteration. The program may not do much at the beginning, until customer priorities are completed, however the program should not crash and will provide successful indications of meeting the goals for the iteration.  Another technique is paired-programming. As I had mentioned in an early post, if this technique is used properly it can be a powerful implementation of peer reviews.

One more thought that comes to mind is the use of programming standards and in-code documentation. I’m not talking about long missives of committee commanded standards. I find that an adherence to common naming patterns and file structures help others in the team, and future people responsible for maintaining and changing code, quickly comprehend any code base.  There are many references to coding standards and naming conventions available, just pick one that everyone can work with, adjust it to fit, and stick with it. Also, a short sentence about a procedure or block of code of the intentions can also be benificial if it is maintained. Maintaining code comments will only happen if they are short, to the point and useful.

There are more references available about agile and lean as applied to software construction. I have mentioned some in past articles and I have a new reference on the menu: “Bookshelf”. Finally, remember the principles of lean, and one in particular – eliminate waste!


Software Construction

January 10th, 2008

The SWEBOK defines Software Construction as “the detailed creation of working, meaningful software through a combination of coding, verification, unit testing, integration testing, and debugging”.

The SWEBOK identifies three sub-areas;

  1. Software Construction Fundamentals
  2. Managing Construction
  3. Practical Considerations

It should be noted that the SWEBOK states that “the software construction process itself involves significant software design and test activity. It also uses the output of design and provides one of the inputs to testing, both design and testing being the activities, not the KAs[Knowledge Areas] in this case. Detailed boundaries between design, construction, and testing (if any) will vary depending upon the software life cycle processes that are used in a project.”

One of the agile techniques is TDD, Test Driven Development, which uses testing to drive coding and sort of flips the inputs and outputs. Of course most all agile techniques emphasize short iterations and blurs those boundaries between design, construction, and testing!

From Steve McConnell’s book, Code Complete, Microsoft Press, 1993;
The role of software construction is:

  • Verifying that the architectural design is acceptable
  • Designing and writing routines and modules
  • Creating data types and naming variables
  • Selecting control structures and organizing blocks of statements
  • Finding and fixing errors
  • Conducting peer reviews
  • Polishing code through careful formatting and commenting
  • Integrating the completed modules
  • Tuning the code to make it shorter and faster

Unfortunately, many people (inside as well as outside the industry) think that the software construction process is all there is to creating programs. While this may be the most labor-intensive effort, it is NOT the only necessary activity in software engineering. I recommend you find the “Computer Program vs. Programming Product” chart from “The Mythical Man-Month: Essays on Software Engineering, Addison-Wesley, 1975” by F.P. Brooks Jr.

I like to think of “Extreme Programming” as addressing “Software Construction” issues and “Agile/Lean” as addressing “Software Engineering” issues. Again this is an over simplification, but helps me keep my head from spinning.