Monday, September 09, 2013

Apache Felix fails to run in background (including nohup case)

How to run Apache Felix in background with nohup. Solution is here. In my case I faced the problem when tried to run Squash TM in background. That application is built over Apache Felix and the exceptions told something like:

java.util.zip.ZipException: ZipFile closed
at java.util.zip.ZipFile.ensureOpenOrZipException(ZipFile.java:642)
at java.util.zip.ZipFile.access$1300(ZipFile.java:56)
at java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:675)
at java.util.zip.ZipFile$ZipFileInflaterInputStream.fill(ZipFile.java:413)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:158)
at org.apache.felix.framework.util.WeakZipFileFactory$WeakZipFile$WeakZipInputStream.read(WeakZipFileFactory.java:707)

or

Spring DM context shutdown thread WARN 06/09 14:35:39 [org.springframework.osgi.context.support.OsgiBundleXmlApplicationContext] - Exception
thrown from ApplicationListener handling ContextClosedEvent
java.lang.IllegalStateException: ApplicationEventMulticaster not initialized - call 'refresh' before multicasting events via the context:
OsgiBundleXmlApplicationContext(bundle=org.squashtest.tm.plugin.testautomation.jenkins, config=osgibundle:/META-INF/spring/*.xml): startup date
[Fri Sep 06 14:35:34 BST 2013]; root of context hierarchy
at org.springframework.context.support.AbstractApplicationContext.getApplicationEventMulticaster(AbstractApplicationContext.java:340)

Certain amount of investigation shown that the common solution in such the case is to exclude the application binding to the console. So do the following:

  1. As it's recommended here check if you have ConsoleHandler enabled for your Log4j and disable it using the following entry: java.util.logging.ConsoleHandler.level = OFF (if you're using property file to configure your logging. otherwise use the xml equivalent)
  2. Drop the bundles holding -shell suffix from your Bundles folder. They could look like the following:
    1. org.apache.felix.shell.tui-version.jar
    2. org.apache.felix.gogo.shell-version.jar
  3. As the alternative of the previous step you may add -Dgosh.args=--nointeractive  option to the jvm arguments on calling your app from sh sctipt
  4. Run your application in the regular way like nohup ./startup.sh &
This should help.

Saturday, August 31, 2013

Yet another hint on "unbound prefix" in your programmatically composed SVG file

If you compose your SVG file programmatically you might experience the following issue. Your resulting file won't be parsed with the following exception coming from the parser:

com.larvalabs.svgandroid.SVGParseException: org.apache.harmony.xml.ExpatParser$ParseException: At line 4, column 0: unbound prefix
This exception basically says that somewhere in your SVG markup you use either the tags or the attributes fromt he namespace you didn't specify in SVG header. Examine your resulting SVG code for such the sort of problem. Fixing should help.

Wednesday, August 28, 2013

How to test mailing functionality or setting up local mail server under windows

Introduction

Sometimes you need to test how your application sends the mail (quite regular objective in QA). Whether the mails are properly configured, whether the mail list is properly configured, the attachments are in place etc. Sometimes you need to test if your application reads the mail in the expected way. At the same moment you do no like to use your corporate mail server or services like gmail as:
  • you have no all required permissions to operate with the server
  • protocol properties configured for existing server do not meet the production case
  • ports or other existing server properties do not meet the production case
  • you violate user agreement of the mail service if you load it with test traffic
  • you're working inside isolated network so all the outgoing traffic is prohibited
  • due to any other reason
The above restrictions do not leave the chance to go without your local mail server deployment.

Test sending the mail

Once you're going to only test e-mail sending feature you may manage with the tool called PaperCut. This is the simple SMTP server which does not push the mail to the addressees but just stores it and lets you to see it in three possible views: raw view, body view and html view. It does not require installation and starts working just by doublecliking its executable file. You are notified about new mails by the animation in your system tray. So, once your have PaperCut running, configure your appliaction to use the SMTP server set to the PaperCut host and test.

Test receiving the mail

If your application receives the mail or somehow operates with the mail on the server the testing might become a bit more difficult. So you obligatory need to setup the dedicated server. This post is mostly devoted to the basics of hMailServer deployment and configuring. hMailServer is running under Windows OS and have user-friendly administration. It also provides all the well-known mail protocols such as:

  • POP3
  • SMTP
  • IMAP

So, here is the example of environment topology that one could use to deploy e-mailing functionality test harness.
Test mailing topology
A couple of words regarding the picture. As it is seen, the mail server stands out of the other intranet cloud. That basically means that we should not allow that server to communicate with other network boxes to prevent the accidental mail delivery. Moreover the application server and tester station should not have any mail servers running or some mail port routing configured.
In my example I'm gonna use Win7 box to host the mail server of version 5.4.


Test scenario

The schema above may vary on the specific circumstances of your personal objectives. So lets guess script the mentioned topology will fit in the best way. I think the simple scenario might be the following:
  • Conditions: 
    • application uses one mail account to communicate with mail server
    • application reads the mail and then, say, parses it
    • parsed data is stored on the hard drive according to some application rules which we are aware of
    • we've got the way to customize the mail interface properties for the application
  • Steps:
    1. Send the mail to the mailbox the application is listening to
    2. Verify that the mail data is parsed according to the application rules
Step #1 looks conflicting with the mail server isolation statement. It really does in some case. In such the schema we'll not be able to send the mail to the isolated mail server account from anywhere except of another account hosted on that server. However lets get back to the problem a bit later.


hMailServer installation and configuration

In this section we're implementing the simple and working configuration of mail server 
  1. Download the distribution from here.
  2. Run the installer and just click Next for each request. Choose recommended Full Installation
  3. Once you're done the administrative console is started automatically
  4. Choose the only server you have yet and proceed (use the password you set up during installation process).
  5. You should now see the main administrative form where all the magic happens
  6. Add new domain. This is what will be following @ symbol in your mail addresses. It should conform your hosting address not to give the problems with mail routing inside your intranet - say, MAILHOST.SUBDMN.YOURCOMPANY.COM (the case does not matter)
  7. In this example we're gonna configure sender and receiver accounts. You may extend the steps for configuring accounts for other roles your application supports.
    1. Expand your domain, right-click Accounts folder and choose Add... from context menu
    2. Specify address and password for the account
    3. You've got the account created now. If you look at the corresponding tree item you see that the account is defined in "full qualified" style. It means that once you're attempting to connect to the server you need to specify that string as your server account. So add two account, specify the following addresses for them: sender, receiver
  8. You now should configure mailing protocols and privileges for certain IP addresses.
    1. So, to make our simple configuration working we may leave the protocol settings set to default values
    2. Lets now configure our IP ranges. IP ranges allow or disallow certain actions for the users whose IP address matches the specified one. To configure IP ranges you should expand the branch Settings\Advanced\IP ranges
    3. By default the server has two configurations which describe the privileges for the host where your server is running and the entire range of available IP addresses. In our example we are to configure the less secure configuration but providing the easiest way to access the mailing feature. 
      1. Click Internet item under IP ranges folder
      2. Check all the check-boxes under Allow deliveries from
      3. Uncheck all the check-boxes under Require SMTP authentication
  9. To ensure our server will be the finish point of any mail even addressed to the boxes hosted at some external domains block outbound SMTP port via the firewall


Configuring clients

We still need to be capable to implement our test script. So now we have the server configured and the last thing to do is to configure the clients. According to the script we should configure the following clients:
  • The application that reads the mail from the receiver box (requires POP3 configuration only)
  • Some convenient "well-known" mail client like Thunderbird where you'll be pushing your test mails from. (Usually mail clients imply both the protocols - for sending and receiving mails - are to be configured)
So use the following settings for your clients:
  • POP3
    • account
      • for sender: sender@MAILHOST.SUBDMN.YOURCOMPANY.COM
      • for receiver: receiver@MAILHOST.SUBDMN.YOURCOMPANY.COM
    • pop3 server: MAILHOST.SUBDMN.YOURCOMPANY.COM
    • pop3 port: 110 (or that one you configured for your server)
    • authentication type: password
  • SMTP
    • account
      • for sender: sender@MAILHOST.SUBDMN.YOURCOMPANY.COM
      • for receiver: receiver@MAILHOST.SUBDMN.YOURCOMPANY.COM
    • smtp server: MAILHOST.SUBDMN.YOURCOMPANY.COM
    • smtp port: 25 (or that one you configured for your server)
    • Authentication: none
So having all your clients mapped to your local server you are free to play with your tests without the risk to send some unwanted mail to your colleague or clients :)

Friday, June 14, 2013

SquashTM (Squash TM) 1.5.0 test management system review

Introduction

Let's  continue exploring test case management systems. This time I chose the one called SquashTM (Squash Test Management or Squash TM) - open-source pretty looking and of broad functional scope tool from French team. Unlike some other products SquashTM offers full scope of functions on free-of-charge basis which definitely gives some extra points to the system.
You also might want to visit my previous review of Zephyr Desktop TCMS. In such the case please follow this link.

Web-site

One of the weakest components of the product is the web-site. The face of a product intended to raise the popularity has the big problem with localization. Shame on me, but I really don't speak French and I guess the most part of the world doesn't. Coming to the site of a product pretending to be the world-wide famous one in some observable future I expect to see convenient English version. I believe that will help to develop and to distribute the product much faster than now.
Well.. Anyway I tried to use google translate and that helped me to manage to download the binary distribution. By the moment the latest available (and stable) version is marked as 1.5.0. The supported operating systems are:
  • Windows
  • Mac OS
  • Linux
The site has a Wiki part. The wiki made me completely confused. Fortunately it is written in English but the table of content has the topics ordered alphabetically. Thus the installation guide can be found somewhere in the middle of the topic list, however the most convenient place I guess is somewhere at the top.

Installation

SquashTM (or Squash TM) is distributed as jar installer (for Windows OS at least). Despite I chose English version the installation wizard still supplied the dialog controls with the captions like "Suivant" and "Quitter". Fortunately the control icons didn't give me the chance to go the wrong way. Just click the button with big green forward-arrow every time and you will succeed.
SquashTM (or Squash TM) fits people who would like to play with the system before take the positive or negative decision and at the same time wouldn't like to spend much time for setting application up as it comes with default H2 database. So you can use it right after the installer finish working. 
Looks like there is no way to migrate the data from the default database to your custom one (MysSQL is supported). There is direct warning in the Wiki at least saying literally the following

 Caution : by switching to a MySQL database, all data previously entered in the on-board database will be lost.

You can also install the system as Windows service so you don't have to care of starting the application each time your server gets rebooted.
SquashTM (or Squash TM) distribution binds to 8080 port by default so don't forget to change it once you've got a lot of web services running at your server. But please be aware of the official instructions which acn e found on this page are not correct. In my case I had to change environment variable HTTP_PORT as well to make my instance working on non-default one.

Features

SquashTM (or Squash TM) offers the standard set of features (see official terms description here)
  • Integrating with SquashTA (the review is coming)
  • Managing the isolated projects
  • Managing Requirements (with customizable fields and versioning)
  • Managing test-cases (with customizable fields)
  • Managing campaigns (with customizable fields)
  • Test steps support fields customization
  • Iterations support field customization
  • Integration with mantis bug-tracking system
  • Managing the users and the teams. Each user should be assigned with the predefined permission set called "Profile". See the details here.
  • Reporting

Usability

 General controls

In general the application UI looks quite attractively. The layout consists of three vertical column. The leftmost one and the most narrow at the same time holds workspaces controls. Using it you can switch between requirements, test-cases, campaign and report workspaces. Once you choose one you see two other columns. 
The middle column holds the tree of entities which are currently under management according to workspace selected. You may add, edit or remove entity, arrange them by the folders and perform searching there.
The leftmost and the biggest column contains the particular entity's details. Another thing I like in the application is that once you edit anything you're provided with rich editor and able to format the text with the broad scope of formatting controls

Requirements and Test-cases

The approach of binding requirements and test-cases that is used in SquashTM (or Squash TM) doesn't look very comfortable to me. It is supposed that you create set of the requirements and set of the test-cases which are not related to each other originally. Those sets might be of a huge number of items. Then you may do one of the following
  • either go through each requirement and bind it (every one) to one ore more test-cases
  • or go through each test-case and bind it to one ore more requirements
This seems not introducing big problems until you have big project with a lot of requirements and a lot of test cases and very complex requirement hierarchy.
The much more convenient way would be creating the test-cases straight from the requirement which is not possible in the current version. Thus having the requirement created (not necessarily by the current user - probably by another one) the dedicated tester could cover it with the set of appropriate test-cases and, say, mark it somehow as covered.

Composing test-cases

Unlike the requirements the test-cases are not just the free-text but some complex entity. Each test-case has some general information containing:
  • Name
  • Description
  • Weight: Very High, High, Medium, Low
  • Nature: Undefined, Functional, Business, User, Non Functional, Performance, Security, ATDD (No idea about what it could mean :) )
  • Type: Undefined, Compliance, Correction, Evolution, Regression, End-to-End, Partner
  • Status: Work in progress, Under review, Approved, Obsolete, To be updated (not sure if there is some BPM engine embedded to drive these states automatically)

This is how the general view of the test-case looks like:

SquashTM test cases

All the properties above have nothing to do with the most valuable part of any test-cases - test steps. The steps should be populated at the dedicated "Script" tab. Each step consists of the list of the actions and the expected result. 
You also can assign the particular step to one of the test-case requirements so that it implies that step verifies one. However I couldn't manage to make it work. First strange thing is that when you're managing the requirement set verified by the particular step you're requested to remove unnecessary ones from the entire list of the test-case requirements. IMHO the more obvious way is to mark ones I'd like to include into the step coverage. Anyway.. the second strange thing is that the requirement gets selected if only to click the cell with requirement number (1,2,3...). I would expect the entry is selected wherever I click in the row... The last but the main problem is that despite I choose the requirement and click "Remove from step" button it still remains in the list and not removed.

Here is the step management view:

SquashTM test cases

In addition you can attach files to the test-case or to the particular step of the script. Here I experienced the problem as well as the application rejected my attempt to attach png file to the test-case as the one of not supported formats.

 Campaigns

The test-execution mechanism is supported with campaign concept. The manager can create a campaign - the aggregation for the set of test-cases. Each test-case can be assigned to some user. The campaign should be broken down by the iteration - the particular test run. The iteration may contain the subset of the entire campaign test-case set. If the test-case appears in several iterations of a campaign and gets the certain status in one of the iterations then it has the same status in other ones. This should look like this:

SquashTM campaigns

Once the tester which is assigned with the tests sees the iteration ready in their campaign workspace they may start it like it is shown on the screen-shot:

SquashTM test execution

Clicking "Begin" button takes the user to the steps of the test-case until the steps are over. Once they are the user moves to the next test-case of the iteration automatically until there are no more test-cases in the scope.

Reporting

Reporting capabilities of the application leaves ambiguous impression. Officially the tool has quite functional reporting. You may generate the report to analyze your requirements and test cases regardless whether you performed any test execution or not. Each report you may export to PDF format to present the results to your management for example.
Its also possible to generate the report for certain executions (aka campaigns). That's all great however I met the problems again. When I tried to filter the report data by my test project only, the filter couldn't ender the project set in the filter dialog so I had to be content with the all the existing projects in my report.

Here are several examples of the reports I managed to generate:

SquashTM test reports

SquashTM test reports

SquashTM test reports

Conclusion

I'm confused and it is quite hard to give the resolution on the tool. From the one hand from functional and usability standpoints it is the best free test-case management systems I worked with. From another hand it is still quite raw and buggy. Here are some problems I found which I couldn't assign to the topics above and which are mostly related to the user management functionality:
  • Whatever profile I assigned to a user I always saw Advanced Tester one when I came back to the corresponding dialog. However when I log in as that user I see the actually assigned profile for them
  • When I compose the team I see the users I recently deleted and able to get to their details by drilling from the team configuration dialog
  • Annoying problem is that the trees in the workspaces are not always updated in time when you add some sub-entities there
Well. I think despite there are a lot of quite annoying problems in the product they are all eliminated with the fact this tool is freeware and provides good functional scope and nice usability. Yet it fits small and middle-size development teams but the suject has all the chances to become domain standard in the observable future. So I give this TCMS 8.5 points of totally 10. I would give more, but those annoying bugs...

Tuesday, May 14, 2013

Creating awesome and simple interactive checklist in MS Excel in three steps

What we need from our checklist

Looking through some solutions describing the check-list creation I realized that a lot of the cases require from me being aware of VBA or some components paradigm for inserting the check-boxes into the cells and make them impact the check-list items in such or another way. That frustrated me so I'm presenting some alternative solution here which will not take a lot of time to create your own and awesome-looking checklist. So what the people usually need from the check-list:
  • Mandatory
    • Check-boxes which are easy to fit into the cell
    • Check-boxes which can visually take two states: checked and un-checked
  • Nice-to-have
    • Check-box states affect the look-n-feel of some other parts of the check-list
    • Scalable structure so we're capable to add the items with no need to configure new rows

What we're going to do

The checklist we're doing will consist of the simple table structure where one of the fields is intended to hold the check-box. Changing that check-box'es state causes the entire row shading or unshading (depending on whether it is checked or not)
The check-box will not require any scripting so you do not need to know VBA.

Nice.. Implementing skeleton

The first and the simplest thing we need to do is to design the fields which our steps in the check-list will consist of. For the presentation lets limit the field set with five ones, where the last field will be holding the check-box. That should look like this:
spreadsheet skeleton without check-boxes yet

Good. Implementing check-boxes

We need some content to make the example more vivid, right? And of-cause we need what we're reading this post for - the check-boxes. So let's do that. Say, we're going to get into the room and we know how to do that in the most effective way. Thus, we'd like to describe the step sequence not to confuse the order when we'd like to do that again. See the example of what we got.
excel checklist with check-boxes
Okay, so.. Howe we managed to do this? The heading row is just colored with blue - not a rocket science. Lets make the check-boxes. To make the check-boxes select the column area under "IsDone" header. Right-click it and choose "Format cells" menu item. Configure the font like it's shown on the screen-shot:
Inserting checkboxes into excel checklist

Align the text in the cells so that the check-boxes are in the center of the cell. That is it. Now when you set the focus to the check-box cell and press "x" key the check-box gets checked. If you press "o" key the check-box gets unchecked. Simple and visually acceptable solution, isn't it?

Awesome! Implementing items highlighting

Let's now add some flavor to our checklist. I'd like my checklist shade the row when I set the check-box checked. I'm going to reach this goal using the conditional formatting. Let's follow the steps (the steps are applicable to the example. you should slightly reconsider the formula according to your check-box relational position in your checklist):
  1. On the home tab click "Conditional Formatting" and choose "Manage rules" menu item
  2. Click "New rule..."
  3. In "Edit Formatting Rule" dialog select "Use a formula to determine which cells to format"
  4. In "Format values where this formula is true" type =($E2="x") as in the example the first check-box resides in the cell E2
  5. Click "Format..." button. Switch to "Fill" tab
  6. Choose the color you'd like to shadow your checked rows. Click OK
  7. Click OK on "Edit Formatting Rule" dialog
  8. You should see your new rule added. Now we should specify the area that rule covers (see the screen-shot for the example). 
shadow the checked rows in excel checklist

Now we have the awesome check-list. You might want to have the row (aka steps) grouping in your checklist. Visit this tutorial to get aware of how to implement row grouping.

Aaaand finally... you should get something like this:

awesome checlist in excel in three steps

Tuesday, March 05, 2013

Zephyr Desktop test management system review

Introduction

This post originates the series of reviews of test-case management systems. All the tools are available for free (some with restricted functionality). The first one is Zephyr Desktop. This tool asserts supporting set of common TCMS use-cases such as manage requirements, test-cases and executions. It also features some stuff that allows to integrate your TCMS system with test-case automation solution.
See also SquashTM test case management system review.

Web-site

Nice web-site. It is easy to navigate and to find required distribution. However it does not differ much from the competitors. Free version is available after few of registration steps here.

Installation

Zephyr Desktop Community Edition is supplied with built-in MySQL server and Apache Tomcat 5.5 servlet container. The installation process is straightforward and should not make a problem for anyone. After you've installed the system you get two new services running: Zephyr server and MySQL server with operational database. Of course you have to ensure all the ports you're going to use are not already used by some other processes.

Features

The free version I've tried offers a set of common-qa-practice use cases. However it limits the number of available project by one. This is not so bad as you're free to work-around such the problem by considering the release term as product-release term (later I'm explaining what all this stuff means). So, the application allows to:
  • Manage the resources (add, remove or edit users)
  • Specify requirements, structure them and map to the test-cases
  • Break-down test-cases by purpose (like regression, user acceptance, etc)
  • Schedule TC execution
  • Analyze project statistics
  • Integrate with bug-tracking system (not touched)
  • Execute external processes (not touched)
  • Interact with external tools via API (not touched)

Usability

So, what's the user flow as it is seen for me: Some test manager logs in and configure the project. They scope available resources and releases and probably fills the project with release-specific requirements. The tester covers the requirements with test-cases. They break-down the test-cases by their purpose. The test manager schedules the test execution for particular test-set type (e.g. regression testing). Tester passes the test cases. They set the status for each one so that once the execution done we have the set distinguished by PASSED/FAILED so we have the statistics to be analyzed.

Look&Feel

Graphical framework

UI looks a bit heavy  and is built on Flex technology. It implements dialog model. It works fast but sometimes it looks overdriven so you can get lost in the numerous dialogs like this
zephyr desktop community edition ui layout

That is a good solution for desktop applications as we have a sort of task-bar in windows or linux-GUI along with capability of switching between the windows using short-cuts (aka hot-keys). However the same approach in Web UI doesn't look so organic.
The frameworks allows drag'n'drop items at some places. It is generally used in tree structures to move items from one folder to another. However it does not allow to move folders to subfolders. However such the feature would look quite useful for me.

Releases

As I said before we're not able to manage several projects in free version of Zephyr Desktop. However the tool offers the release management feature which separate the workspace into several isolated areas. This looks like the picture below:
zephyr desktop community edition release tabs
Once you select particular release tab you're able to manage requirements, test cases and other stuff in isolation from other releases. Thus, you can establish the convention saying that release is not actually just a release but PRODUCT-RELEASE. Hence you may manage several projects in one.

Requirements

The tool allows managing the requirements. This is great as (isn't is strange?) not all the TCMSs have requirements as the basic entity in the structure. However the this area has a bit of drawbacks as well. Say, why not to allow the user creating test-cases straight from the requirement? Does that look okay when you have hundreds of requirements and then should create thousands of test-cases and only after all finally bind them to each other? Anyway this way is how all happens in Zephyr.

Test-case management

All the test-cases may exist either being bound to certain requirement or separately. Application offers useful feature called "Repository". The repository allows breaking the test-cases down by the purpose you'd like them to target. You may want to include the test-case in, say, user acceptance set along with regression set. This is what repository will help you to achieve. However I couldn't manage to map the test-case to several repos at once, so I had to clone them to move separated items each to their dedicated repository folder.
Test cases consist of the steps. Each step can be described in comprehensive way including attaching documents as files or specifying URLs for those resources. You also can keep the release-wide documents uploaded so they are visible from everywhere.
zephyr desktop community edition managing test-cases

Another feature that I like is test-case change-history so you can track what changes have been made to the test-case since its creation.
Test cases (as actually most of the entities in the system) are stored in a sort of folder structure so you're free to order your stuff in the way you like. However I would expect I see (or have the option to see) all the items  in selected folder and its sub-folders. Now if you want to observe all the tests you have to bypass all the tree-folders manually.

Scheduling

This function seems to be the only one I couldn't cavil at. It allows scheduling the execution of your tests breaking it down by phases and sub-phases with responsible resources and timing boundary. It should look like this:
zephyr desktop community edition scheduling test execution

Reporting

After you (or your guys) have executed all the tests you've been scheduled for, you may check the statistics of your project. In free version it looks quite poor. It delivers some simple stat of what part of your tests failed and what didn't, which part of requirements is covered with the test-cases, etc. It represents as the release-specific statistics as the project-wide one. All that looks like this:
zephyr desktop community edition statistics

zephyr desktop community edition statistics

zephyr desktop community edition statistics
You may also observe the main dashboard but this has not a lot of sense in free version where you're not able to manage several projects.

Not covered features

As I mentioned before I had no chance to play with a few of features in Zephyr. However when I tried to integrate the tool with JIRA it warned me of I will not be able to change my choice in the future and this decision should be final. Doesnt look pretty flexible for me :)

Conclusion

Anyway despite a set of drawbacks found in the tool I find this TCMs pretty fitting the needs of the ordinary QA team of 5-10 persons which manages not very large test sets (say 300-400 test-cases against 50-100 requirements) and which does not plan to change bug-tracking system :)
This tool will also be found comfortable for the teams which do not spend time for deployment and configuring huge and complex systems and want to start managing the tests as soon as possible.
I give this TCMSs 7.0 points of totally 10.0.

Wednesday, February 13, 2013

Semi-automated regression testing

Automated regression testing is pretty much expensive stuff, but what to do if you have few of resources but your you're in strong need to have some fast way to assess if your new build got affected with regression? You're lucky enough in case your application produces some standardized output like table-data in csv file format. 

The approach may look like this:
  1. Define the reliable output of the application version you're considering the stable one
  2. Keep that data as golden-set
  3. Each time you've got the new dev-build scope your new piece of data to make it relevant to your golden set
  4. Use some comparison tool to detect if your new piece of data has the differences against your golden-set
  5. Analyze the root-cause of the differences in case they are
Pro:
  • You save the resources from wasting
  • You know about the new problem in case your application has produced improper data but you have proper one in your golden set
  • You know about the old problem in case your application has produced proper data but you have improper one in your golden set
Con:
  • You do not know about the problem in case you've got improper data in both the sets