Application Testing (based on Variable Data Printing applications)

By Kirill Lebedev, DataArt

The idea of VDP is to create templates for various documents such as advertisements, booklets and brochures, which can be later filled with various content (text or images) and printed on demand. This technology allows to significantly speed up the process of creating printed materials, reducing efforts and the time spent on its development.

As a result of this research, I came up with a rough description of the testing process and worked out the basis of testing VDP applications. Its general ideas are presented below.

Testing Workflow

Most importantly, a testing methodology should be put in place. Testing is often performed unsystematically, which significantly reduces its effectiveness. Everything depends on the knowledge of a particular tester and testing quality becomes hard to control. The tester might find obvious errors, but one cannot be sure if he will determine less significant ones. A more qualified tester has a test plan and a set of things in need of checking, based on his experience. It is a good idea to share this plan with less experienced and qualified testers, and it's not hard to achieve.

All the typical errors for a certain application (VDP for instance) can be registered and organized systematically so that they can be reused later. With this system, initial testing can be reduced to checking a number of standard errors, which can be done by less-qualified testers. If a typical error occurs, there is no need for explaining its reason - a tester simply has to send a message to the developer - "test number #N has failed". After that, the developer opens the test description and sees all the appropriate details. All of this makes testing faster and much more effective. Besides, it allows employing less qualified testers for the major part of the testing process without reducing the quality of testing.

Another important idea for testing of sophisticated applications is practicing a multilevel approach. This means the testing process can be split into several phases or levels. For a VDP application with a Web interface, the first and simplest level is a spell check and a correction of grammatical errors on the page. In addition, some symbol errors can be checked, for instance assuring the usage of © and not (n).

The second level is examining the arrangement of controls. They should all be placed according to the corresponding standards and specifications. It is also recommended to test the general usability. For example, if you are using a TAB to switch between the buttons or to edit boxes, they should work consistently, without switching from the top of the page to the bottom. Or if the edit boxes are intended for entering large texts, they should be big enough, so that the user is not faced with a prospect of entering 20 pages of text into a couple of lines of the edit box.

The third level can be called controls interaction. For example, when an administrator deletes a user from the user list, he should be marked off from all the lists and pages. All displayed information must be refreshed. And so on.

Therefore, these two approaches (errors registry and dividing testing into several phases) give us a convenient and very effective testing plan. Any registered mistake will be found with a 100% assurance, and any tester can determine it easily. This allows using more qualified testers for more complicated tasks. After the initial testing is complete, the qualified testers start looking for less common errors, adding newly found errors to the list of the existing ones. Thus, the qualified testers do not have to spend time on routine tasks.

Creating Tests (based on PDF Compiler)

When a test plan exists, the testing process is easy and does not require much work. However, sometimes one has to test new applications without the appropriate experience.

One way to test an application is to simply try it, according to your own vision of how it should work and to check for things you feel need to be checked. Unfortunately, this approach gives extremely relative results which depend on many factors.

Another approach is to work out a testing plan before starting the actual tests. This makes the proess more technology driven. For some applications it is a rather challenging task, that's why this approach is not used as often as it could be.

Let's consider this issue for the testing of VDP applications. One of the applications we worked on consisted of three main parts - a Web-interface, a plug-in for Adobe InDesign (where templates are created) and a PDF compiler that inserts the variable text and images into the templates compiling a final document. Below is an outline for creating a test plan for the PDF compiler.

The first problem a tester encounters is the multiplicity of factors. The documents include text and images with many parameters and each of them may take dozens of variables - text font, color, emphasis, paragraph alignment. Text variables can be words, sentences, paragraphs, lists, etc. All these parameters are processed by a compiler, which means they all must be taken into consideration when testing the compiler.

Our goal is to test these variables. The task may seem frustrating at a first glance as there are too many things to check. To make the task less complicated it is useful to break the testing into several levels. This allows tosee many parameters and factors from one level to another. The first level testing might include testing text variables. At first, we check that the text variables of any type are compiled correctly. We should check if the compiler works properly for words (including very long ones), sentences, paragraphs, etc. To achieve this, we need no more than 20 tests. Below is a table showing a few tests of this kind:

Then the text formatting should be checked. This is the next level of testing and it is much more complicated as we have quite a lot of parameters. We have to check various fonts, font sizes, font colors and emphasis. As these parameters take many different values there is almost an infinite number of combinations and we cannot check all of them. The following approach can be used here.

At first, we take only two parameters - font and font style. It's good to check all the fonts that should be supported, but let us assume we have only a few of them - let's say Times New Roman, Arial and Courier. Then we have to pick font sizes to be checked. We can't check all of them as there are too many, so we can pick them from some range - for example 6 to 72 with some interval. Besides, we have to check all the frequently used ones (usually from 6 to16) and the ones at the bounds (the biggest and the smallest). All the chosen parameters can be conveniently written down in the form of a table containing text parameters and a text sample with these parameters.

Then we want to add the next parameter - emphasis. So, we create the tables as the one described above for each emphasis - regular, bold, italic, italic bold - with corresponding samples.

After we have tested all the samples in the tables, we have to add the last parameter - color. To do that, we pick some of the values from the existing tables and 'give' them a color. We do not have to check all the combinations, but it is essential to test the most frequent ones, the boundary ones and some of the rare combinations.

The same kind of tests can be worked out for other types of variable, like images for example.

Advantages of this approach are obvious:

• It makes testing more consistent and ensures that nothing is accidentally overlooked.

• It provides a test plan that can be re-used anytime for applications of the same kind. This means it can save you a lot of time and effort.

• After a test plan is created you can employ less experienced testers to do routine tasks and allocated experts to more complicated tasks, assuring the quality of testing.

Translated by Eugene Sokhransky