Now, this is not the plot for the latest transformer’s movie although “autobots” is the overall theme. However, robots developed for test automation are to begin with not quite as autonomous and has been designed to actually make the lives of the average tester that much easier. Something that has become increasingly important as test automation is an area that most operators realize has the potential to both increase quality and reduce cost. The goal is to make the STB, Software or TV platform completely foolproof before exposing it to the end user, but also to make it strong and durable enough to deal with everyday wear and tear, and the inevitable updates that the pace of technological development will require.
Already most modern development is done with some degree of test automation. But the important thing to understand is that there are different ways of testing and different levels to perform them at, so the purpose of these automations steps must be clear from the start. A common mistake is to take too much of an engineering approach and only look at how to perform the test, rather than focusing on what you really want to test and how to get the most accurate result within those parameters. How you test is just a means to an end and never a purpose in itself.
One size won’t fit all
Once you have decided what to test and determined what you want derive from doing so, we at Labatus try to use the ISTQB standard when we test or certify new boxes or software, this means that we classify all every steps the test as belonging to one of five categories: module, integration, system, acceptance or user test.
When an approach and a plan of action has been decided upon it is then a matter of choosing the right tools for the job, and which ones that should apply to each level of testing. The starting point and the early phases of test automation should focus on smaller functions, as well as APIs.
These kinds of tests should be done by, or in close relationship with, the development team. It is something that is quite time consuming and resources intensive, but if it is done right it verifies the quality of the core code, shortens time to market and thereby more than makes up for the cost of performing these test.
At Labatus we instead work mainly with test automation for products testing, and specialize on grey and black box testing. We develop our services with a clear focus on the need of the operators and the requirements of the end users. This approach has lead us to divide our test automation into two separate categories, classic STBs and the TV Everywhere.
The Set Up
We currently use automated testing for qualification within two areas, operator specific STBs and CA security. Labatus has chosen to work with Witbe as the equipment provider for automated tests. Witbe was chosen based on the fact that it makes us both more effective and that it allows us to add test cases to get a more complete coverage.
In addition to this we have also designed and built equipment in house, in those cases where we haven’t been able to find a suitable or satisfactory existing solution. Examples of these that we are currently using as the smart card arm, a software for stream players and a smartcard simulator that allow us to perform even more rigorous testing and to certify a standard of quality that we are happy with.
Using the equipment from Witbe together with our own customized solutions means that we can now use a black box approach and test any STBs for both security and function. Currently we are focusing on automating testing for three key areas; endurance, robustness and performance.
One example is where test automation already in place is our work with a Scandinavian operator where Labatus develops and executes test of VOD services and where we can re use tests that will enable us to monitor the VOD services whenever changes occurs in the overall eco system.
The Android, Web and IOS devices are all a little different. We can always monitor the output but the inputs are harder to simulate with today’s black box equipment. In order to get around that we are now taking more of a grey box approach, where we use third part libraries to emulate the input and where we can analyze output and the difference in output.
Where automation still isn’t possible
We are currently of the opinion that there still isn’t any good enough overall solution to do grey or black box testing for TV sets. We have seen trials where cameras that monitor the output have been be used, but since we certify a large number of TV Sets with different sizes and different UIs, we found it very difficult to do this in a way that will scale to a point where all testing is automated.
Today we have more connected device and viewable screens than ever before and both development and consumer behavior is changing at an unprecedented rate. Naturally this means that what we do in testing and development must also reflect this in order to meet the challenges that this poses. We believe that balancing the benefits of intelligent manual testing and automation is the best way to do this, and that test design will become increasingly important.
As CPU power and automated approaches is getting better, automation will cover more and more of the test being done. But with new formats such as 4K coming, the amount of data is increasing, so this then becomes a race where automation will get close but never be able to cover the full user experience.
Peter Paunonen, COO at Labatus