The 4th Workshop on Model-based Testing in Practice (MoTiP) took place in Dallas, Texas, November 27, 2012. This was a half-day event with 6 presentations and a keynote.
Despite of the glorious promises of increasing the quality and decreasing the testing costs of software, MBT has still not been endorsed by the industry on a wide scale. Where are the obstacles?
Several workshop presentations tackled this issue.
Dr. Wolfgang Grieskamp from Google Cloud Division (former Principal Architect at Microsoft and previously a Senior Researcher at Microsoft Research) in his presentation pointed out that there is a high interest in the industry for MBT. MBT theory is mostly consolidated. There are many tools available today.
But what are the reasons that MBT still hasn’t took off?
According to Wolfgang’s statistics from Microsoft, 7 teams out of 10 seemed to drop MBT after the adoption. Why is it so?
Some answers from Wolfgang’s keynote:
1) Adoption often bounds to individuals. As those people leave the team, the MBT will die in the team. We can conclude that there is something very hard in MBT that it survives in organizations only if there are MBT fans who drive it.
2) Paradigm shift does not stick when people are rotating in shipping cycles.
3) Maintenance problems. For example, at some phase people may start to make changes to the generated test cases instead of models…
Are the MBT benefits really as great as is presented by many tool vendors? The biggest model-based testing project to date was executed by Microsoft (2007-2011). Testing 300 of their protocols against the documentation. The overall effort of the project was 250+ person-years. The Microsoft Spec Explorer tool was used in the project and the project was mostly executed in China and India. Not all the tests in the project were generated from the models. 31% of the tests were automated by the same people writing manually the test scripts. Time effort was recorded in both cases that included the whole testing workflow starting from understanding the requirements, designing the test cases, executing and analysis them. According to Wolfgang, model based testing gave just 34-42% effort reduction compared to the traditional test automation. This is very different from the claims of the MBT vendors who promise 5-20x effort reduction. What is the reason of such big differences in efficiencies? The skills of the people involved might explain the differences.
Wolfgang pointed out that the tooling is the most serious issue in MBT adoption. The tools tend to be horrible complex. He listed the requirements for the tools that can succeed on industrial scale:
– test selection via symbolic state space exploration
– ways for engineers to control and visualise exploration and test selection – no push-button technology will do
– a modeling notation which meets requirements of modern programming languages (expressiveness, modularisation, etc.)
– seamless integration into an IDE, code assistance, intellisense, etc.
– being rocket-stable and reliable
All those are mostly engineering problems, not research problems. According to Wolfgang’s prediction it will take another 5-10 years until we are there.
Marijan Dusica from Simula highlighted social aspects in adoption of MBT in industry in her presentation.
– testing is a socio-technical rather than a technical process
– limited resources and commercial pressure for deliveries put less priority on the activities to adopting new technology
– modelling is not as widely accepted by the industry as some might think
Marijan listed practical guidelines for successful application of MBT in practice:
– MBT approach must be sound, well suited to the SUT and scalable to its complexity
– used tools must be efficient and user-friendly, not increasing the complexity of a MBT approach itself
– team adopting the MBT approach must invest in learning the new approach and make a firm commitment to it
Model-Based Testing User Conference 2012 was held from 25th to 27th of September and was a success according to participants. During three days ninety participants from eleven different countries attended the event. In total, six tutorials took place and nineteen presentations were given. In addition, three keynotes were given by Jan Plasberg from Skype, Bob Binder from System Verification Associates and Jeremy Dick.
Specialists of model-based testing also had a great opportunity to exchange experience and socialize in a more informal atmosphere at the social event in Estonian Open Air Museum.
Robert V. Binder, the President of System Verification Associates, who is the keynote speaker at Model-Based Testing User Conference 2012, gave an interview to uTest.com. Among other things, interview also focused on testing mobile applications and the basics of model-based testing. Read more here.
At TTCN-3 User Conference in Bangalore Elvior today released a free TTCN-3 verification service ttcn3online.com.
The 11th Edition of ETSI T3UC-2012: ETSI TTCN-3 User Conference and Model Based Testing workshop will be held in Bangalore, India from the 11th to the 14th of June 2012. TTCN-3 is a widely used testing technology with an over 10 year history of use in many industries and applications. The flexibility and adaptability of TTCN-3 has allowed it’s rapid deployment in the telecommunication domain as well as the automotive, medical, aerospace, transportation, and many other industries. In addition, TTCN-3’s applications have expanded from purely functional conformance testing into load, performance and interoperability testing.Elvior’s TestCast test suite is one of the most popular commercial TTCN-3 test tools on the market. We are happy to promote TTCN-3 technology on the Indian markets and supporting ETSI T3UC-2012 as gold sponsors. Elvior is happy to show you our TTCN-3 test automation suite that allows users to design and execute the tests automatically. Today the test automation industry’s hot domains are LTE, M2M and Internet-of-Things. We hope to see you at the conference this year so that we may demonstrate our solutions to support test engineers in these domains.During T3UC 2012 Elvior is holding the following presentations:Jun 11th, 2012, 11:30-13:00 Introduction to SW Test Automation and TTCN-3, Andrus Lehtmets (Elvior). Jun 12th, 2012, 12:05-12:30 LTE UE Conformance Test Environment, Mirko Naumann (Nomor), Andrus Lehtmets (Elvior)We are looking forward to meeting you in Bangalore. If you have any questions in advance, please contact us at elvior@elvior.com.MORE INFORMATIONTTCN-3 User Conference (T3UC) 2012 scheduleTTCN-3 User Conference (T3UC) 2012 registration page
The Nordic Testing Days are a software testing conference taking place from 04.06 to 05.06 2012 in Tallinn, Estonia. The event is for you if you like to… ..hear unbelievable stories from real life,..share your experiences with others,..hear about not only success stories but also failure stories,..get your hands dirty with tools, methods and bugsTo register simply visit the following page: http://nordictestingdays.eu/?id=10Elvior’s testing expert and CTO Andrus Lehtmets will be offering an introductory workshop on our model-based TTCN-3 Technologies on Monday, the fourth of June at 10:30. The workshop will be based on a practical example. Students will learn:Theoretical background of model based testingBenefits of using TTCN-3 in model based testingHow to build state models from system requirements specificationRole of System Adapter in TTCN-3 test environmentHow to apply model based testing on practical testing taskResults of TTCN-3 model based testing in2 industrial case studies provided by Elvior Comparison of manual TTCN-3 testing and model based black-box testing is given.We encourage everyone interested in the field of model-based testing to attend.
Elvior test automation experts Andrus Lehtmets and Andres Kull gave a two-day workshop on software test automation and TTCN-3 in Bangalore29th-30th March 2012. The workshop was done incooperation with the Edista Testing Institute.The workshop’s target audience included:• Test Engineers• System Engineers and Designers• Project Managers responsible for Testing and DevelopmentThe target of the workshop was to:• Understand TTCN-3 and review the code• Develop TTCN-3 tests on your own• Become aware of TTCN-3 technology and know areas where TTCN-3 is applicable• Track and report defects based on the risk test strategy.• Understand and develop automated testing environments on your own• Execute test cases and analyse results (TTCN-3 logs)What we saw outside of the classroom in BangaloreOur students were extremely diligent to learn TTCN-3Develop your system as simple as possible. Then there is less to test.This manual sewing machine is very reliable. It never requires reset.A hard-working taxi driver tells us about his life.
Nomor and Elvior partnered to offer the first pre-integrated solution for building LTE UE test equipment and testing LTE UE stack software. This completely integrated solution combines the Nomor eNodeB LTE stack with a TTCN-3 interface that turns the Nomor stack into a system adapter for Elvior’s TestCast TTCN-3 tool.
Nomor’s and Elvior’s joint offering allows a quick start to developing a test environment for LTE UE testing. Using TTCN-3 technology to test both – 3GPP 36.521 (RF) and 36.523 standards.
Download leaflet of the solution.
Nomor offered more information about Nomor’s and Elvior’s joint LTE testing solution at MWC 2012. We would like to thank everyone who visited.
. Elvior is proud to host the second ever Model-based Testing User Conference (MBT UC) 2012 from 25. to 27. september 2012 in Tallinn, EstoniaWe aim to continue to delight all the participants and sponsors with the best model-based testing conference in the world. Staying true to the spirit that made the first conference a success For the participants
As last year, this year’s conference will start with a day of tutorials, on the 25th of September and continue with two days of presentations , lasting from the 26th to the 27th of September.
The registration form, with a list of discounted lodging opportunities in the best hotels of Tallinn. will soon be on the conference’s site: http://www.elvior.com/model-based-testing-uc-2012
Call for proposals
We encourage interested parties to submit their proposals as soon as possible by visiting the following page: http://www.elvior.com/model-based-testing-uc-2012/call-for-proposals
Topics of interest include
• Industrial experiences and case studies about applying model-based tests in different application domains
• Success stories and challenges of applying model-based testing for different types of testing
• Deployment of model-based testing
• Evaluations of model-based testing deployments
• Standardization efforts related to model-based testing
• Experiences in training and educating about model-based testing
Important dates
Submission of abstract: 30th April, 2012
Notification of acceptance: 8th June 2012 ,
Submission of final presentations: 7th September 2012Call for sponsors
Those companies and institutions that like to support MBTUC, are invited to become a sponsor for MBTUC, please contact Robert Jakobson at Elvior ( robert.jakobson@elvior.com).
Additional Information
The venue, the programme and other details will be announced step by step on the conference’s website as we reach nearer to the milestones noted above.
We invite everyone to visit the conference’s website at http://www.elvior.com/model-based-testing-uc-2012
or make your voice heard by contacting Robert Jakobson at robert.jakobson@elvior.eefor more questions and comments regarding the conference.
About Tallinn and Estonia
Tallinn was one of the 2011 Capital of Culture for Europe and it has not stopped offering incredible array of cultural events.
Estonia is a North-Eastern European country with a vibrant economy. It is a permanent member of OECD (the organisation of developed countries), the European Union and NATO. From 2011 the local currency is the euro, providing an stable and comfortable basis for all transactions.
After the start of the financial crisis Estonia reoriented it’s economy to an export-based model. The strategy has bared fruit with economic growth reaching over 8% of GDP (y.o.y) by 2011.
Software, telecommunications and IT are one of the fundamental export articles. Thus making this country an ideal host for this conference.
Leading European companies from the automotive, aerospace and rail industries have come together for an ARTEMIS research project called MBAT. MBAT’s goal is to bring model-based analysis and testing to the transportation industry. It is a 3-year collaboration that started in November 2011 and consists of 38 partners from 8 countries (Austria, Denmark, Estonia, France, Germany, Italy, Sweden, UK).
Participating in the project are some of the world’s largest manufacturers including, Daimler and Volvo from the automotive world, EADS, Dassian and Rockwell Collins from the aerospace industry as well as, Alstom, Siemens and Ansaldo from the rail transportation industry. The project also includes eleven research institutions, and ten technology and tool providers (including Elvior).
MBAT will provide European industry with a new leading-edge validation and verification technology that will enable the production of high-quality embedded systems at reduced cost. This will be made possible by a new approach to testing by which model-based testing technologies are combined with static analysis techniques. Specially designed test & analysis models could also be used and re-used as a basis for model-based validation and verification. The MBAT project will work to create a reference technology platform (RTP) that will make it possible to integrate the different model-based analysis and testing tools into a smooth workflow.
Industrial use cases will be provided by the manufactures. Different model-based analysis and testing tools will then be applied to each use case. The RTP will be built to ensure as smooth an integration of the tools as possible. The methods and tools will be evaluated for each use case.
Elvior is one of the tool vendors involved in MBAT. Elvior’s TestCast MBT will be used as one of the model-based test tools in the testing branch of the project.
This project is an opportunity for Elvior to better understand the transportation industry’s model-based testing needs and improving our technology accordingly. Integration of Elvior’s tools into the MBAT RPT gives us the opportunity to integrate our technology into customer’s tool-chains.
12.01.2012
Elvior moves it’s office to a new building at Tallinn Tehnopol Science Park in summer 2012.
Tehnopol Science Park hosts many start-up companies, as well as research institutions like Eliko Technlogy Competence Center, Competence Center for Cancer Research, IT companies Cybernetica, Girf and many others. Tallinn University of Technology, Estonian Information Technology College and headquarters of Skype Technologies are situated nearby.
3rd floor balcony gives excellent possibility for barbeque
CEO is satisfied with lake view from the window.
