TOPIC:

Dear all,

We are working on a TTCN-3 code generator and we have a question
regarding how to synchronize the SUT and the Test executable. For
instance if the first step in the test case is to wait for a message
coming from the SUT, that means the SUT should be started after the
system interface has been defined and mapped. After going through the
standard we could not find any way to start or synchronize the SUT from
the test case. How is that supposed to be done ?

Regards,

Emmanuel Gaudin

--
Emmanuel Gaudin
PragmaDev Founder& CEO
Model Driven Engineering Tools for
Real Time and Embedded Software
tel: +33 1 42 74 15 38
fax: +33 1 42 74 15 58
www.pragmadev.com

Dear Emmanuel,

TTCN-3 is a generic testing language, it does not know your SUT. Stimulating your SUT to start an action depends on your SUT. There are some conventional ways:
- use the TTCN-3 action statement if you are expecting an intervention from the human tester (e.g. raising the handset of a phone)
- connect to your SUT via its management interface (e.g. SMTP, telnet etc.) and send the stimuli via this interface; for this, of course, you need a TTCN-3 port mapped to the management interface of the SUT
- emulate some other equipment connected to an another interface of the SUT using TTCN-3; e.g. to force starting a signalling flow on the NNI of a mobile exchange you can emulate a call from a mobile handy on the UNI of that node.

Of course, there may be several other "tricks", depending on the concrete SUT.

BR, GYorgy

Dear Emmanuel

In simple create the server interface on test system side and client to SUT
side. So your server starts first and waits for the client (SUT) message.
Port mapping can happen along with the cient connection to the server
(TestSystem).

Hope this helps or else let me know the tool you are using for this.

Thanks
Guru Prakash
+91-9900591977

Hi,

May be I should explain a bit further what we are trying to do:

Historically, my company provides a modeling tool based on SDL and
SDL-RT called RTDS. RTDS can generate code for a target out of an SDL or
an SDL-RT model. We implemented TTCN-3 support in our tool in order to
test the SDL and SDL-RT models. So far it was only possible to
co-simulate TTCN-3 against an SDL system on a host. We are now
finalizing our TTCN-3 code generator and we would like to provide our
users a way to run both the SDL system and the TTCN-3 test suites
generated codes in the same debug environment (as we provide graphical
debugging and traces). At this point a user would select both an SDL
system and a TTCN module in our tool and expect the overall generated
code to run by itself. But we ran into the synchronization problem I
tried to summarize in my first e-mail.

Let's take a very basic example of a cash machine. The first thing the
SDL model will do is to send a Display message with the "Enter card"
string as a parameter. The first step of the first test case will check
this message has been received:
sdlPort.receive(EnterCardMessageTemplate). For the test case to catch
the message, the System Interface should be defined, and the ports
should be mapped. But it is actually not possible for a tool to find out
when the test case is ready and when to tell the SUT to start. Only the
tester who wrote the test case knows when his test case is ready.

kr,

Emmanuel

Emmanuel,

How are you executing the TTCN3?

If you are using a commercial TTCN3 tool (compiler/libraries) then
your question is one that is best addressed to the tool vendor.

If, as I suspect, you are compiling or interpreting the TTCN3 within
your own tool then it must perform what you call 'synchronisation' at
the start of each executed testcase. I'm sure more detail will be
forthcoming when we have the answer to the question above :-)

Regards
Richard Spence

Richard,

You suspect right: we are the tool vendor !

kr,

Emmanuel

Emmanuel,

ETSI specifies an execution model for TTCN3 that implies that the SUT
comes into existence with testcase execution and is destroyed at
testcase completion. TRI (ETSI ES 201 873-5) specifies
triExecuteTestCase and triEndTestCase for this purpose. A compliant
tool will call these interfaces to create/destroy the SUT (or attach/
detach in some interpretations).

Strictly, you need to arrange things so that your tool executes the
TTCN3 and testcase execution starts and stops the SUT (the SDL model).
If this style of execution does not suit your environment then you
should perhaps take a look at ETSI ES 202 781 (Configuration and
Deployment). This TTCN3 extension package makes static test
configurations possible.

Good luck,

Richard Spence.

Dear Rich

This is not quite true: certainly, SUT cannot simply be created or destroyed - yes, for some software cases, but no for many other cases.

Nor does TRI says anything about the existence of the SUT - but just about the connections to the SUT - no matter, when the SUT started or ceases to exist.

Coming back to the original question: it would be easiest to have a link into the SDL environment via which the SDL model is parameterized, started, terminated as needed. One can use then use this functionality explicitly in TTCN-3, e.g. via an additional management port or via SUT actions. Or, one uses it within TRI in triExecuteTestcase and triEndTestcase as appropriate.

As an aside: there have been now and then discussions about the need for some functionality to "manage the SUT" via TRI for the sake of setting up or tearing down a test campaign... while there are various workarounds how to link and manage tests in relation to an SUT, the example by Emmanuel could also be solved by some kind of (new) triSUTSet and triSUTReset functions in TRI which are to be called when starting or terminating a test campaign.

Best regards,

Ina

I agree with Ina..

Dear Emmanuel,

this does not change my previous notes. For a moment put aside guard timers for a moment: your test case will wait forever to receive the expected message unless you invoke it to send the message. This invokation you shall include into your SDL model. It shall send some stimuli on another interface than sdlPort or at least it shall be addressed elsewhere within that port. This stimuli shall result either a TTCN-3 action statement to notify the tester to reset the ATM or send a kind of a reset signal to the ATM directly (reset is just an example, you should know in which circumstances your SUT will send "Enter card"). Again, TTCN-3 does not know anything about your SUT, you have to include every event that should happen during the test execution into the code.

BR, GYorgy

Hi,

I believe that you treat the wrong problem. It is exactly the same test-controllability problem that you would face with "real" SUT. How do you trigger an expected behavior from an "real" SUT if the test case is supposed to start by waiting for a message instead of sending a stimulus first?

You *must* figure out how would you solve that real problem, then include the very same trigger mechanism into your SDL(-RT) model and your original problem will disappear.

I know that that my suggestion "complicates" your life in the sense that you face yet another challenging problem instead of the original one, but the benefit would two-fold: you could reliably test either your abstract SDL model or any concrete "hardware" implementation.

Nevertheless, you might have workarounds:
1. The stupid solution #1: use artificial delays and timers. I.e., trigger the SUT somehow from the test case (e.g., via a separate test coordination interface (port) that should be mappable from the beginning), then wait for a given delay before attempting to map the real test port. It is "stupid", since the solution is not robust with respect to the delay interval, i.e., your test case might wait too short time ("test case too fast").
2. The stupid solution #2: try to launch the SUT from within the control part, i.e., before launching the test case. It is "stupid", since the solution is not robust with respect to the delay interval, i.e., your test case might fire up too late and perform the mapping *after* the SUT has already sent the message ("test case too slow").

For a robust solution you *must* remove the dependency on the startup time of either the SUT or the test case proper, i.e., you *must* explicitly perform the synch between the two peers. See my suggestion at the beginning of my mail.

BR,

Gábor Ziegler M.Sc.E.E., dr.univ
Senior Specialist, Generic Technologies Test & Verification
Test Competence Center
Ericsson Hungary Ltd.
Research & Development
Budapest, XI. Irinyi. J. u. 4-20.
www.ericsson.com Office: +36 1 437 7361
Fax: +36 1 439 5176
Mobile: +30 30 219 9805
Email: This email address is being protected from spambots. You need JavaScript enabled to view it.
Ericsson Magyarország Kft., Székhely: Budapest, Laborc u.1. Nyilvántartó cégbíróság: Fővárosi Bíróság. Cégjegyzékszám: 01-09-070937

Hi,

Thanks to all for your replies. I will use Gabor's mail to gather my
comments below.

I believe that you treat the wrong problem. It is exactly the same test-controllability problem that you would face with "real" SUT. How do you trigger an expected behavior from an "real" SUT if the test case is supposed to start by waiting for a message instead of sending a stimulus first?

Yes, this is exactly our problem, how do you tell the SUT or the tester that the test system is ready ?!

You *must* figure out how would you solve that real problem, then include the very same trigger mechanism into your SDL(-RT) model and your original problem will disappear.

Imagine a bunch of PTCs exchanging messages, and mapping or unmapping ports before actually starting the test case... How can a tool figure out when the test is ready ?! Only the guy who wrote the test case knows
for sure.

I know that that my suggestion "complicates" your life in the sense that you face yet another challenging problem instead of the original one, but the benefit would two-fold: you could reliably test either your abstract SDL model or any concrete "hardware" implementation.

This is a very good point because we actually do not have this problem when co-simulating SDL and TTCN-3 together and let me explain why. When co-simulating we get full control on the overall architecture of both the system and the test cases. Because the simulator is the scheduler, we actually run the Test case first and then the SDL system. If the Test case sent a message in the meantime, the simulator will hold it and re-route as soon as the SDL system is executed. This is not possible with generated code on a Linux target for example.

We have had long discussion internally regarding that issue and this is what came out:

It looks like this synchronization problem is raised because the System interface is not static. Each test case can define its own system interface. When running one test case after another, the system interface actually disappears with the ending test case and is created again with the new test case. So what happens if theSUT send messages before the system interface is defined or in between two test cases is undefined. This is very strange because the system interface on a real tester does not change and should actually be defined statically. The test cases could map or unmap to that static interface but the overall system interface does not change. The introduction of a dynamic system interface has introduced a synchronization problem the language does not address in its current version.

Considering the language as it is today, the tester writing the test case should somehow be able to indicate the test case is all set and ready with a standard TTCN-3 statement.

Nevertheless, you might have workarounds:
1. The stupid solution #1: use artificial delays and timers. I.e., trigger the SUT somehow from the test case (e.g., via a separate test coordination interface (port) that should be mappable from the beginning), then wait for a given delay before attempting to map the real test port. It is "stupid", since the solution is not robust with respect to the delay interval, i.e., your test case might wait too short time ("test case too fast").
2. The stupid solution #2: try to launch the SUT from within the control part, i.e., before launching the test case. It is "stupid", since the solution is not robust with respect to the delay interval, i.e., your test case might fire up too late and perform the mapping *after* the SUT has already sent the message ("test case too slow").
Frankly we would not consider these as proper industrial solutions.
For a robust solution you *must* remove the dependency on the startup time of either the SUT or the test case proper, i.e., you *must* explicitly perform the synch between the two peers. See my suggestion at the beginning of my mail.

The action statement solution you mentioned looks like a good workaround for handling this synchronization issue but we see major drawbacks in that solution:

This is a tool specific solution for a language problem.

This implies existing test suites will have to be modified to run.

I don't think that is what a user wants if it is a standard test case.


kr,

Emmanuel