An approach to the acceptance test of AUTOSAR

Abstract

Referenced to the ferry test method, a new approach is proposed to check the invoke state of the appointed application interfaces (APIs) of software package, which is an important test item for the acceptance test of AUTOSAR. The core factor of this approach is the point of control and observation (PCO), which is injected into the compiled executable binary data directly without changing only source code of the application project. Each time the program runs to the PCO, the CPU is took over by a specially designed program named supervisor. Then the supervisor works as a ferry to translate data and commands between the test tools and the system under test. After that, the supervisor returns the CPU to the application program and waits for the next trigger. At the end of the paper, a test case is selected to verify the validation of the approach.

Keywords

Acceptance test AUTOSAR ferry test method PCO supervisor

1. Introduction

Monitoring the context of the internal application interfaces (APIs) is an important test item for the acceptance test of software package conforms to Automotive Open System Architecture (AUTOSAR) [1, 2]. The context of API contains timestamp, variable values and the stack information is important for analysing the performance of the system under test. So an approach that can monitor the invoke state of the internal APIs is needed.

Most of the vehicle electronic control units (ECUs) have not strong process ability and full functional operation system, which makes it impossible to test the program as flexible as it does on a personal computer. The normal solution for monitoring the internal API of ECU is adding instrumentation codes into the source code. Before the API is invoked, the context of the APIs is printed out by the instrumentation points [3]. An instrumentation method based on sequence block is brought forward to conduct statement coverage testing, which can effectively reduce the number of stubs, so as to reduce the impact of stubs on the testing process [4]. There are some problems in this solution: excess codes are imported into the software package, which destroys the integrity of the code package; the whole software project should be rebuild, and the instrumentation codes must be added or removed for each time, which brings more extra work to developers; exceptions are usually required to be injected into the system under test (SUT) dynamically during the test [5], which is hard to be realized in this solution.

A new test approach referenced to the ferry test approach [6] is proposed in this paper. This new approach injects the points of control and observation (PCO) into the compiled binary data directly without modifying sources codes. A data channel is established on the embedded system under test (SUT) by the PCOs, the value of variables, timestamp, context information of the APIs are able to be read out through the channel and the external test commands also are able to be sent into the SUT through the channel. Compared with the normal solution, the source code needn’t to be modified and rebuilt any more, which reduces workload of the testers and protects the integrity of the source code; dynamic test commands such as inject exception are able to be executed during the test, which makes it possible to complete more complex testing even remote test [8]. At last, a test case of the acceptance test of “Unified Diagnostic Services (UDS) conforms to AUTOSAR specification” [10] is selected to verify this new approach.

Figure 1.

Structure of ferry test approach.

2. Overview of the test method

The ferry test approach is widely used in the testing of the communication protocol, the core idea of this approach is to separate the complex testing process and the adapter of the service application interfaces [9]. The data generated during the test are returned to the test system and performed complex processing, at the mean time the test system send the test commands to the SUT [12]. The interaction between the test tools and the SUT is similar to the person cross the river with the help of the ferry. The structure of the ferry test is shown in Fig. 1.

According to Fig. 1, the data is moved to the test system with the help of the ferry channel and the test processes are executed in the test system. The integrity of the IUT is ensured without any test logic being executed in the system under test.

Referenced to the ferry approach, a supervisor is created, besides the application program under test, the test channels can be established by the supervisor. The context of the APIs is able to be monitored through these channels. The structure is shown in Fig. 2.

Supervisor is a special designed segment of executable program independent with application program. It is necessary to establish the test channels. Supervisor has two work modes: configuration mode and monitoring mode. Working under the configuration mode, the supervisor processes the command from PC, saves the context of the target position of point and then injects the PCOs to the binary data of the application program. Working under monitoring mode, supervisor is invoked when the application program runs to the PCO, then transmits the context data of the API to PC and executes the test commands from PC. After all these works, the supervisor hands over the CPU to the application under test and waits to be invoked the next time.

The configuration software on PC is used to extract the map file of the binary data and provides a visual window to the user. This window contains a name list of APIs and the variables, which is used to add or remove the PCOs.

Figure 2.

Structure of the test system based on supervisor channel.

Figure 3.

PCO work process.

3. Implement of supervisor

Point of control and observation is the core of supervisor. According to Chapter 2, the theory of the PCO is shown in Fig. 3.

To realize the PCO as in Fig. 3, the following 3 problems shall be solved first.

•
How to trigger the program counter (PC register of the CPU) jumps to the supervisor when application program runs to the PCO.
•
How to add PCO and save opcodes which are modified by the PCO.
•
How to restore the opcodes and return to the application program from supervisor.

3.1 Trigger mechanism of the PCO

Interrupt is one of the basic functions of MCU. If there are events or exceptions happening, the MCU will save the context of the program and then jump to the interrupt service routine [7]. Referring to the interrupt mechanism, the PCO can be realized by replacing the corresponding application opcode with software interrupt trigger opcodes. At the same time, the supervisor function address is registered to the SWI interrupt vector. Each time the application program runs to position of the PCO, the SWI opcode will be executed to make an interrupt deliberately. The program counter will jump to the SWI interrupt function and the supervisor is invoked automatically.

Figure 4.

Progress after power on.

Figure 5.

Workflow of the supervisor under configuration mode.

3.2 The workflow of the MCU with supervisor after power on

There are two different mode of the MCU with supervisor: configuration mode and monitoring mode. Configuration mode is used to add or remove the PCO and some other parameters for testing; monitoring mode is used to execute the application program and monitor the result received from the supervisor. After power on, the supervisor starts first and waiting for 500 ms to confirm whether request of configuration mode is received from PC. If received, the MCU stays in the supervisor and runs under configuration mode; otherwise jumps to the application program and runs under monitoring mode. The workflow is shown in Fig. 4.

Figure 6.

The steps to add a PCO.

3.3 Supervisor under configuration mode

When working under configuration mode, the supervisor implements the commands from PC. Four types of commands are defined: Add/Remove PCO; Add/Remove monitoring variable. The work flow is shown in Fig. 5.

Adding monitoring is to replace the normal opcode with SWI instructs. It will change the binary data of the application program. So it needs to save the opcode to somewhere, which will be restored to be executed after the SWI interrupt. In this paper, a ram area named OpcodeDict is used to store the opcode from the PCO. The struct of the record is:

typedef struct

{

UINT8 valid;

UINT32 Address;

UINT8 opcode [1];

}TOpcodeRecord;

And the steps how to add a PCO is listed in Fig. 6.

Removing the PCO is the reverse operation of adding: rewrite the opcode to the PFlash and set the corresponding item in the OpcodeDict to invalid.

Adding/removing monitoring variables is to save/remove variables in an appointed area in the RAM, which is named as VarDict. The type of variables is defined as:

typedef struct

{

UINT8 id;

UINT8 DataType;

UINT32 Address;

}TVarRecord;

When MCU runs into the SWI routine, the supervisor reads the values of the variables listed in the VarDict, and then send to the PC.

Figure 7.

Normal solution to restore the PCO.

Figure 8.

Memory map of supervisor and application binary data.

3.4 Restore opcode and return to the application program

After executing the testing commands from PC, the CPU needs to restore and implement the opcode. The first step is to read out the return address of the interrupt routine from stack, which is also the address of the PCO. The second step is to use this address as key to read out the opcode from the opcode dictionary and write back to the PCO address. Now the binary data of the PCO is restored. After that, set the trap flag of the MCU, which trigger an interrupt routine when each opcode is executed. The trap interrupt routine is used to reset the PCO again. The workflow is shown in Fig. 7.

3.5 The memory map of supervisor and application binary data

Application and supervisor are existed in the SUT, which needs to assign a special Flash and RAM area for supervisor [9]. The flash area is used to save the binary code of the supervisor and the ram area is used to save the opcode dictionary and variables dictionary. The size of the opcode and variable dictionary depends on the number of the PCOs are needed at the same time. The memory map of supervisor and application are shown in Fig. 8.

4. The software on PC

Three PC software are required to coordinate with the supervisor in the SUT: hex combination software, PCO configuration software and testing component.

4.1 Combiner of the binary files

Supervisor is an independent module with the application program. The two parts need to be combined to one binary file before downloaded to the MCU.

Combination of the two modules is to read out the binary bytes from the files and write these bytes to one file. Two classes named “filestream” and “binaryreader” in C# language are used to program this combiner software.

4.2 PCO configuration software

To add PCOs in the flash of the MCU, the key step is to get the address of the APIs and variables. So there should be a solution to get the function and variable list in the application binary data [13]. The MCU with CPU12S core of NXP is used as an example to illustrate how to design the configuration software.

After compiling the source code and linking the object files, a memory map file will be created. This file contains the details of all the procedures and variables as Table 1.

Table 1
Items in the map file

Item	Description	Sample
Name	Name of the variables or procedures in the application program	“Function1”
Address	Address of the variables and	0xFF8019
HSize	Size of the variable or procedure(Hexadecimal type)	0x26
DSize	Size of the variable or procedure(Decimal type)	38
Ref	Referred times	2
Section	Where the variables or procedures located	.text

Figure 9.

Combination binary data and download to MCU.

A simple text analyzer is built to get the procedure and variable list. The main steps are as follows:

•

Create filestream of the map file, and move to the start of the object allocated section by search with the key string “OBJECT-ALLOCATION SECTION”.

•

Then read the string to a new string named ObjectInfoStr until the next section.

•

Then use “ $\backslash$ r $\backslash$ n” label to split ObjectNewStr into string list, which is named as ObejctInfoStrList. The next step is to analysis each item in ObjectInfoStrList.

•

Use 0x20 to split every item in ObjectInfoStrList into substring arrays. If the number of members of the array is 6, it means this array contains the information like Table 1.

•

Then read out all the information of the procedure and variables and displayed in a tree structure.

The data structure of each sub node of the tree is as:

Struct TNodeStruct

{

Bool selected;

UINT8 procOrVar;

UINT32 address;

UINT8 varDataType;

};

The varDataType is useful only the procOrVar is set to 1 as variable type. User can select or deselect the nodes of the tree. The selection of the tree nodes will be translated to adding/removing PCO or variables in the MCU.

Table 2

Command set

Name	Value	Description
CMD_NO	0x00	No define
CMD_AddMP	0x01	Add PCO
CMD_RemMP	0x02	Remove PCO
CMD_AddVar	0x03	Add variables
CMD_RemVar	0x04	Remove variables
CMD_ChangeVar	0x05	Change value at appointed address

Figure 10.

Structure of the testing component.

4.3 Testing component

There should be a bridge between the PC and the supervisor of the MCU. This paper named it as testing component. The structure of this component is shown in Fig. 10.

Testing component is supplied to the upper application as a dynamic library. It contains 2 basic function interface: send_CMD and register_data_callback. And it contains a command set, which defines all the command supported by the supervisor in the MCU. The CMD set in this paper is listed in Table 2.

5. Verification

CAN/LIN bus is widely used in vehicle as the communication network. Based on this network, the protocol of “Unified Diagnostic Services (UDS) conforms to AUTOSAR specification” is designed to diagnostic the state of the whole vehicle, which is important for the safety and security [15]. The acceptance test of this protocol is selected to verify the validation of the new approach. One of the test cases is as the following table:

Table 3
One test case of AUTOSAR UDS acceptance test

Main test execution
Test steps		Pass criteria
Step1	[LT $<$ diagnostic $>$ ]Send UDS request “ReadDataByIdentifier” for DID $=$ 0xF0FE to SUT
Step2	[RUN $<$ ConditionCheckRead $>$ ]Wait for invocation	[Run $<$ ConditionCheckRead $>$ ] is called by the RTE (SWS_Dcm_00439)
Step3	[RUN $<$ ConditionCheckRead $>$ ]Return 0 as error code and E_OK as return value
Step4	[RUN $<$ ReadData $>$ ]Wait for invocation	[RUN $<$ ReadData $>$ ] is called by the RTE (SWS_Dcm_00437)

Table 4

Filters of the configuration software

Name	State
All	List all the APIs and variables
Source file name	Only list the APIs and variables in the appointed files
Static	List the static APIs and variables
Address range	List the APIs and variables within the range of the address
.bss	List uninitialized global or static variables
.data	List initialized global or static variables
.constdata	List the constant

Figure 11.

Combiner software.

According to Table 3, the main content is to check whether the APIs are called by the RTE, which is proper to verify the new approach.

After compiling the package of source codes, the executable binary file and memory map file are generated. By the combination software on PC, the application executable file and the supervisor binary file are combined to one binary file. The combiner software is shown in Fig. 11.

The memory map file is loaded into the configuration software, all the APIs and variables are listed in the form. Usually most of the function and variables in a project are not useful for the test, so the filters are provides for the user to select the target items quickly. The detail of the filters is listed in Table 4.

The main form of the configuration software is shown in Fig. 12.

According to Fig. 12, the details of the configuration are as follows:

The filter is configured to list the APIs and variables from the source files of Dcm_DspExternal.c and SenderSWC.c. For APIs, Dcm_ConditionCheckReadHandler, Dcm_ReadDataByIdentifierHandler, Dcm_WriteDataByIdentifierHandler, Rte_DcmGetSeed, Rte_DcmCompareKey [14] are selected to be added PCOs. For variable, DIDdata, UTexcutesequence are selected to be monitored during the test.

The next step is to run the test cases and check the result. A windows form is designed in the paper to display how the APIs are invoked during the test. Analysis the result according to Fig. 13.

Figure 12.

Configurator of the PCOs.

Figure 13.

The invoke state of the APIs under test.

Second step: First step:

Rte_DcmGetSeed is invoked at 5.437 ms, UTexcutesequence $=$ 0 and DID $=$ {0x00, 0x00, 0x00, 0x00}. It indicates the test tools is applying the seed from the SUT, which is the necessary step to get the access right of the SUT.

Second step:

Rte_DcmCompareKey is invoked at 5.511 ms, UTexcutesequence $=$ 1 and DID $=$ {0x00, 0x00, 0x00, 0x00}. It indicates the test tools has received the seed from the SUT and calculated the key with the seed, then sent back the key to the SUT. Now the SUT is checking whether the key is right.

Third step:

Dcm_ConditionCheckReadHa-ndler is called at 5.519 ms, UTexcutesequence $=$ 2 and DID $=$ {0xF0, 0xFE, 0x00, 0x00}. It indicates the test tools has passed the verification of the SUT and sent the command of “read data by id”, now the SUT is checking whether it is ready to read data by data id. The value of the data id is {0xF0, 0xFE}.

Fourth step:

Dcm_ReadDataByIdentifier-Handler is called at 5.520, UTexcutesequence $=$ 3 and DID $=$ {0xF0, 0xFE, 0x00, 0x00}. It indicates the SUT is reading the data corresponding to the data id.

According to the above analysis, the test of UDS command “reading data by id” is finished, SUT passes this test case. This execution of this case shows that the new approach is valid for the acceptance test of the software package conforms to AUTOSAR.

6. Conclusion

In order to design a better approach to do the acceptance test of software package conforms to AUTOSAR, a new approach referenced to the ferry test method is proposed. This paper analyses the theory of the PCOs and supervisor, especially the invoke flow between the supervisor and application program. After the theoretical analysis, a test case of the acceptance of UDS conforms to AUTOSAR is selected to verify this new approach. The test result shows that the approach is effective.

References

Svenningsson

Johansson

Arts

et al., Formal methods based acceptance testing for AUTOSAR exchangeability, SAE International Journal of Passenger Cars – Electronic and Electrical Systems 5(1) (2012), 209–213.

Tan

W.Z.

Kuang

J.S.

Wang

D.Z.

et al., Research on AUTOSAR conformance test, Computer Engineering 38(4) (2012), 43–45.

Sun

H.L.

Wang

Z.M.

and Wang

W.L.

, Instrumentation technology for embedded software statement coverage testing, Journal of Computer Applications 30(10) (2010), 2738–2737.

Fan

and Wang

, An instrumentation method based on sequence block for embedded software white-box testing, Modern Electronics Technique (2014).

Barry

M.M.

Mcevoy

J.C.

Stewart

M.P.

et al., Automated software testing system and method, US, US7373636 (2008).

Zeng

H.X.

Chanson

S.T.

and Smith

B.R.

, On ferry clip approaches in protocol testing, Computer Networks & Isdn Systems 17(2) (1989), 77–88.

and Ning

, Transforming IAP into ISP based on processor mechanism, Microcontrollers & Embedded Systems (2013).

Bauer

Mcintyre

Sherry

et al., An extensible platform for building remote experiment control1, Journal of Computational Methods in Sciences & Engineering 17 (2016), 1–12.

Parakh

N.J.

, Design and implementation of a ferry clip test system, PhD Thesis, University of British Columbia, 1989.

10.

AutoSAR Partners. AUTOSAR_ATS_DiagnosticServicesV1.1. http://www.AutoSAR.org. 2011–10.

11.

Pimentel

J.R.

, An incremental approach to task and message scheduling for AUTOSAR based distributed automotive applications, in: Proc of the 4th International Workshops on Software Engineering for Automotive Systems, IEEE (May 2007).

12.

Kasurinen

Taipale

and Smolander

, Software test automation in practice: Empirical observations, Advances in Software Engineering 2010(4) (2010), 18.

13.

Cox

D.R.

, Analysis of binary data, Springer New York, 2008.

14.

and Luo

, An automated testing method for uds protocol stack of vehicles, International Conference on Measurement, Instrumentation and Automation (2016).

15.

Wen

and He

, Network security situation abnormal detection method based on hypothesis test, Journal of Computational Methods in Sciences & Engineering 16(3) (2016), 505–518.