Debugging of embedded Linux applications on ARM9/ARM11 processors

by Heiko Riessland & Frank Noack , TechOnline India - June 24, 2009

Embedded Linux as an operating system for modern ARM processors? Maybe not such a bad idea? Linux is a multitasking operating system and therefore, each process must be assigned its own process address space. However, this partitioning greatly complicates the debugging of processors and inter-process functionality. So what can be done to tackle this? This article illustrates some possibilities how you can successfully achieve your goal.

Embedded Linux as an operating system for modern ARM processors? Maybe not such a bad idea? Linux is a multitasking operating system and therefore, each process must be assigned its own process address space. However, this partitioning greatly complicates the debugging of processors and inter-process functionality. So what can be done to tackle this? The following article illustrates some possibilities how you can successfully achieve your goal.

Nowadays, two different debug approaches are normally still used with software development on embedded systems. On the one hand is mostly the classic debug tool, with which the behavior of the Linux kernel is examined. This tool helps the developer to detect problems during the boot phase of the operating system, detect faults in self-coded driver modules or display the back trace of kernel routines.

For debugging at application level, generally a special debug server is started on the target system. This server offers an interface to receive commands from the developer for debugging. It usually communicates with the host PC via the Ethernet interface. The operating system is not influenced by the debug session.

Both approaches have some very specific advantages. However, these are cancelled out by diverse disadvantages: The different tools and components, respectively, do not operate synchronously and, worst case can even obstruct each other! It can even happen that the connection of the application debugger to the target system is interrupted, because the operating system was halted by the kernel debugger and the debug server also comes to a halt. Additionally, a change between both debug tools must always be made thus making the debugging awkward and confusing the developer.

In the following text, the technical principles will be made clear in detail and, with the freeze-mode debugging, a method presented, which helps to avoid most of the disadvantages.{pagebreak}Depending on the hardware configuration and requirements specification, up to two communication channels are used for the connection between the host PC and the Linux target system (figure 1). When application debugging, the connection can take place either via Ethernet or the JTAG port. The serial connection is optional and can be used as a second terminal in addition to the JTAG console. However, for the kernel debugging definitely only the JTAG port is a possibility.

Random access to the virtual mapped memory and a resources-sparing access to the "on-chip" debug module can only be implemented via the JTAG port.

Fig 1: Possible debug communication channels to the embedded Linux target To see a bigger version of this graphic click here.

The application debugging (execution-mode debugging)

Let us first of all elaborate on the application or also execution-mode debugging. Here, the target system is permanently in a running state. For the debugging, a modified GDB Server is used, which is started on the target system when the kernel is booted. The server waits for a connection request of the host debugger via the selected communication channel and initiates the debug process after an appropriate request by the host debugger. Either an Ethernet connection or the JTAG port is a possibility here as communication channel (figure 2).

Fig 2: For the connection to the GDB Server, either an Ethernet port or JTAG port can be used {pagebreak}The method of operation of the GDB Server is simply structured. The host debugger initiates the loading of the application, which is to be debugged, by sending this "load request" to the GDB Server. The GDB Server starts a new process and, before the start of the actual application, activates the system call "ptrace" in order to gain control of the application process. This system call ensures that the process is marked as "TRACED" and all signals occurring are forwarded to the parent process.

If now during the debug process any signal is sent to the application, the Linux kernel recognizes - by means of previously set flags - that this process is executing controlled (traced) and initiates the halt of the application. The signal, which was essentially determined for the application, is subsequently sent to the waiting GDB Server. The GDB Server can then process the signal or forward it to the process. With help of this technique, an analysis and debugging also at signal level is possible.

In addition to the debugging of signals, classic methods - such as step-in or step-over of functions, a single-step or the setting of breakpoints - are also possible. The basis for these methods are classic breakpoints; in other words, halting of the application when predetermined program positions are reached. Therefore, the GDB Server inserts at the applicable position in the code a break code, which when executed initiates a "SIGTRAP" signal.

With this signal, it is communicated to the GDB Server that the application was halted and the control is once again assigned to the GDB. Via further requests, the host PC can then, for example, readout the current register contents or retrieve information about the "call stack".

Additional features for embedded applications

In the system described, a modified GDB Server is used. In addition to the basic functions described, this GDB Server offers the user some interesting further options, specifically for the use in embedded systems. Some of the main features are:

* the JTAG interface can be used as communication channel for the application debugging

* application ELF files can be loaded from the host PC in the target system via Windows Share or Samba "Common Interface File System" (CIFS)

* the support of the debugging in "shared libraries" (*.so Files)

* the upload of ELF files via Ethernet or JTAG to the target system {pagebreak}The "remote serial protocol", which was enhanced with several features, is used as communication protocol between the modified GDB Server and the host PC (figure 3).

The protocol thereby consists of three parts: The first part is a header byte at the beginning of the data stream. Following this is the data part, which contains the packet data. Since the probability of a consecutive redundancy in the packet data stream is very high, a compression algorithm was inserted that consolidates equal parts in the data stream.

By means of this method, not only can more information per packet be transferred, but also the transfer of identical successive patterns is prevented. As completion of the actual data, follows 1 Byte as terminating character, followed by a 1 Byte width checksum. The checksum is calculated from the cross total of the payload stream and serves as protection against manipulation or damage. Due to the checksum length and its structure, a reconstruction of the data stream with help of it is not possible.

Fig 3: Construction of the remote serial protocol (simplified representation) To see a bigger version of this graphic click here.

The execution-mode debugging described can always then be meaningfully used when the main attention is paid to the application development. With this constellation for the debugging, a halt of the entire system is definitely not necessary (figure 4).

Fig 4: Linux application debugging (execution-mode) with the Universal Debug Engine (UDE) To see a bigger version of this graphic click here.

Kernel and application debugging (freeze-mode debugging)

In contrast to execution-mode debugging, freeze-mode debugging is mainly applied for debugging of the kernel. The main difference is that the entire system is halted when a break point is reached. This debug method has proven useful if, for example, with kernel oops/OOMs the boot process of the Linux kernel must be examined, the loading or unloading of a driver module fails or the driver has to be checked for faults or performance bottlenecks during operation.

All standard debug functions can be used as tools, such as: the setting of code or data breakpoints, step-in/step-over routines, asynchronous breaks triggered by the user or the read out of the "call stack".

{pagebreak}In addition to the debugging of kernel routines or driver modules by means of break points, the display of information of currently started processes, services or kernel threads is also possible.

In addition to the debugging of kernel routines or driver modules by means of break points, the display of information of currently started processes, services or kernel threads is also possible.

In the overview in figure 5, not only are all process names displayed, but also information such as the current running condition, the dynamic and static priority respectively, process flags, the current assigned processor time (time slice) or the virtual address of the page, which contains the displayed application, are shown. The level of detail can be adjusted.

Fig 5: With freeze-mode debugging, UDE enables the developer to get a quick overview of all main parameters of the processes To see a bigger version of this graphic click here.

Incidentally, by the use of a special JTAG I/O driver, process information can not only be evaluated and analyzed in the halted state, but also in the running operation. This driver can be downloaded as module or integrated as built-in driver in the kernel.

The performance of the target is thereby only slightly influenced. In addition to the displaying of processes, the debugging of applications is also possible. After an application has been started, the debugger automatically recognizes this and halts the system depending on configuration during entry in the "main" routine. From there, a stepping as with execution-mode debugging is possible. A debugging of applications that have already been started is also allowed. The desired application to this is selected from the process list.

Integrated terminal window

Via the SimIO window, which is integrated in the debug tool, a terminal window with a minimum shell (ash) is available during the debug session (figure 6). By means of this shell, for example, applications can be started and kernel module drivers loaded or removed. Debug outputs from driver modules or applications are also displayed in this terminal window. Depending on configuration of the Linux kernel, kernel oops/OOMs or kernel back traces can also be shown in the terminal window. Since the terminal connection with freeze-mode debugging takes place via the JTAG channel, no additional connection to the target system is necessary.

Fig 6: Linux kernel debugging (freeze-mode) with terminal window in the Universal Debug Engine (UDE) To see a bigger version of this graphic click here.

{pagebreak} In addition to the debugging of a Linux kernel that has already been started, with help of the UDE it is also possible to copy a newly built kernel image in the RAM of the target system.

This has the advantage that the debugging of the kernel is possible from the first line, in other words even before initialization of the MMU. The boot parameter "ATAG list" or "kernel command line" can also be comfortably configured via a HTML page that is integrated in the UDE, and copied to the correct position in the target memory (figure 7).

Furthermore, by means of a selection box, it can be decided if the target system loads an "initrd" (initial RAM disk) or via NFS (Network File System) the root file system. If the decision is made for the loading of an "initrd", then with the UDE a minimum "initrd" generated with the build process can be copied direct on the target. The use of external tools can therefore be entirely omitted.

Fig 7: Initialization of the boot parameter of the Linux kernel To see a bigger version of this graphic click here.

Summary

To perform Linux kernel and application debugging on ARM9/ARM11-based embedded systems in parallel and without today's usual risks of deadlock scenarios, a synchronization of the conventional debug types is necessary. However, in order to accomplish this, new innovative tools - such as the Universal Debug Engine (UDE) - are required. The UDE allows a kernel and application debugging via a single JTAG interface and by the integration of both approaches combines their advantages, without the user therefore having to accept the disadvantages of missing synchronization.

References

[1] Linux Kernel Organisation: The Linux Kernel Archives, 2009

[2] Free Software Foundation (FSF): gdb Server Documentation, 2009

Heiko Riessland & Frank Noack are with pls Programmierbare Logik & Systeme GmbH, Technologiepark, 02991 Lauta, Germany. info@pls-mc.com

Comments

blog comments powered by Disqus