In a highly competitive mobile phone market, handset manufacturers and network operators constantly struggle to find additional competitive edge and differentiating elements to persuade consumers to select their brand instead of a competitor's. Of course, many factors influence the end customer purchase decision. It can be price, features, brand name, perceived ease-of-use and robustness, marketing exposure, ratings and recommendations, industrial design of the physical device and so on. Needless to say, a vital part of the success of a device is also related to the user interface (UI), which will directly influence the purchase decisions and satisfaction during use, and indirectly influence pricing as well as media exposure in, for instance, network operator campaigns.

This article covers key components and approaches when building successful mobile device UIs, today and in the years ahead. Even though the focus is mainly mobile phones, most of the underlying mechanisms remain the same for any embedded system with a display and graphical user interface (i.e. automotive, consumer electronics, etc.), although at different maturity levels.

The User Interface as a Strategic Asset
When talking about user interfaces for mobiles, it is impossible not to mention the iPhone. With the introduction of this iconic device, the perception of what a UI in a mobile phone should look like was brought to a whole new level in terms of graphical richness, responsiveness, and innovative input mechanisms. More importantly, UI strategies were suddenly on every executive agenda, and the need for a solid and differentiated UI is no longer questioned. Despite this, very few manufacturers have managed to deliver phones with the same level of compelling UI experiences. Why has this proven to be so difficult, and what are the missing pieces needed to reach an iPhone UI experience, and even go beyond?

Before answering this question, we need to get back to what a UI and user experience (UX) actually refer to. In many contexts, UX is used as a synonym to UI. There are, however, fundamental differences between the two expressions. A UI of a device normally refers to any graphical, auditory, sensory or textual information the applications use in order to communicate with the end user, in combination with the control sequences the end user in turn employs to control the device (e.g. keyboard input, selections with finger or stylus on a touch screen, voice commands, gestures, and so on).

UX on the other hand is a term used to describe a complex ecosystem (See Figure 1) with many stakeholders and actors. UX spans across all aspects of the interactions between a user and a product or service, including things like experiences of purchasing, unpacking, setting up the product, quality of any collateral material, network coverage, customer support, available applications and services, etc. In other words, it covers the whole range of services and components, whereas the UI is one essential part, since this is what users face every day when interacting with the mobile device.

It is clear that the success of the iPhone is due to the entire user experience, and not just the user interface. Nevertheless, the UI obviously plays a vital role, and essentially two key components are involved - excellent underlying technology (software and hardware), as well as an extremely focused vision and design process executed with perfection. Thus, in order for any UI to be successful, the vision and the work process (i.e. the concept design and the visual design / direction) must be there, and the technology must facilitate the implementation of such a vision. The UI vision is the undisputed law as a fundamental part of the device story telling and brand positioning. Everything else is subordinate. Get these parts right, and it is guaranteed you will be able to create an iconic device (at least in terms of UI). Ok, makes sense. But what is then required by the technology? What components need to be there, and how does it work?

The UI Architecture
During the past 10 years, we have seen an impressive evolution within the mobile UI space. At the time of the introduction of full graphical displays and later, color screens, UI in mobile was still in its infancy. UI was very much a last-minute cosmetic final layer in products driven by technology where feature was king. The UI design was heavily constrained by software and hardware being tightly coupled together (typical phones from that time included Nokia 7110, Motorola V50, Ericsson T28, etc.).

After this first phase, gradually device UIs started to become a tangible asset, and products were increasingly driven with UI as an important tool for differentiation. Over time, new factors started to have influence over the UI, like network operators and content and service providers. Today third-party developers also add value to the device user experience with the mobile phone essentially transforming to a generic service platform.

To support these growing needs from different stakeholders, a modern user interface system constitutes a complex piece of software. It is often referred to as a UI Toolkit or UI Framework, and it provides re-usable software components and developer tools to significantly ease the creation of user interfaces. Figure 2 shows a simplified overview of how this type of software typically is structured (client side). Coupled with this are normally PC-based tools to simplify the actual authoring of the UI.

Graphics rendering backend
A vital component in all UI systems is the graphics rendering backend. This takes care of low level drawing of primitives such as images, geometry, text glyphs, etc. The rendering backend can be entirely implemented in software, interfacing with a framebuffer. However, in high-end devices, it normally also takes advantage of hardware accelerated graphics functionality provided by the underlying SoC.

The most common APIs used in mobile for this are the Khronos standard APIs, such as OpenGL|ES 1.x or 2.0 or sometimes OpenVG. OpenGL|ES supports acceleration of both 2D and 3D, graphics (with full 3D meshes), and provides full, raster graphics based, control over individual pixels (at a finite resolution). OpenVG on the other hand is targeted for vector graphics with less control over each pixel, but with essentially infinite resolution. It supports scaling, rotations and other transforms as well as animation of shapes. It also has advantages in terms of smaller silicon die area, and more efficient representation of data at higher resolutions.

The key drivers for hardware acceleration of graphics have mainly been the display evolution with higher resolutions like WVGA/720p/1080p, in combination with requirement for multiple screen support. Other drivers are requirements around responsiveness and quality (e.g. with the introduction of touch input), 3D UIs and media acceleration. It is, however, not without concerns that these technologies are integrated--increased BOM, power consumption and complexity (system architecture, organization and competence, to name a few.

Some hardware systems may provide simplified support for hardware acceleration, e.g. only alpha blending (transparency) and accelerated blitting process without support for 3D. This is common in industries such as automotive and consumer electronics.

Window manager and composition engine
Coupled to the graphics rendering backend is usually a window manager. The window manager enables different parts of the UI components to share the physical display real estate and hides this complexity from the applications (e.g. how one window is partly obscured by another, or when a pop-up dialog appears on top, etc). Rendering graphics and dealing with window management with solid real-time performance are processing- and memory-intensive activities. On top of this, the increasing demand for UIs with animated properties, transitions and effects has made the traditional window manager approach where the applications draw directly to the display obsolete.

To solve this problem, a compositor-based approach to window management is needed (See Figure 4). In short, the compositor approach enables elements of the UI to be treated as layers, which then are composed (i.e. combined) together to produce a final image. The layers may be the target for different application drawing, resulting in the final compositing step being decoupled from application drawing, and in doing this, the bottlenecks are removed. The compositor can easily animate window positions, sizes and add effects such as transparency and drop-shadow with high performance--enabling a much greater UI experience. Components such as images, anti-aliased text and geometry primitives, effects and external sources such as video elements can also be combined in real-time. In order to further improve performance, things like intelligent overdraw minimization typically can be added in the compositor implementation (for software rendering). The composition functionality is an ideal area for hardware acceleration, and lately even standard APIs have been created for this purpose (Khronos OpenWF, recently released).

Controls and widgets
Typically a user interface system will offer a set of reusable, static and dynamic UI building blocks such as labels, lists, input fields, icons, buttons, scroll bars and dialogs. This is commonly referred to as widgets or controls. Each control is usually configurable by setting certain properties, e.g. to define size and position on the screen. Some properties need to be "localized," i.e. adapted to a certain language system. Most significant are the text strings shown in the UI, which changes depending on the consumer's native language. To support localization (sometimes also referred to as internationalization or I18N), a resource management system is normally implemented, which also supports things like theme control of the user interface.