Ambient Intelligence: Vision,research,and life

1. Introduction

The vision of ambient intelligence as conceived within Philips [2] and elaborated by the Information Society and Technology Advisory Group (ISTAG) [15], extended the earlier concept of ubiquitous computing [36] with the ambition to deploy machine intelligence for sensing user activities and context, in order to adapt to them and anticipate user needs, aiming to enhance ease of use, comfort, and enjoyment. In the last fifteen years the Ambient Intelligence vision has played a major part in stimulating European research in information and communication technologies. Inevitably, it has been the subject of reflection and critique which have, in their turn, helped conceptions of Ambient intelligence to evolve and mature. Today Ambient Intelligence is still discussed in future perfect tense, even though many envisaged solutions have materialized in the meantime. The sections that follow elaborate this apparent paradox charting related developments, and identifying shifts in priorities and goal setting of the Ambient intelligence research community. The paper ends by discussing directions the Ambient intelligence research needs to pursue and by identifying challenges spanning the different strands of contemporary research relating to the Ambient intelligence vision.

2. A retrospective on the field and its evolution

OPEN IN VIEWER

Fig. 1.

An illustration of major developments of the Ambient Intelligence field, relating key events that shaped the vision, to technological milestones marking the progression from early prototypes to commercial systems. The last row indicates research challenges that attracted lots of attention, and in gray boxes, anticipated advances in artificial intelligence and nanotechnology, which have not take central stage in the AmI research community.

The evolution of Ambient Intelligence has been a gradual progress marked by shifts in the rhetoric of its protagonists paralleling technological and market developments. It is beyond the scope of this article to provide an exhaustive historical account of this field, and such a narrative would inevitably be a partial and limited presentation of a very broad set of developments that transcend the scope of this research field and which were pursued independently by different actors. Rather, the aim of this brief and broad brush retrospective is to draw links between the iterative formulation of the vision and some broader societal and technological developments. In doing so it will motivate arguments on how Ambient Intelligence as a vision can or cannot be useful in guiding future research efforts.

Figure 1 illustrates in very broad strokes the evolution of Ambient Intelligence along three parallel axes. The top row shows key moments in the development of the Ambient Intelligence vision, the middle row shows some notable technological milestones, and the bottom row attempts to summarize how different research themes have attracted increased attention by the Ambient Intelligence community over time.

The top left of the figure features the publication of Weiser’s article in Scientific American [36] that introduced the notion of ubiquitous computing, an article that has inspired and influenced a whole generation of researchers and policy makers since. Key milestones featured in the top row are the initial formulation of the vision of Ambient Intelligence by Philips [2], its revision in 2011 [3], and similarly the ISTAG advisory report [14] that sketched the Ambient Intelligence research agenda for the year 2010, before this research agenda was revised in 2009 as a vision of a future internet infrastructure [20]. The critical insights by the SWAMI group in 2008 [37] have played a key role in this evolution as will be discussed below.

The second row of Fig. 1 illustrates technological breakthroughs or milestones. Initially, these refer to renown early research demonstrators that gradually have given way to the launching of commercial products and services by companies not necessarily subscribing to the vision of ambient intelligence as such. For example, at the far left in this row the Active Badge system by Olivetti, represents an early and influential experiment well before the term of Ambient Intelligence was coined, which identified relevant technical challenges for indoor location awareness and flagged user acceptance issues [37]. The introduction of products that enabled full body interaction at consumer level devices, first with the Eye Toy by Sony was indicative of more natural and embodied forms of interaction that have been consistently explored in the Ambient Intelligence research field, and are still present in the market today with offerings such as the Kinect by Microsoft, or the Wii by Nintendo. While not a first in kind technology nor a technological breakthrough as such, an important milestone has been also the introduction of the iPhone which was a turning point that led to the almost total adoption of smartphones in several countries. Although not conceived and not marketed in relation to Ambient Intelligence, one could argue that smartphones have become the canonical device for connecting people to the internet and its wealth of services, but also to computing and context sensing resources and infrastructure in the vicinity of users or even in their body area. Another important development, exemplified by the launch of Amazon’s Web Service [21] in 2006, is the notion of cloud computing [6] which promises to turn computing into a commodity that is scalable and not tied to a specific physical resource, thus relieving people from limitations of relying to a limited number of physical devices and of the onus of managing such devices directly. Social networking services such as FourSquare and games such as Ingress by Niantic Labs and Pokemon monster by Nintendo are representative of how location awareness has been explored as an enabler for novel and engaging interactive experiences that take center stage in our social and leisure activities. The launch of IFTTT (ifttt.com) in 2011 illustrates how earlier calls for enabling end-user programming for users to support intelligibility and control of Ambient Intelligence [25], are already available commercially albeit at a limited form of connecting triggers to system responses. IFTTT standing for if-this-then-that is a web service that allows users to create ‘recipes’ for invoking web services based on the outcome of other web services, which allows connecting the behavior of devices in their vicinity with applications available over the web.

The bottom row of Fig. 1, illustrates what one might consider as motor themes for Ambient Intelligence research; early research was concerned first with exploring how to embed computing in the physical environment, and exploring the potential of this embedding. Over the years research on different technologies enabling context awareness has remained constant, while research in middleware solutions necessary for connecting context sensing technologies in a reliable and seamless manner, seemed to peak around 2005. Interest in solutions for controlling context aware environments peaked around that time borrowing from and extending end-user development approaches originally related to programming graphical user interfaces. This research topic is becoming again very relevant as the technological infrastructure is maturing and applications are scaling up to support the notion of the Internet of Things [19,22].

Looking back at this evolution that has spread over more than a decade, and which involved actors from several countries and research establishments, some notable digressions are noted compared to earlier expectations introduced with the original vision documents regarding Ambient Intelligence: major breakthroughs in artificial intelligence and nanotechnology anticipated with the ISTAG scenarios in 2001, have either not materialized or have not yet impacted Ambient Intelligence research as dramatically as expected. Rather the penetration of smartphones, the almost total adoption of social networking services, and the increasing popularity of cloud services seem to have had a much more profound impact.

As we approach 2020, and perhaps resulting from the shifts in the official European research agenda, the emphasis on ambient intelligence has given way to the related notion of the Internet of Things [19,32] where identifying objects uniquely enables their localization and consequently the development of context sensitive services that manage virtual representations of their physical counterparts.

The Internet of Things pertains to very similar concepts and scenarios as those of ubicomp and Ambient Intelligence, that differ perhaps only in their emphasis upon improving infrastructure and scaling up applications to achieve the intended societal and business impact. Clearly, as Ambient Intelligence technology becomes increasingly available as infrastructure, it becomes increasingly meaningful to diversify and specialize research agendas and visions for different application domains and target user groups. Similarly, different application areas for Ambient Intelligence research have evolved into flourishing research communities on their own right, such as Ambient Assisted Living [10,11], Hybrid Cities [33], Pervasive Games [4,31].

In the following sections we reflect on some key questions for the Ambient Intelligence research community, relating to the extent that Ambient Intelligence as a vision has materialized. This reflection commences with the notion of ubiquitous computing that predated and inspired work in Ambient Intelligence. It then discusses briefly some turning points in the history of Ambient Intelligence research.

The notion of ubiquitous computing [36] advocated a shift from stand alone computing devices to networks of devices of different form factors, ‘tabs’ (wearable centimeter sized devices), ‘pads’ (decimeter sized devices) and ‘boards’ (meter sized devices) and interfaces integrated in the environment (e.g., peripheral displays in windows, alarm clocks supporting simple speech based interaction, etc.). Such networked devices (sometimes called information appliances) would allow interaction that is easy enough that it would become an automatic cognitive process, something that people would do effortlessly and even unconsciously. Like many similar descriptions that were inspired by Weiser in the years that followed, his narrative was imbued with techno-optimism. The scenario he presented to communicate this vision describes how in the course of daily mundane interaction a wealth of information would be captured by technologies operating in the background and would be made easily available as a layer of information and interaction superimposed on the real world, at the periphery of the user’s attention ready to be noticed and acted upon. In so doing, many tasks that are effortful or dull could be made easier, automatic, and even pleasant catering to actual information needs or even just the curiosity of the user.

In terms of diversifying the form factors in which computation is available to people, one can say that to a large extent we already live in the era of ubiquitous computing. Smart watches, smartphones, tablets, and interactive surfaces are the contemporary role incumbents for Weiser’s three different form factors mentioned above, which are by now widespread, relatively affordable, and support ubiquitous access to a wide and increasing range of services. This ubiquity of computing and internet connectivity has by now become the norm; it is an infrastructure that we notice only when it is absent or malfunctions; one can hardly conceive of stand alone appliances that do not offer Internet connectivity. The seamless integration of devices and applications pursued by ubiquitous computing research is becoming so ingrained in our daily interactions with mobile computing, that mobile applications have become the de facto bar for user experience. Housekeeping operations which once required considerable effort are now performed in the background, e.g., connecting to the internet, sharing data from different appliances through the cloud, synchronizing audio visual content across devices, migrating interaction sessions from one device to another, adapting to our location and prior use of different services (e.g., bookmarks, or browsing histories), etc.

Progress is perhaps most obvious when one considers contemporary consumer-level technologies that directly implement pursuits of the past twenty five years ubiquitous computing research:

Views at to how much this vision has been achieved are mixed. As early as in 2010 Schmidt has argued [29] that we live in an era of ubiquitous computing. Indeed, Weiser’s scenarios are feasible with current technology, though admittedly their adoption and ubiquity is perhaps less than the visionary implies in his narrative. Still today, computing technology while pervasive is not as ubiquitous as assumed or implied by Weiser’s scenarios: While modern alarm clocks can be automatically synchronized with satellite data, and some products are designed to enhance our ‘wakeup experience’ through digitally controlled lighting, e.g. the Philips wake-up light, which are typical Ambient Intelligence information appliances, one could argue that the majority of alarms today are still stand alone ‘dumb’ devices no different to those available in the eighties. The specific example of projecting information on windows matching user’s gaze and interest has been demonstrated since, e.g., see [24]; nevertheless windows and walls in most houses and shops do not embed interactive displays. Recently the public’s imagination was captured by alternative technologies for superimposing a layer of information on physical surroundings such as Google glass, e.g., see the Cyclo device to provide contextualized information in the field of view of cyclists [30]. Such devices are still not very widely available and adopted and whether they are perceived as providing value to people in order for them to adopt them is not at all certain at this time.

In short, technologies that are wide spread and very robust have brought ubiquitous computing from vision to reality, even though the scale of ubiquitous computing, its robustness and ease of use are not such as implied during the inception of the vision. Below we discuss the extent to which the vision of Ambient Intelligence as formulated at the turn of the millennium [3,15] has been achieved.

4. Intelligence: Proximal and mundane

Continuing such a comparison of past visions to their realization, it is worth revisiting the narratives that elaborated what Ambient Intelligence could mean. Specifically it is worth examining the scenarios that the ISTAG group developed fifteen years ago [15]. Without considering every single element of these scenarios it appears safe to say that they also have been partially realized. For example, consider the scenario “Carmen: traffic. Sustainability, and commerce” which envisions a mobile service that brokers rides between commuters, as well as grocery shopping on the internet, services widely used today. Rather than technological bottlenecks, it is the infrastructural and socio-political elements of that scenario which remain challenging today, e.g., automatic debiting for road usage and remote speed capping of vehicles are already implemented in several countries though they meet considerable reaction by different stakeholders and pressure groups. Already in 2001 the ISTAG group anticipated that such socio-technical issues would represent the main obstacles for the realization of this scenario.

The ISTAG scenario entitled “Maria: personal ambient communicator” [15] foresees an ubiquitous infrastructure of services, that a user invokes implicitly through her handheld device, to automate transactions as diverse as visa application, passport control, booking cars and hotel rooms, paying for them, communicating to friends and family, customizing digitally controlled lighting and AV environments, etc. While obvious sociopolitical and operational challenges hamper the full implementation of this scenario across borders, its technical aspects are already feasible. For today’s smartphone users, the level of integration and seamlessness may not yet be as implied in the narrative but most of the services involved in that scenario (including booking tickets, ground transport, visas, hotels) are quite easily accessed with a few simple steps.

So could we claim then that at least at a technical level Ambient intelligence has also been realized? Considering the defining characteristics of Ambient intelligence [2] embedding in context, context awareness, personalization, adaptation to and anticipation of user needs, we see that Ambient intelligence research has made strides towards the first two aspects which are shared with ubiquitous computing as noted above. However, there is less compelling progress regarding the latter two which pertain to the embedding of machine intelligence in daily life contexts.

Early expectations that context awareness would allow models of human activity to be created and to be used as enablers for ambient intelligence services, have given way to more nuanced expectations of what machine intelligence should be: situated, social, fitting expectations and norms of people [11,13] had been that were brushed over in early writings. E.g., in the ISTAG scenario the car unlocks for “Maria”, the scenario’s heroine, as she approaches the car, ignoring the fact that perhaps she is just walking past it without the intention to use it. [7] points out how the certainty by which inferences are made is an important requirement that has not yet been addressed adequately; depending on the severity and importance of the decision that the computer needs to make designers and engineers need to be able to reason and communicate clearly about uncertainty. This requirement requires first the ability to communicate effectively to users the basis of inferences made [27], but also to allow for reasoning probabilistically, and helping users to anticipate and comprehend ambiguity.

Further to the reliability of the inferences made, ambient intelligence researchers have argued for extending the notion of intelligence to include a more holistic interpretation of social context and acting in a socially intelligent manner [12,26]. At first sight, it appears that reframing and broadening the notion of intelligence pushes it further to the future [18,23]. On the one hand, a reframing of this kind is necessary to nuance the role intelligence could play and make more compelling the value we expect such technology to deliver to society. On the other, what we envision as an intelligent system behavior is not objectively and immutably defined but appears to be a vague concept and value judgment that is subject to a continuous process of adaptation and socialization. For example, as we get used to systems translating text we do not consider translation to different languages any more a sign of computer intelligence than performing mundane calculations. Neither are we impressed when a highly personalized message appears on our smartphone from the pizza shop nearby, or when the smartphone can recognize a tune that is playing or that we hum (e.g., the Shazam application for Apple iphones), or that it can issue a search based on a picture of an object (e.g. Goggles by Google) or ostensibly conduct a spoken dialogue with the user (e.g. Siri by Apple), etc. This list could be extended with numerous applications of artificial intelligence in every day life.

This phenomenon is a socialization process well known to artificial intelligence researchers who have remarked that as soon as an artificial intelligence technology succeeds in its aims, it stops being perceived as ‘intelligent’. Still the current prevalence of artificial intelligence techniques in every day life is very impressive, even if not necessarily noticeable, e.g., when our internet browser adapts to our behavior and tastes, when our smartphone starts proactively warning us of traffic signs ahead based on inference it makes from our past behavior, or when cameras use face recognition for supporting auto focus, etc. The recent demonstrations of autonomous driving (despite the recent occurrence of the first traffic accident involving autonomous vehicles in 2016) have captured the imagination of the public. One can imagine that following previous analogues this technological feat will also be considered a mundane function that is taken for granted in the not too distant future. A rather different perspective is offered by Jose et al. [23] who argued that the intelligence often assumed in Ambient Intelligence scenarios is general common sense of a human being, which is often termed the classic/general artificial intelligence. Recent analyses predict that this level of artificial intelligence will be attained in about 20 years time, though one could comment that this is roughly what the expectation had been at the first steps of the field of artificial intelligence [5].

Recently scholars in the field of ubiquitous computing and Ambient Intelligence have questioned whether machine intelligence is feasible or even desirable [23]. The complexity and situatedness of human activity hamper any automatic inferencing about user activity and intentions, and the very desirability of a system anticipating people’s needs is questionable also. This line of argumentation suggests a more nuanced expectation of how machine intelligence could be used, fit for the emerging sociotechnical landscape of the next decades, e.g., where large data sets are cheaply generated and shared, but where comprehending them and acting upon them exceeds human cognitive abilities, or where computation and intelligence involve orders of magnitudes more devices and services, as in the internet of things [17,32]. Starting from the notion of enabling the automated tracking of large populations of physical objects, e.g., for logistic purposes, the internet of things has evolved to a vision of an open infrastructure that will enable large populations of services to be assembled by users to provide customized interactions with the physical environment surrounding them. This type of infrastructure is not widely available yet, but is growing steadily and will gradually enable the type of scenarios discussed above.

5. Addressing actual societal needs

More profound than the reframing of expectations regarding computational intelligence is the shift of values underlying this research field. Early discussions of Ambient Intelligence have advocated very emphatically the expected benefits of the technological vision, implying a sense of inevitable a self-fulfilling prophecy that would be achieved if repeated and if endorsed by the research community. Driven by such optimism, a concentration of resources and efforts has enabled strides towards turning the vision to reality. However, this optimism has been countered by skeptical considerations of the pitfalls of Ambient Intelligence [37], triggering a broader discussion and a more reflective stance on technology development, acknowledging the need to provide value to broader sections of society, to consider personal freedoms and the need to invent sustainable models of development.

For example, early Ambient Intelligence work at Philips [14] demonstrated potential advantages of seamless networking connecting consumer electronics devices at home, and speech based interfaces with an animated conversational agent. Envisionment videos of such an agent anticipating one’s arrival at home, setting the lighting, the music, even filling the bathtub were often discussed as examples of what Ambient Intelligence could mean for the future home environment. This idea of a helpful agent working almost in the background to anticipate user needs is also exemplified in the “Maria” scenario discussed above and in the D-Me scenario of [15] in the latter case, taking a proactive, and perhaps one should also say invasive role, in managing social interactions with one’s social network.

Casting Ambient Intelligence in the role of a butler who preemptively acts on assumed wishes of users was a very prevalent ambition at the outset of this field. Conceived pre-September 11 and before the advent of the financial crisis of 2008, fuelled by post cold war optimism, Ambient Intelligence research seemed to aim for comfort and luxury for affluent individuals. The scenarios of the time emanate an unmoderated techno-enthusiasm, brushing over dangers and potential misuses of technology.

In his keynote address to the Ambient Intelligence conference in 2010 (summarized in [3]), Aarts admitted that the original ambient intelligence vision may have reflected how people aspired to live anno 2000, but not the concerns and ideals of a decade later in the midst of a financial crisis that deeply affected societies worldwide. A few years later, an even more sombre climate prevails today, given the increasing threat of terrorism, armed conflicts, a refugee crisis, and threats to the world climate, etc. To address this disparity between the vision and the real life context in which Ambient Intelligence will need to be deployed Aarts put forward the notion of synergetic prosperity [3], which he defined as the development and application of eco-affluent innovations that allow people to flourish.

The technologies that Aarts describes as exemplary of synergetic intelligence, are a far cry from those sketched by the four ISTAG scenarios, in terms of complexity, context sensitivity, and intelligence. Perhaps the most extreme departure is the example of a reading light that is intended to be used in parts of the world without adequate power network coverage, that could bring low cost to users, and have small ecological footprint. Highlighting this product by Philips does not signal a lowering of the bar in terms of the technical challenges pursued but, rather, a major shift in priorities, and values driving Ambient Intelligence research, namely to support sustainability, people empowerment, to address emerging markets, to provide societal and market value.

A second example described by Aarts [3], concerns a heart monitoring application, indicative of the quantified self movement that was to follow and the current popularization of personal health monitoring devices and services [34]. Next to healthcare where the societal benefits are obvious, one can also see how contemporary Ambient Intelligence research in different application domains, like assisted living, traffic management, can extend their benefits to broader sections of the world population seeking to satisfy diverse and pressing needs rather than a luxury lifestyle.

6. Future research

Ambient Intelligence technologies may be entering our every day life, but long standing technical challenges remain and as a result promised or expected benefits to society have still to be reaped. Two important trends driving developments in Ambient Intelligence remain those of miniaturization and disappearance of computing and the integration of devices and services. However, compared to the first years of Ambient Intelligence research there is a qualitative transformation taking place. Current efforts aim for scaling up Ambient Intelligence installations in physical size, number of users, level of interconnectivity of devices and services. Rather than feasibility demonstrations for context awareness, current research and industrial developments are putting into place the required infrastructure that may pursue further the integration of physical and virtual, as in the Internet of Things, and will enable users to collect and utilize vast amounts of data [8] and services through cloud computing [6].

Based on this observation one should expect a growth in interest in Ambient Intelligence research. However, this is not immediately visible as Ambient Intelligence research is increasingly diversifying, branching out in various application domains as mentioned above (Assisted Living, Autonompus Driving, Healthy Lifestyles, etc.). Rather than reviewing these trends individually, it is worth to examine what constitutes the core of current Ambient Intelligence research and the shared interest of its different branches and application fields.

A number of challenges transcend any specific sub area of Ambient Intelligence:

Acknowledging major contemporary developments and trends in the research world does not help describe clearly what the ambition could be for Ambient Intelligence research. In line with the seminal works that inspire this field, it is perhaps more useful to illustrate contemporary challenges of Ambient Intelligence research using a brief scenario:

“Alison is rehabilitating from a heart operation. She builds a personalized coaching program for eating and physical activity, by combining off the shelf health monitoring devices and services, with services provided by retailers in her neighborhood. She decides not to share data with commercial providers; after some time she pulls in some readily available gaming elements. Some friends ask to use this self-made service, but they ask extra reassurances about privacy and control of their data. As Alison attempts to share her solution with her friends on a social networking service, she is guided as to how to satisfy different privacy needs.”

This scenario is not put forward here as a blue print of what might constitute a ‘true’ and contemporary blueprint of Ambient Intelligence. Rather it emphasizes the growing impetus towards personalized health care, and emphasizes how users become active shapers of their technical habitat and participants in communities of innovation [16]. It points out how inseparable technical developments are from supporting socio-technical ecosystems, advocating an active role for industry in empowering end-users and to foster creative communities. It also suggests the need to develop viable business models to resolve the question of who will pay for the development of Ambient Intelligence service and infrastructures.

7. Conclusions

Ubiquitous computing has entered our daily lives though we are seeing only the tip of the iceberg. The notion of a disappearing computer is bound to apply to ever more artefacts embedding computing and context awareness. Added to this machine intelligence will be needed not only for making inferences about a user’s context and intentions as once foreseen by pioneers in the field of Ambient Intelligence, but also to assist people in managing an information and functionality rich environment. Managing these systems is not necessarily a case of automation and supervisory control, but more likely a challenge for enabling user empowerment and control.

While we may not be able or inclined to proclaim a system as truly intelligent in the near future, we can strive to extend human cognitive and physical abilities, their awareness of their own body and surroundings, and the leverage they have to act upon them. The original vision of Ambient Intelligence has served its purpose in aligning European research and development. For years, it has motivated the development of enabling technologies and more recently the development of an enabling infrastructure that will enable Ambient Intelligence scenarios to make a transition from research installations to real life. Such an infrastructure is gradually developing, but it is yet not at all clear how challenges that are key to the acceptance and adoption of Ambient Intelligence will be resolved. Ensuring security, fair information practices, personal freedoms, entrepreneurship in the technological landscape that is emerging, necessitates an even larger alignment in the following decade, as well as political and social engagement of different stakeholders to conjointly shape our future technological landscape. Next to industries and government sponsoring, community and grass roots initiatives may play a key role in creating or appropriating technologies that will shape how Ambient Intelligence materializes in real life and will shape it in the next decade.