Implications of augmented reality in the management of television audiovisual information

1. Television and audiovisual archives

Television as an invention has not even reached 80 years of history. This is nothing compared to the history of mankind, yet an eternity if analysed from the perspective of the 20th and 21st centuries and their ever-changing technological environment. We cannot just talk about a constant technological evolution in communications and information, but also about a technological revolution in view of some changes that lead to a paradigm metamorphosis in the way we create and understand information.

Television from the 1950s in the last century has nothing to do with the current reality. The way it was broadcasted and produced has changed dramatically, and apart from the many changes of television in the form of production, the generation of audiovisual products and the evolution of television genres, there is a fundamental variation that translates into a paradigm shift: digitization.

This evolution has led to changes in the way television industries undertake production, as well as the methods of capturing and broadcasting audiovisual signals. Digitization processes have modified many consolidated aspects on television, enabling digital audiovisual production, online information management, access to contents from mobile platforms, network interconnection creating a new reality like Social TV, etc.

The audiovisual digital paradigm has transformed television into a television that is social, accessible, multi-format and multi-support, mobile, and in continuous evolution toward personalized and individualized content broadcasting. Television is much more than a mere formula for disseminating information, it has become a platform for exchange.

And many of these changes have also mutated the document management services of audiovisual companies, services that have adapted to a changing reality under a different production model, and as such, a model of rapid and quality interaction with users (now more than just the journalist) tailored for each of the different communication platforms employed in television.

Under this changing environment, information services are required to adapt their habits to implement their techniques and mechanisms to the ways of making television, and to be alert to the formulas perceived today as potentials or chimeras and that will be available to users in the coming years.

The scientific literature about television audiovisual collections always identify this reality as ‘media archives’ derived from the regulations and recommendations carried out by UNESCO, however we have identify that if we adopt this terminology to the widely understood by the scientific community. We have to understand this work as part of the Information Management System, specifically in the Media Asset Management in the moment that they are working with audiovisual information in digital format. Therefore, this work should not be understood from the archival perspective from an overall view.

It is true that there have been major technological failures, or that great inventions that were to change habits and models did not produce the expected results, and some of these tools have even disappeared. However, we have to be alert to the changes that may be generated and that are somehow already being happening in relation to augmented reality (AR) and its implementation by television networks [1].

This essay will show the current situation of AR, what it is and what are the lines that are currently being developed. But we will focus in particular on the analysis of AR in television, describing both potentials and the current situation, and identifying how we can change the models for exploiting audiovisual images in television archives.

2. State of affairs: Augmented reality

Mankind has senses for interacting with its environment. These senses facilitate the identification, and in some cases, the transformation of the elements of such environment. For this, machines have been created to help manipulate and control these actions. The evolution of machines has been frenetic, the potential and uses thereof have changed the way we live, especially with the irruption of computers and robotics.

This technological evolution offers increasing irruption potentials. AR, unlike other technologies or applications, does not supplant reality but rather complements it and improves it to some extent, implementing a real-time combination of virtual and real objects.

The first experience of AR dates back to 1966, when Ivan Sutherland [2] created the first AR system. This experience consisted in wearing a helmet with a screen for viewing simple information. It was not until 1992 when the term AR appeared, registered by authors Caudell and Mizell [3], who used it to identify the overlapping of materials generated by computer on real-world images.

Two years later we found some major work on AR by Milgram and Kishino [4], who analysed the concept in relation to the HMD (Head-Mounted Display) technology. A year later, the same authors [5] consider AR as a compromise between virtual environments and the reality. Other authors, successively, have come to the definition that AR is a mixture and/or union between the real world, mainly, and virtual elements that accompany, complement or clarify the real world. We also note the classification of AR applications made by Parhizkar et al. [6]: Desktop, Mobile devices and the Web.

Azuma [7] describes the peculiarities of AR from the technological standpoint, focusing on ‘increase’ and ‘viewers’. He defines the first element as the ability for devices to include as much information about real-world objects (visual, sound, etc.) and addresses the need to have viewers to combine the real world with virtual information, using optical or video technology.

AR is clearly expanding, and in addition to educational purposes, its usefulness and applications are focused on fields as diverse as medicine, business, the video game industry and also the television world, which have already been using AR for many years now. The video game industry has bet strongly on the use of AR due to the type of users and requirements, although each area has been analysing possible ways of using this technology, offering great potential for education, which is still being analysed and tested.

AR is still a future technology that needs to be further investigated and developed to reel off all challenges and opportunities that it can offer. In the scientific literature consulted, we found a series of technological advantages and disadvantages related to this technology. Fermín Lobo [8] points out the main advantage of obtaining value-added information on a particular aspect in a ubiquitous matter. Ultimately, information that complements, supplants and/or enhances the production of knowledge assimilated from reality.

Flórez Aristizabal [9] identifies that AR has display devices, computers, input and tracking devices, and Olmedo and Augusto [10] highlight three technologies:

The main hardware for implementing AR are:

In terms of software for the creation of AR, it is easy to use and manage, without requiring much knowledge from the technological standpoint. There are many options in the software market, both free of charge and by paying a fee, of different prices and on various platforms such as PC, Web, Mobile or with the possibility of using several platforms [11].

3. Augmented reality in television

Scientists, engineers, humanists, etc. have all used computer image synthesis as a tool to represent their ideas visually, and also to represent environments yet to be built in real life. The continuous attempt to control and use images goes far beyond the use of computers; it dates back to photography and film.

The tendency of using visual metaphors to represent the world has made it possible, and somehow necessary, to do research for image rendering by developing increasingly sophisticated software and hardware – three-dimensional image, for example [12].

The realism achieved through computer imaging is impressive. This evolution can be clearly seen in the development and quality of video game images, as well as virtual reality and other experiences like Dome Theaters.

Images are not only copied from reality, but they are implemented to provide new informational, educational and entertainment details.

AR has already been tested in some arts, such as performing arts, through the incorporation of audiovisual elements by implementing Virtual Reality between actors [13]. There are also some examples of media using AR in their electronic editions, either as a marketing mechanism or to provide information. Some examples include US magazine Esquire, British newspaper Guardian.co.uk, Wallpaper from the UK, Italian weekly fashion magazine Grazia, SZ Magazin from Germany or Spanish magazine Fotogramas [11]; their use of AR is not prolonged in time, but rather in isolation and usually for advertising purposes.

It is worth noting that AR is no stranger to television and that it is now incorporated in many scenography and advertising sections, etc. of audiovisual companies. The incorporation of AR in television has led to changes in some stages of production and post-production by including computer-generated elements [14].

Galán [15] points out that television has used virtual scenography in Virtual Reality applications since the 1990s, when the reverse version of AR was implemented in some cases: in television, real images are overlapped on the virtual environment, while AR, by definition, in its video-see-through format, overlaps virtual images on real images. Virtual Reality should be understood as the predecessor of AR in television, whose clearest example is the overlapping of images in Chroma-key.

For this reason, we can talk about three different stages in the use of AR in television:

The first phase is the contribution made by Virtual Reality. The second phase incorporates information on the actual image, of a different nature (AR as such). This use of AR is more common than expected. An example of this would be advertising in the field of sports competitions, team shields, lines delimiting the sports field, etc. Other ways of using AR is by implementing graphics in a way that the host appears as if he/she was an actual hologram, real-time statistics, using Google Maps to access the exact location of the images, among other things.

The third phase, still in its early stages and undergoing research, is the one that can contribute the most and can really change the way of watching, using and interacting with television. This third phase, or the development of AR, consists in providing textual, hypertext, sound, visual and/or audiovisual information through external devices to the television screen, interacting with it and obtaining information that complements, enhances and increases the information provided through the traditional method.

The objects that are visible through AR may only be perceptible on television by using external devices.

The evolution of mobile devices such as tablets, iPads, Smartphones or iPhones, will be the best way to obtain the new information provided by AR, although we should not disregard traditional AR helmets, nor the development of glasses that allowed us access to additional content.

The nature of the material provided can be extremely broad, of any documentary nature or with possibilities of interacting with the network. An option provided by AR in television could be a photograph, text, audiovisual material that complements the images, and even a hypertext. The possibilities are endless, although it is expected that the most spectacular development of AR will be the incorporation of audiovisual material, in a way that the ‘television game’ will take place in two levels of representation.

Great accomplishments are being developed by Japanese companies, such as those advanced by NHK (http://larepublica.pe/blogs/realidad-aumentada/tag/nhk-hybridcast/), which is working on the interaction with television through tablets or smartphones. Hybridcast (or hybrid broadcast) is based on broadcast interconnection with information on the network. There are some visual examples in the operation network of Hybridcast linking access to the network with information, in some cases, related to AR, such as the inclusion of statistical data, information and identification of players in real-time sports competitions, etc. Although still in test mode in Japan, this technology will also improve Social Television significantly, and it is being implemented by large technology companies like Sony, Panasonic, Mitsubishi, Toshiba or Sharp. The race toward a real-time interaction with the network is already underway.

Another Japanese company, NTT, is developing a software using AR that allows access to audiovisual content that can only be accessed through a second screen, and that will be displayed synchronously with the traditional television signal. In short, two screens that need to be watched at the same time to understand the television message. The screens required to view this second image will be mobile devices, such as Tablets or Smartphones. Visual SyncAR (http://www.neoteo.com/visual-syncar-realidad-aumentada-sincronizada-con/) also measures the distance and angle of the viewer to the screen in order to determine volumes, perspectives and angles, thus generating emotions and optical illusions.

The possibilities that this technology can offer are yet to be determined, discovered and tested; what is clear is that there will be an informational and/or educational enrichment by completing and improving audiovisual content.

4. Exploiting audiovisual archives and AR

Caldera [16], among other authors [17], has shown us the changes that are happening in the audiovisual information management departments of broadcasting companies, once digitization of the editorials has been achieved and, consequently, of the documentation contained in the archives. They point out that digital systems fragment and classify audiovisual documents, turning sequences into small units of analysis, selection and treatment, and that such sequences, although inheriting metadata from the matrix, have their own metadata that individualize them.

The evolution of documentary habits have also changed how the description of images takes place, because although it remains a set of referential elements that lead us to the original document, in this technological tessitura, it is not necessary to review as many referential elements since we can access the full document, quickly. Why carry out such an exhaustive description if we can have the image (in low resolution) with a simple click.

The descriptions ‘soften’ down, information management accelerates, the availability of information to users-journalists is much faster, and such information is even accessible without the collaboration of information managers.

It modifies aspects such as the selection [18, 19] of the material integrating the archive, the intake and management of images, and it changes the control of traditional access elements like personal names, geographic, chronological and thematic factors [20].

Although any type of content may be distributed through AR, we will focus on the audiovisual material from the audiovisual archive using, as a link, the components disseminated or issued by networks in the traditional way (received by TV sets and broadcasted through the network)

Using devices external to television, we can display AR contents while watching the news or any entertainment programme.

They are aware that in order to generate 1 minute of a news programme or a feature programme of longer duration, hours of material have been captured in some cases, which, once analysed and selected, becomes part of the audiovisual archive of the television network. This material is related to the material produced by metadata which identifies the source of the images used, whether the material has been captured on the same day or comes from the archive.

We therefore want to reveal that the information produced is linked, in the document management system, to the material used for generating the information or audiovisual product, both material captured exclusively for such information and the material captured from the archive.

As a result, we have a set of images related by some material produced that can be joined together via links, thus creating an authentic audiovisual ontology, given that the material generated for a specific information may interact with other different ones in the future.

This relationship is interesting from several aspects: from the creation of a set of related images that will help us provide more and better information at the time of documentary recovery, and the possibility of jointly providing users with the required material and also with related material that could be of interest.

From this second point of view, it is important for the marketing development of audiovisual archives, not thinking or focusing solely on internal exploitation by journalists but also by viewers.

Viewers, in the context of persons who just watch television, are becoming increasingly obsolete. Viewers are evolving into cyber-viewers, persons using other media while watching television, who increasingly have more knowledge about Social Television, and understand and use social networks. And they not only watch television, but also provide content and require more content.

Exploiting the knowledge already acquired by them and the synchrony of tasks by these new cyber-viewers, the media, using AR, can undertake new and interesting ways of facilitating such content.

From the display of contents, and by using tablets or smartphones, users cannot just watch the real-time content broadcasted by the network but AR also facilitates access to the content with which such information was created. It is as if we could access interviews, additional content, movie trailers, etc. while watching movies at the cinema. But the purpose here, in addition to broadcasting new content and generating user loyalty by providing new and innovative tools, is to present a form of selling this material, disseminating the contents and giving options to professional viewers –advertising agencies, news agencies, freelancers, etc. – to link these contents so that they can be purchased.

The proposal is therefore based on accessing the material of the archive, which is attached in the document management system, from an external device other that the TV set, and that the material from the archive can be simply viewed or sold from this system, offering not only audiovisual material linked directly to what is being observed but also material relating to the information deposited in the archive.

To ensure that the content remains a fundamental asset of the network and its use does not go beyond private and personal consumption, the material can be provided in low-resolution, so that users who require such material can be then offered in high resolution.

The form of selling such content goes beyond the original idea of this essay. However, proposals may include payments per minute/hour, premium versions, etc., all depending on the scope of the archive to be priced for accessing content of a different level of utility and therefore economical category.

All this also incorporates another important factor that we are currently analysing and developing in other works. If users can browse through the contents accessed after using an AR device while watching television, and they are also given the ease of searching and retrieving information, we may have free social indexing and we can then incorporate internal and external elements for retrieving audiovisual material, thereby achieving social indexing for such content (indexing that is currently being developed by the information manager).

5. Conclusions

AR can change any of the immutable paradigms of broadcasting and access to television information. It can produce major mutations and endless possibilities in the access to new contents, especially through external devices other than TV sets as such. AR glasses, Tablets, Smartphones and Apple mobile technologies facilitate opportunities of generally having information on two levels accessible through television screens and secondary displays.

The proposal is extremely simple and technologically feasible considering that the documentation departments have already digitized all content and these can be made available through one of the many existing AR tools.

Making archived material available to users, or new users, ensures greater external dissemination of its contents, and therefore greater knowledge, which will result, for some user profiles, in the potential purchase of these images. Also, we must not forget that television networks must constantly reinvent themselves to find novelty elements to retain their viewers, especially among digital natives, and a clear way to do so is by offering ease, potential and certain free contents.

Not only directly related information is provided, but viewers may also interact with the contents of the archive, thus obtaining information that was not initially of their interest.

Sales, marketing, viewer loyalty and ultimately the sale of AR-associated contents would generate important benefits for networks, not only from the economic standpoint but from the position of corporate marketing and insight into the information society, where ‘digital natives’ have become content predators, especially of the audiovisual kind.