Interactive projection information generation and feedback for reading behaviour

Abstract

Interactive reading is a new reading behavior in the age of intelligence. The dual-track development of new media and interactive technology has led to changes in both reading behavior and reading experience. This paper has discussed the information generation and information feedback of interactive reading, and analysed the important applications of projection technology, gesture recognition and QR code technology in interactive reading behavior from various projection interaction technologies. Through the development process of the interactive projection of the atlas series named “City Flowers of Beijing”, we aim to demonstrate the design of interactive and sustainable reading forms, deliver the multi-dimensional, multi-level and diversified reading information, create a reading experience that is both physiologically and psychologically satisfactory to the audience in the age of intelligence, and provide a new direction for the development of human-computer reading behavior in the new media age.

Keywords

Interactive reading projection information gesture recognition QR code technology

1. Introduction

The increasingly diverse forms of human-computer interaction in the age of intelligence have led to the emergence of new reading behaviour, which is a rational realization of the emotional needs, physiological senses and cognitive thinking of information receivers. The assistance from artificial intelligence technology has enriched the interactive medium, trigger method and perception form of reading, and driven the intelligent service and experience improvement of reading, making the audience’s access to information, showing the perceptual characteristics of the transformation from static traditional graphics to dynamic interactive audio-visual.

In the current new media era, the help of artificial intelligence technology fully mobilizes the reader’s senses, driving mobile reading wisdom services and experience upgrades, making information acquisition present the perceptual characteristics of traditional text images to dynamic interactive audio-visual transformation, integrating media thinking and providing multi-sensory reading experience services. Interactive reading intermingles the interaction of paper media and digital media, and merges the dynamic convergence of reading objectives, which will become an important means to lead information dissemination in the new era. The current interactive reading has the following three problems:

First, the limitations of the participating groups. From the perspective of the information sender, the information needs to be sent to the content of the overall organic processing, which will limit the original inspiration and direct thinking of the sender, increasing the cost of its own information sent; from the perspective of the information receiver, older groups and groups that have not been exposed to new interactive technologies lack the opportunity to read interactively, and the means of interaction increases the complexity of the reading process of the group. Second, the blind use of new technologies. New technologies and new media are being used in an endless stream. Currently, the actual application scope of these technologies is relatively narrow. The products pay too much attention to the degree of formal innovation, ignore the transmission of essential information of reading, and even test the working performance of new devices only through reading behavior. Third, the lack of interactive reading places. Traditional reading through paper or electronic media, has formed a perfect library, bookstores, electronic platforms and other reading places to establishï¼› interactive reading because of the combination with new media, need such as projectors, sensors and other more technological devices, related equipments need to display space, debugging personnel and other participation, so also limit the further development of interactive reading.

2. Related research

2.1 Interactive reading

Interactive reading combines rich information carriers and information formats, and places more emphasis on the combination of the reader’s subject activism and the openness of the reading material object. The reading behavior is undergoing an intelligent and humanized turn, and the core issue of information access is shifting from how to be clearer and faster to how to be more vivid and deeper. This is a change in the social and spiritual connotation level within the block of cultural understanding, with the outer layer of change wrapped in the inner layer of differences in mode of existence and values [1].

The theory of the interactive reading model was proposed in the reading research community in the late 1970s, in which it was theorized that interactive reading is not a unilateral interaction between the reader and the reading material, but a multilevel, cross-cutting, continuous, feedback-based communication process, which in turn improves the reader’s reading efficiency [2]. In an article published in 2006, some scholars further pointed out that the context of interactive reading is a space of mutual subjectivity that allows the construction of inferential structures [3]. Abroad, the theories and models of interactive reading have been studied and explored to some extent, and the areas of practical application are relatively extensive. Guo Zheng, a scholar in China, pointed out that in addition to the change of information acquisition in the past, interactive reading has adjusted the connotation, rules and power deeply embedded in the way of reading, which is an overturning and renewal from the outside to the inside [4]. In 2014, Ni Jincheng, a scholar, proved that the interactive reading model has significant superiority compared with the bottom-up and top-down reading models through a controlled experiment on three groups of students’ English teaching models, and superiority is due to its consistency with the cognitive process of reading [5]. With the development and extensive implementation of various new media technologies, the degree of technological development will influence the reading industry in all aspects in China, and interactive reading based on machine perception and multi-sensory experience will become an important area for the future development of the cultural cognitive industry [6]. The combination of future interactive reading platforms and various media will strongly compete for the reading market in order to adapt to readers’ higher demands for reading experiences. At present, all aspects of the domestic market are paying more and more attention to interactive media art, and many institutions of higher learning are building new media art labs specifically, which will produce more and more exciting interactive reading works in the future [7].

The application advantages of interactive reading are as follows: Firstly. Increase the experience of reading.Interactive reading reduces the tedium and burnout of the reading process through multi-directional and diverse information exchange, enriches the form options and sensory experience of reading, and at the same time, forms the core driving force of reading with the help of social media platforms, reverse reading generation, virtual reality and other technologies, increases the reader’s sense of value, socialization and immersion, and makes it easier than traditional reading to achieve aesthetic, emotional, reflective, value, social and immersion and other dimensions of the experience of the sense [8]. Secondly. Enlarge the pattern of communication. In the process of reading material delivery, the content and structure of interactive reading is also constantly updated, increasing the dynamic volume of communication, forming a new unified association among different fields, cultures and groups, and building a multi-dimensional communication system with the full range of information and multiple communication perspectives. Thirdly. Enhance the expression of content.Readers’ understanding of information depends on their own knowledge reserve, comprehension, inductive ability and association ability, not on the information itself. The information sender strengthens the accuracy and effectiveness of its own thinking expression through rich and diversified transcription and clever and free media selection, while the reader repeatedly understands the author’s communication intention and information flow through multi-level and multi-dimensional reading experience, and gives correct feedback to the reading material by integrating the reader’s own understanding under the constant trial and error of the interactive device. Fourthly. Generate multiple information.In the intelligent era, readers’ reading needs are becoming increasingly complex, and the traditional content generation of reading has been unable to meet the expectations of readers’ existing knowledge. In interactive reading, the reading material is no longer a one-way inspiration delivery of the information sender, but becomes a process of being read and understood, so that the elements of information are combined and derived again. After the receiver reprocesses and reproduces the acquired information, the reading is truly completed. Interactive reading expands the channel of content generation, adds the form of content generation and changes the traditional mode of content production.

Table 1
The difference between interactive reading and traditional reading

[height=0.8cm,width=3.8cm]Contrast dimensionReading type	Traditional reading	Interactive reading
Content structure	Relative static	Relative flow
Reading thinking	Continuity	Dispersibility
Sensory participation	Single sensory participation	Multi-sensory participation
Communication mode	One way communication	Multidirectional communication

Figure 1.

Interface interactive reading.

Figure 2.

Spatial immersion reading.

2.1.1 The applied media of interactive reading

Print media

In the era of rapid development of new media, print-based reading creation has begun to find a more multidimensional extension direction on the basis of adhering to the essence of reading. New production materials have changed the original traditional paper material colour, volume, odor and other elements; new production process has changed the traditional printing ink and shaping method, replacing the traditional coloured body and connecting filler with modern spray printing substances, such as fire paint, hot stamping, fluorescent pigment, and heat-sensitive ink and improving the solid qualitative methods such as string, glue, and hard ring buckle in a freer direction; realistic multi-sensory interaction [9, 10, 11] enhances the reader’s experience from sight, touch, hearing, smell and taste.

Digital media

Interface interactive reading, also known as screen-based interactive reading, is now the most commonly used interactive medium in the digital age. The information receiver interacts with the reading material through touch screen, sound, joystick, body sway and other actions, and the screen or digital wall facing the receiver generates feedback to transmit the information quickly to the receiver through sensory stimulation (Fig. 1).

Figure 3.

Card reading category.

Unlike interface-based reading, Spatial Immersion Reading emphasizes the indirect effect of information feedback to the receiver after the interactive behavior of the receiver (Fig. 2). Spatial immersion prefers natural, original human states, such as breathing, footsteps, body shape, heartbeat, sight, and other non-compulsory and non-eliminable vital actions, to the intuitive and guided human-computer interaction of an interface. This type of digital media usually requires a closed and independent spatial display environment, with the help of ambient lighting, ambient sound, scent diffusers and other dynamic devices configured in the environment to create a three-dimensional interactive reading device [12].

2.1.2 Human-computer interactive reading technology in the new media era

New media is the fastest-growing, most influential, and most promising comprehensive art display media nowadays, and new technological means such as hologram, virtual reality, interactive projection, Internet of Things communication, and nano-core have emerged to provide more diverse and innovative ways for human-computer interaction. According to the reading and processing process of information, we have divided the human-computer interactive reading technologies into the following three categories [13].

Figure 4.

Sensor touch category.

Identification and switchover of information – Card reading category

The first step of reading information into the brain is to be identified. After the brain signal confirms the volume, content, and category of information, it is “packaged” and sent to the orderly network in the brain for subsequent actions, and the new media interactive reading forms a card-reading human-computer interaction technology based on this principle (Fig. 3).

Collection and association of information – Sensor touch category

Sensor-touch human-computer interaction technology is currently the most popular R&D area for new media devices. The principle of this technology is that the information receiver sends commands through its own actions and interacts directly with the information itself (Fig. 4). As a monitoring type of device, sensors are able to record measured data and convert them into electrical signals or other output formats according to set rules to realize the interface between human physical senses and control codes of new media devices. Sensors are widely used in everyday home appliances and are usually divided into a dozen categories based on their basic sensing functions such as photoelectricity, temperature and pressure. At present, the design field is more often involved with LIDAR, Kinect, and Leap Motion, and they are often used in combination with TouchDesigner, Processing, Unity and other interaction-driven software.

Information dissemination and application – Mobile device category

Mobile device category crosses the shackles between information carrying medium and medium, and it can quickly realize the reproduction and transfer of information, and can selectively complete the flexible exchange of public and private information. On the one hand, mobile devices can act as external extensions of reading devices, adding interactive solutions to the reading process through their own functions. On the other hand, mobile devices can also act as an extension of the information content structure to realize the multi-directional dissemination and regeneration of reading materials.

2.2 Interactive projection

The interactive projection technology is based on capturing the action data generated by the device on the corresponding projection image of the target object, and then issuing commands through the system analysis to make the projection image change, thus generating the action of the captured object. This motion data is combined with a real-time image interaction system to create an interactive and tightly integrated effect between the participant and the screen.

Projection mapping technology can date back to the late 1960s, when it was known as video mapping, spatial augmented reality, or coloured lights. By 2001, Microsoft [14] began experimenting with its research and development as a means of technological advancement. After selecting or creating the object to be projected, the physical structure of the entire virtual copy needs to be created, and a new trend in video mapping technology is now interactive projection mapping. Many research systems have demonstrated that the combination of projection devices and depth sensing cameras can turn almost any flat surface into a touch-sensitive display. In 2016, Xiao [15] and others proposed a haptic tracking system called DIRECT, which strategically fuses depth and infrared data from infrared depth cameras to greatly improve the accuracy of touch tracking (average error of 4.9 mm) and detection rate ( $>$ 99%), opening new horizons for the exploration of interactive projection on special touch interfaces. Yan [16] mentioned a desktop projector touch system based on projector and IR camera and IR light field. This system uses a camera to detect the light spot changes in the infrared field, and after calculation it can be converted into touch coordinates, thus enabling interactive projection. This system provides a more portable human-computer interaction, and participants are able to experience a more novel and intuitive interactive reading. Song and Cheng [17] proposed a vision-based projection interaction method that detects whether a touch-control event occurs on the projection screen by judging the degree of integration between the user’s finger and its projected shadow on the screen, a technique that avoids the extensive use of capacitive or resistive arrays. Participants can achieve natural interaction with the computer using only their fingers, which ensures reading enjoyment while greatly reducing production costs during interactive reading.

2.3 Recognition technology

Gesture recognition technology is the use of mathematical methods to understand and classify hand movements. Traditional gesture recognition requires human-computer connectivity based on a tracker worn on the user’s hand, while the emergence of holographic projection technology frees interaction from the use of equipment and the location of application is not restricted. Yuan Guan [18] elaborated a gesture recognition algorithm based on spatial location information of sensors based on spatial graph neural network, which can characterize the spatial connectivity between gesture data and solve the problem of temporal and long-distance dependence of gestures, making up for the low recognition accuracy caused by most of today’s gesture recognition methods that ignore the temporal and spatial connectivity of gesture recognition. The introduction of this technology will greatly improve the freedom and flexibility of participants in interactive reading, and make the interactive experience qualitatively better.

QR code recognition is already a very common recognition method nowadays, and it is mostly applied to cell phones and tablets on a daily basis. Scanning the specified QR code can obtain relevant information, and it is essentially to decode the coded information in the QR code. High-density coding makes the QR code information capacity huge, and it is dozens of times the information capacity of a typical barcode, and the coding range is wide; It can encode a range of content information such as text, numbers, images, and sounds. The application scope of such a practical identification method is becoming more and more extensive; in addition to information acquisition, it can also be applied to security and confidentiality, inventory, backup and other fields, its fault tolerance and decoding reliability is high. In the interactive reading process to increase the QR code recognition function is the use of the QR code recognition information acquisition function, to ensure the rapidity and accuracy of information delivery to the user.

3. The projection form of interactive reading

3.1 Concrete form – From reading to projection

The concrete form of reading refers to the retention of the reading process in a narrow sense, i.e., first intuitively receiving part of the reading material then following instructions for more behavior, and stimulating interaction with the projection through the reception of information. The reading process in the figurative form is: information reception – information association – interaction – information re-reception – information processing. In this process, the interactive projection technology is responsible for the presentation rather than the generation of content, which itself has a high level of maturity and requires the execution of the reader. Each reading material is a subset of the projection mapping, which does not rely on the interactive algorithm of the projection technology itself to find the core information generation solution, but rather on the method of extracting part of the information from the given reading material to achieve interactive projection reading.

3.2 Abstract form – From projection to reading

Interactive projection technology is responsible for the generation and presentation of reading information. Compared to concrete reading, the abstraction of the reading form makes the content of the information itself more important for the reader to expand through their insight and to master ideas, theories and concepts by generating effective reading material through the reader’s interaction with the projection. The process of reading in abstract form is “Interaction – Information reception – Information association – Information processing”. Using interactive algorithms to analyse and cluster unlabelled data sets, readers discover hidden information patterns or data groupings without the author’s directive intervention, and perform exploratory information analysis, cross-cutting content collection, and dynamic segmentation and reorganization based on similarities and differences in information. Since the abstract reading form requires the reader’s non-directive information behavior to trigger interaction, it is often combined with necessary daily human behavior, such as walking on the ground and dancing with the body.

4. Perception and generation of projection information

4.1 Perception type of projection information

The perception types of projection information are mainly divided into two categories: Chip-sensitive and Sensor-sensitive. Chip-sensitive projection (Fig. 5) does not support finger touch, but is equipped with an electronic pen-like object to click on the screen for interaction. A special chip, invisible to the naked eye, is embedded in the projected image so that the chip can sense and track the pen’s movement path with the electronic pen. When the user moves the pen on the screen, it is digitally reflected in the projected image according to a specific program that simulates the focus movement of the mouse. This type of projection is still essentially an extension of the interactive screen, without breaking away from the medium. Sensor-based projection has a camera or sensor that has a touch detection module pointing off-screen to the interactive area, and the detection module generates data when there is active information in the area and sends the data back to the projector. This type of interaction is usually direct touch or somatosensory interaction that occurs directly with the projection without the need for an indirect device and projection screen. This form is also the interactive projection technology that is discussed in this paper.

Figure 5.

Chip-sensitive Projection.

4.2 Technical components of the interactive projection system

4.2.1 Signal acquisition

The signal is equivalent to blood in the system, and only when the cyclic process of sending and collecting signal is completed, this system is considered operational. This part is responsible for capturing the human behavior, sending data information based on the range signal: the operating frequency of the array, the speed of wave propagation, the plane wave of the sensor (single element) or sensor array, the type of incoming wave and the weights of the different elements. Information acquisition instruments include infrared, ultrasonic, microwave, and radar-based motion detectors, as well as CMOS-based image sensors.

4.2.2 Data processing

This is the core part of the interactive projection system, equivalent to the heart, and it is the control centre of the whole system, which is usually done on a computer. The control centre analyses the data obtained from the signal acquisition part quickly, transforms the body action language from the signal acquisition according to the behavior rules developed by the software algorithm, and docks the analysed semantic command information with the projector’s image area in real time to realize the transformation of human behavior semantics and projector screen control commands.

4.2.3 Image display

The imaging part is the medium to which the projector screen is mapped, which can be either a common projection screen, paper or wall, or an irregular surface medium depending on the nature of the sensor.

4.2.4 Other auxiliary equipment

In addition to the basic interactive projection equipment, other devices will be added to be used in combination according to the function of the installation work, such as stereo audio, code sweeper, body-sensitive gloves and so on.

Figure 6.

Technical composition.

5. Interaction and feedback of projection information

Interactive projection is the use of a projector to project pictures, images and other visual effects onto a carrier. When the participant enters the coverage of the projection, the behavioural actions of the participant are recognized to stimulate the interactive picture effect and realize real-time interaction. Interactive projection information can be interacted with in three main ways: first, use the information to realize the interactive effect between the participant and the projection interface; second, use the camera to capture the transformation of the light source and the computer equipment to analyse and manipulate the projection information interface to achieve the interactive effect; third, use the induction switch facility to sense the change of the participant’s body to achieve the interactive effect. So, when the user touches the sensing point, the effect is triggered.

The interactive projection system is mainly divided into the following categories: wall interactive projection system, window interactive projection system, desktop interactive projection system, façade interactive projection system, ground interactive projection system, interactive back projection system, countertop interactive projection system, air flip book projection system, and air telepresence projection system. The specific presentation methods of interactive projection technology include collision interaction, water ripple interaction, erasure interaction, avoidance interaction, flip interaction, follow-up interaction and other forms [19].

The interactive effect is produced by installing an interactive induction screen on the carrier or projecting a programmed projection onto the carrier. The projected image is integrated with the surrounding environment and cooperates with background music to create a beautiful audio-visual feast, so that participants can switch between the dual senses of reality and illusion and be in the immersive space.

6. Experimental validation of interactive projection reading

6.1 Preliminary research

Prior to the research practice, a simple questionnaire was administered to 110 members of the general public, with questions asking about the general population’s level of knowledge and preference bias towards interactive reading. The choice of reading behavior was almost equally divided between print media, e-book media and new interactive media, with the number of print media reaching 30%, a slightly smaller percentage overall (Table 2). However, only 15% said they had experienced interactive reading, and 23% said they had not heard of this form of reading (Table 3). 20% thought interactive reading devices were rarely set up and often vandalized, and 14% had barriers to understanding how interactive reading devices interacted (Table 4). 80% of those respondents believed that interactive reading devices that are fun, thought-provoking, and innovative in the way they interact can spark the interest of the reader in the experience (Table 5).

Table 2
Favourite reading behaviours

Option	Number of respondents	Percentage
Print media	34	30.91%
Kindle e-book	41	37.27%
Html5 interactive type	35	31.82%

Table 3

Interactive reading experience

Option	Number of respondents	Percentage
Never heard of it	25	22.73%
Heard of it but never seen it	36	32.73%
Seen it but never tried it	33	30%
Tried	16	14.55%

Table 4

Reasons for low exposure to interactive reading devices

Option	Number of respondents	Percentage
Not interested	23	20.91%
Too many people, or open for too short a time or often damaged to participate	23	20.91%
The device is boring, not enjoyable, and participation does not provide the enjoyable experience as expected	22	20%
Comprehension barrier: interactive devices do not have clear signs	15	13.64%
Most public spaces in China do not have such a function; no similar habit	8	17.27%
The way to participate is cumbersome	10	9.09%
Other	9	8.18%

Table 5

Points of interest in interactive reading devices

Option	Number of Respondents	Percentage
Highly interesting	27	24.55%
Deeply meaningful and thought provoking	30	27.27%
Innovative interaction	31	28.18%
Attractive appearance	11	10%
Hands-on creation experience (e.g., participation in painting, playing a musical instrument and music)	11	10%

Through the analysis of the above questionnaire, it can be concluded that people are highly receptive to both traditional and modern media, and are more looking forward to the organic integration of media; the device should get a more scientific way of setting and be guided based on the current social cognitive degree; and people are still more inclined to participate in interactive reading for content acquisition rather than output, and it is important to do a good job of interactive connotation of the reading material itself.

6.2 Interactive technology development platform

In the material classification selection, we mainly start from the interactive software selection and hardware equipment selection. In terms of interactive software, there is the programming software Processing which is suitable for generating art, the algorithm building software Skyline which is based on code to generate music videos, and the new media design software Touch Designer which is used for real-time interactive creation, and the Japanese Team Lab team is good at using Touch Designer to create immersive art installations. For Html5 interactive animation, there are Flash, various commercial online platforms and the prototyping software Hype4, which has a timeline and effectors that can create pleasing animation effects on web pages and create rich interactive animations without coding. It doesn’t need coding, suitable for more rigorous book-like interactive interface production. Therefore, Hype4 is chosen as the production software for the presentation of reading materials based on the technical implementation.

In terms of hardware equipment selection, in addition to the paper materials needed for paper magazines, the new media technology also requires the use of virtual imaging, such as projection or VR, and interactive control devices, such as sensors, central processors, and so on. Taking the reading environment of interactive books as a starting point, the ease and flexibility of reading should be retained, so projection is a better choice of medium. By projecting the information content directly onto the print medium, the viewer can retain the reading habits of the original paper book without the need to wear a device. It is also more suitable for simultaneous reading by multiple people in the field of culture and education without the need to consider the number of appropriate instruments.

Human-computer interaction is the process of data contact between human and computer. In order to realize the reading behavior of human-computer interaction in a real environment, hand capture is the key. The commonly used gesture recognition devices are motion sensor-based data gloves, high frame rate mono-binocular camera systems, and infrared laser sensors. Data gloves acquire gesture-related data through motion sensors, which has the advantage of accuracy, but requires users to wear professional equipment, has a narrow scope of application, and is costly. The high frame rate mono-binocular camera system uses infrared camera to locate and recognize gestures, but it is less light resistant to the environment and cannot guarantee that the monitoring results of hand behavior can be provided accurately in real interaction [20].

Therefore, in this paper, an infrared laser sensor is used. Since the hand action in the reading operation is only a simple click-type operation without complex gesture judgment, the LIDAR sensor is chosen for the specific device. Radar touch technology can effectively send data information on two-dimensional planes such as virtual reality back to the control centre of the reading material. This enables the unification of the touch area with the content interaction to build a closed loop of information flow in a more economical way. Radar, as a basic device for multi-domain applications, is more compatible with commercially available hardware and software and can enter the commissioning phase more smoothly. In addition to this, a QR code scanner is used to simulate the effect of turning pages to achieve card switching information-based human-computer interaction.

6.3 Technical route

The technical route (Fig. 7) is divided into four parts: pre-preparation, technical strategy summary, technical practice, and testing and refinement. The pre-preparation is divided into basic research in both theory and design; after the research is completed, a summary of the technical strategy is conducted, emphasizing three parts: sense of innovation, flexibility, and experience; after that, the technical practice stage is entered, considering hardware and software adaptations from both interaction procedures and hardware connections, and presenting the effect according to the selection results, and conveying the action signals back to the device itself through interactive experiences: reading, clicking, and flipping. Finally, we enter the testing and refinement phase, and cross-reference with the strategy summary again.

Figure 7.

Technology roadmap.

6.4 Implementation process

6.4.1 Interactive content production

The reading information in the interactive projection of the atlas series named City Flowers of Beijing focuses on science promotion and educational connotation, and the dynamic effect production also maintains the overall unity and harmony of the picture. The display information is divided into two parts: Moonflower and Chrysanthemum, with different visual and interactive designs for different chapters to show the spirit of two types of Beijing city flowers.

The interactive click on the Html5 page is to build a new transparent button graphic on top of the original graphic in Hype4 to establish the interactive effect, and add a guide sign to prompt the reader. When the specified button is clicked, it will enter the corresponding new timeline and complete the set interactive behaviours, such as the text display of the target introduction, the dynamic effect of the screen, and the information switching of the data. To avoid the production lag problem, each page was made into a separate Html5 file, packaged and organized, and uploaded to the Tencent cloud storage bucket in a unified manner to complete the publication of local web pages, obtain the URL and its corresponding static QR code, and wait for the data call from the hardware device.

Figure 8.

Radar debugging process diagram.

6.4.2 Hardware debugging

Control pivot – Computer

As a control hub, it must be compatible with all devices and software, so a laptop with Windows system is used. As the hub, it is necessary to ensure the rendering of Html5 pages in the browser as well as the installation and application of each hardware engine.

Information output medium – Projector

The projector has no special model and function requirements, and is connected to the computer via HDMI cable to display the screen synchronously with the computer screen to realize the presentation of electronic reading materials projected on the paper book model.

Signal sensor – LIDAR

LIDAR needs to install the driver and engine on the computer. After installing the driver, find the radar serial number and connect it in the engine. Through the scanning sensing of the radar, drag the four red, yellow, blue and green points on the screen to coincide with the four corners of the paper book model imaging in the black area, so that the computer screen can be covered and aligned with the book model, so that the two-dimensional plane of the book model becomes an inductive area to replace the computer screen. Turn on the mouse function on the right side of the engine software window and select click only, after which the new signal scanned by the radar in the two-dimensional plane of the book model will become the mouse focus delivered to the computer hub (Fig. 8).

Information identifier – QR code scanning module

Commercially purchased QR code scanning module needs to be set up independently first. We will adjust the setting scanning mode for a single scan, and set the code system to QR code, and re-sweep the code and turn off power and restart in the setting driver to complete the initial settings. Download the c language development package from the official website of the device, get the application file from it and double click to open it directly to test the function on the PC for the acquisition path, open the software scan channel. ( $\backslash$ Txq_csharp_sdk $\backslash$ Txq_csharp_sdk $\backslash$ bin $\backslash$ Release $\backslash$ Txq_csh arp_sdk.exe).

It is found in the initial scanning results that the text information of the QR code, which is the URL of the QR code, can be scanned, but this work needs the QR code to automatically open the specified browser and jump to the web page after scanning, and does not affect the normal work of the mouse focus within the web page.

This leads to two solution ideas. 1) In the browser, fix the input cursor in the top address box and select the URL in full, while the mouse can still click in other locations on the page (or every 5s the mouse focus automatically returns to the address box and selects it in full). 2) change the output of the QR code scanning module c language development kit software, adding new commands. After several attempts based on the above two options, the problem was solved by option 2. Referring to the SDK c# development Txq_csharp_sdk project file, the Form1.cs file was modified in the visual studio2019 environment. After the modification, open the .sln project file and click on debug start to execute the generated executable. Use the module again for scanning to realize the jump command to open the web page automatically (Fig. 9).

Figure 9.

Flowchart of work interaction.

6.5 Scene construction and device testing

The scene of the interactive projection City Flowers of Beijing is built in a low light environment, with the computer as the control centre in the non-display area, connected to the projector, QR code scanner and radar device respectively (Fig. 10). The projector project downward from the desktop stand to transfer the digital screen image to the blank paper book model; the radar, which creates a virtual two-dimensional plane, was placed next to the book model to capture hand signals; the scanner was set up above the QR code booklet, and when the QR code booklet was flipped to switch pages, the scanner would capture the new QR code, causing the computer to jump to a new web page, and the image on the book model would then switch. The scene is also equipped with sound to play music to render the reading atmosphere (Fig. 11).

Table 6
Feedback from the Interactive Reading User Test of City Flowers of Beijing

User age	Fluency	Reading experience	Interest point
13	Low	High interest value but easy to touch by mistake	Moving pictures
22	High	Good interactive experience	Complete assembly of the devices
51	Higher	Good interactive experience	Non-electronic screen interaction
72	Lower	Not easy to see the content of strong light projection	Click interaction effect of content

Figure 10.

Schematic diagram of the scene construction.

Figure 11.

Display effect.

6.6 User testing and problems

6.6.1 User testing and feedback

After the work is completed and uploaded to the cloud, the Html5 version of City Flowers of Beijing is actually tested to obtain the feedback data of the interactive reading device after being used by users of different age groups (Table 6).

According to the test, it can be concluded that the interactive reading device has a high experience among the young and middle-aged groups, and their interest point focuses on the combination of projection technology and paper media information, not a single paper media reading or electronic screen reading; for the elderly and young children groups, the interest point is more reflected in the richer interactive experience compared with traditional reading.

6.6.2 Problems and solutions

Hardware and software selection

At the beginning of the experiment, the authors tested Makey Makey, Leap Motion and Kinect respectively. Makey Makey requires alligator clips and the connection of the actual circuit board, suitable for the device wall but not for multi-page paper reading media; Leap Motion because of the emphasis on the specific capture of finger movements, too complicated for the effect to be achieved in this paper, and it is difficult to find relevant guidance tutorials online, and the development is more difficult; Kinect also has the problem of earlier versions, and it is more inclined to the capture of overall human body movements and joints than 2D desktop reading, and is not sensitive enough to capture the subtleties of the desktop. Finally, after the relevant information is found, the choice of LiDAR is made. In the software selection, we have initially used Touch Designer for testing; in CHOP execute we have made conditional restrictions on video playback based on mouse coordinates; we have finally chosen the Hype4 software platform due to the issues related to multi-layer overlay, multi-page switching, and fixed board style.

Screen debugging

The paper book-mold has a certain thickness, and there will be a certain curvature when it is opened and laid flat, so some adjustments are made to the book-mold size, browser display ratio, and projector screen ratio. After the screen adjustment is finished, the radar simulation of the two-dimensional plane is also reconstructed to improve the accuracy of the interactive technology.

Display environment

The projection interactive reading device requires a reading environment that is not too brightly lit and needs to be conducive to focused reading concentration, but the device on the other hand also limits the conditions of the scene layout, the device needs to be displayed in a smaller closed environment, so we can replace the projector with a better performance in order to improve the brightness of the screen.

Interactive experience

Although the radar can sensitively capture the signal and react in the two-dimensional plane, it may also cause the occurrence of mis-touching, and the reader needs to try to click the finger perpendicular to the screen, and the image recognition ability of the radar needs to be further modified later.

General objective environment

The Html5 version of the atlas series Flower of Beijing City in the WeChat of Mobile End has been promoted for reading through the social platform approach, allowing visitors to remotely understand the entire display of reading materials, which is also a supplement and improvement of the work in terms of mobile device type human-computer interaction information dissemination and application.

7. Conclusion

In this paper, the interactive reading device based on interactive projection combines three technical forms of new media human-computer interaction: the manual page turning based on QR code recognition technology adopts card reading type, which can reduce the waste of print media and realize the sustainable development of the whole interactive reading device by switching the booklet content, changing or adding reading materials; the hand-touch information interaction realized by radar sensing technology adopts sensor touch type, which adds a diversified reading experience and is a re-enrichment of the intelligent form of interactive reading; the mobile Html5 version adopts mobile device type, which facilitates the wide dissemination of the content of the atlas Flower of Beijing City and provides a virtual network information extension for the interactive reading device loaded by physical projection.

As a complex cognitive process, reading learning requires communication and interaction to improve the reader’s cognition, and triggers new reading experiences as well as research on reading media based on interactive forms in the new era. Both paper media and digital media should find a more humane way out of expression in the current era of information explosion, and human-centeredness is always the core of reading methods and even the development of society as a whole. The interactive reading study of Flower of Beijing City is a comprehensive study based on multiple dimensions of plant science, technology science and art science. It has a practical application market and educational significance for all reading audiences with behavioral control, and can be widely used in specific educational and activity projects such as museums, exhibition rooms or teaching demonstrations in schools.

It is our “original intention” to express traditional ideas in a new and modern form, and gain information content more conveniently and vividly; it is our “ultimate goal” to avoid using electronic touch screen, but to use traditional paper as a projection canvas, and insist on the experience quality of reading paper magazines. “It is easy to set our original intention, but hard to keep our ultimate goal”. Human-centeredness has always been at the heart of the development of reading activities. The rapid development of the digital age has increased people’s new expectations and requirements for reading behavior. Interactive reading comes to people’s vision in a novel and interesting form. Promoted by new media technology, interactive reading starts from different media and different senses, and diversifies information from various aspects such as reading design perspectives, layers and dimensions, so that participants can produce reading behaviours such as physiological and psychological activities in reading materials.

Footnotes

Acknowledgments

The authors would like to thank the research team for their great cooperation, and are also very grateful for the funding and support from the Beijing Social Science Foundation Project (19YTB021), and sincerely thank the reviewers and the editor-in-chief.

References

Guo

. New Media “Post-Reading”: Content, Dissemination and Cultural Significance. Shanghai University. Shanghai. 2019; doi: 10.27300/d.cnki.gshau.2019.000018.

Ren

. Study on the Structure of Reading Methods in Network Era. Library Construction. 2013(10): pp. 61-66.

Makdissi

Boisclair

. Interactive reading: A context for expanding the expression of causal relations in pre-schoolers. Written Language & Literacy. 2006; 9(2): 177-211. doi: 10.1075/wll.9.2.02mak.

Guo

. Post-reading under new media: content, dissemination and cultural significance. Shanghai University. 2019. doi: 10.27300/d.cnki.gshau.2019.000018.

. An empirical study on the interaction role reading model and its effectiveness. Journal of Xi’an International Studies University. 2014; 22(3): 72-75+84. doi: 10.16362/j.cnki.cn61-1457/h.2014.03.028.

Luo

Zhu

. Interactive installation art: The influence of machine perception on contemporary art creation. Art Market. 2021(8): 104-105.

Gao

. Innovative design and new media art and practice of experimental center. Laboratory Research and Exploration. 2016; 35(4): 166-169+178.

Fan

. A study on the composition of reading experience. Library Forum. 2020; 40(1): 19-28.

Zheng

. The Application of Interactivity in Book Design under New Media Context. Editors Monthly. 2020; (1). pp. 94-98.

10.

Dong

, Multi-dimensional Reading Experience of Paper Book Design in Digital Era. Zhuang Shi. 2017; (4). pp. 136-137. doi: 10.16272/j.cnki.cn11-1392/j.2017.04.042.

11.

Zheng

Yan

. Exploration on the application of multi-sensory interactive experience in the editing and publishing of paper books. Chinese Editorials. 2020; (10). pp. 43-46.

12.

Liu

Chu

Leifang

. Research on the Expression of Immersive New Media Device Art. Art Education Research. 2022; (7). pp. 48-49.

13.

Zhan

. The Aesthetics of Technological Intelligence in New Media Interactive Device Art [Master’s thesis]. Fujian Normal University. Fuzhou. 2020. doi: 10.27019/d.cnki.gfjsu.2020.001380.

14.

Yoo

Kim

. A study on the media arts using interactive projection mapping. Contemporary Engineering Sciences. 2014; 7(23). p. 11817. doi: 10.12988/ces.2014.49147.

15.

Xiao

Hudson

Harrison

. Direct: Making touch tracking on ordinary surfaces practical with hybrid depth-infrared sensing. Proceedings of the 2016. ACM International Conference on Interactive Surfaces and Spaces. Ontario. Canada. 2016; pp. 85-94. doi: 10.1145/2992154.2992173.

16.

Yan

. An Infrared Light Field Camera based Projection Touch System. New Technology and New Products of China. 2021; (13). pp. 42-44. doi: 10.13612/j.cnki.cntp.2021.13.013.

17.

Song

Cheng

. Vision-based projection-camera fingertip touch system. Journal of Integration Technology, 2017; 6(1): 8-15.

18.

Yuan

Bing

Liu

Dai

Zhang

Cai

. Hand Gesture Recognition based on Spatio-temporal Graph Neural Networks. Acta Electronica Sinica. 2022; 50(4): 921-931.

19.

Luo

. Different manifestations of interactive projection. Science and Technology Innovation. 2018; (32): 80-81.

20.

Liu

Zhang

. Implementation of a media-less floating projection interactive system. Intelligent Computer and Applications. 2021; 11(5). 119-123+128.

21.

Hutson

Montilus

Herrick

, et al. Designing a Digital Reading Tool for Middle School Students and Teachers: A Grounded, User-Centred Design Approach. HCI International 2022 Posters: 24th International Conference on Human-Computer Interaction, HCII 2022, Virtual Event, June 26–July 1, 2022, Proceedings, Part III. Cham: Springer International Publishing. 2022; 27-34. doi: 10.1007/978-3-031-06391-6_4.

22.

Ben-Yehudah

Eshet-Alkalai

. Print versus digital reading comprehension tests: does the congruency of study and test medium matter? Brit J Educ Technol. 2021; 52(1): 426-440. doi: 10.1111/bjet.13014.

23.

Cao

Xiao

. Research on Digital Reading App Design to Stimulate Reading Motivation of Teenagers. Advances in Human Factors in Training, Education, and Learning Sciences: Proceedings of the AHFE 2020; Virtual Conference on Human Factors in Training, Education, and Learning Sciences, July 16-20, 2020, USA. Springer International Publishing. 2020; 63-69. doi: 10.1007/978-3-030-50896-8_10.

24.

Wang

Cheng

Liu

, et al. How do people read eBooks on mobile devices? Implication for digital reading environment design. Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries. 2018; 393-394. doi: 10.1145/3197026.3203872.

25.

Sietas

Hansen

Ali

, et al. Tangible Browsing for E-Books: Design and Evaluation of a Haptic Approach for Digital Reading. Proceedings of Mensch und Computer. 2019; 737-741. doi: 10.1145/3340764.3344900.

26.

Silva

Vieira

Amorim

, et al. Using Multisensory Content to Impact the Quality of Experience of Reading Digital Books. ACM Transactions on Multimedia Computing. Communications. and Applications (TOMM). 2021; 17(4): 1-18. doi: 10.1145/3458676.

27.

Nadarajah

Pedersen

Schnatterbeck

CGH

, et al. What Is the Cat doing? Supporting Adults in Using Interactive E-Books for Dialogic Reading. The Interplay of Data, Technology, Place and People for Smart Learning: Proceedings of the 3rd International Conference on Smart Learning Ecosystems and Regional Development 3. Springer International Publishing. 2019; 146-158. doi: 10.1007/978-3-319-92022-1_13.

28.

Iyare

James

Amonde

. The effectiveness of integrating interactive technology in reading comprehension: A case study of Jamaica’s grade school. Journal of Information Technology Education. Research. 2018; 17: 227-246. doi: 10.28945/4098.

29.

Fan

. Geometry Calibration Control Method with 3D Sensors of Large Screen Interactive Projection Imaging System. Proceedings of 2022 Chinese Intelligent Systems Conference: Volume I. Singapore: Springer Nature Singapore. 2022; 593-601. doi: 10.1007/978-981-19-6203-5_58.

30.

Deng

Luo

Tang

. Digital Design and Research of Ink Art Based on Infrared Interactive Projection Technology-Taking the Work of “Listen to the Ink” as an Example. Human-Computer Interaction. Recognition and Interaction Technologies: Thematic Area, HCI 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26–31, 2019, Proceedings, Part II 21. Springer International Publishing. 2019; 409-418. doi: 10.1007/978-3-030-22643-5_32.

31.

Chen

Zhou

, et al. A two-dimensional location method based on digital micromirror device used in interactive projection systems. Optoelectronic Imaging and Multimedia Technology. SPIE. 2010; 7850: 148-154. doi: 10.1117/12.870655.

32.

Van Damme

Legrand

Heyse

, et al. Effects of haptic feedback on user perception and performance in interactive projected augmented reality. Proceedings of the 1st Workshop on Interactive extended Reality. 2022; 11-18. doi: 10.1145/3552483.3556456.

33.

Kurata

Ishita

Miyata

, et al. Print or digital? Reading behavior and preferences in Japan. J Assoc Inf Sci Tech. 2017; 68(4): 884-894. doi: 10.1002/asi.23712.

34.

Schwabe

Lind

Kosch

, et al. No negative effects of reading on screen on comprehension of narrative texts compared to print: A meta-analysis. Media Psychol. 2022; 25(6): 779-796. doi: 10.1080/15213269.2022.2070216.

35.

Shu

Lin

. The finger-based interactive projection using a monocular camera. International Journal of Pattern Recognition and Artificial Intelligence. 2019; 33(14): 1954034. doi: 10.1142/S021800141954034X.

36.

Paulovich

Eler

Poco

, et al. Piece wise laplacian-based projection for interactive data exploration and organization. Computer Graphics Forum. Oxford. UK: Blackwell Publishing Ltd. 2011; 30(3): 1091-1100. doi: 10.1111/j.1467-8659.2011.01958.x.

37.

Pérez

Zhang

Schaefer

, et al. Interactive feature space extension for multidimensional data projection. Neurocomputing. 2015; 150: 611-626. doi: 10.1016/j.neucom.2014.09.061.

38.

Lim

Whitehead

GEK

Choi

. Interactive e-book reading vs. paper-based reading: Comparing the effects of different mediums on middle school students’ reading comprehension. System. 2021; 97: 102434. doi: 10.1016/j.system.2020.102434.

39.

Clinton-Lisell

Seipel

Gilpin

, et al. Interactive features of E-texts’ effects on learning: a systematic review and meta-analysis. Interact Learn Envir. 2021; 1-16. doi: 10.1080/10494820.2021.1943453.

40.

Clinton-Lisell

. Investigating Reading from Screens and Mind Wandering in the Context of Standards of Coherence. Sci Stud Read. 2022; 1-18. doi: 10.1080/10888438.2022.2125320.

41.

Wilujeng

Lestari

. Effect of interactive multimedia e-books on lower-secondary school students’ curiosity in a Science course. Educ Inf Technol. 2022; 27(7): 9619-9639. doi: 10.1007/s10639-022-11005-8.

42.

Spjeldnæs

Karlsen

. How digital devices transform literary reading: The impact of e-books, audiobooks and online life on reading habits. New Media Soc. 2022; doi: 10.1177/14614448221126168.

Interactive projection information generation and feedback for reading behaviour

Abstract

Keywords

1. Introduction

2. Related research

2.1 Interactive reading

Table 1 The difference between interactive reading and traditional reading

2.3 Recognition technology

3. The projection form of interactive reading

3.1 Concrete form – From reading to projection

3.2 Abstract form – From projection to reading

4. Perception and generation of projection information

4.1 Perception type of projection information

4.2.1 Signal acquisition

4.2.2 Data processing

4.2.3 Image display

4.2.4 Other auxiliary equipment

6. Experimental validation of interactive projection reading

6.1 Preliminary research

Table 2 Favourite reading behaviours

6.3 Technical route

6.4.1 Interactive content production

Table 6 Feedback from the Interactive Reading User Test of City Flowers of Beijing

6.6.1 User testing and feedback

6.6.2 Problems and solutions

7. Conclusion

Footnotes

Acknowledgments

References

Table 1
The difference between interactive reading and traditional reading

Table 2
Favourite reading behaviours

Table 6
Feedback from the Interactive Reading User Test of City Flowers of Beijing