Abstract
In professional audio production the setup of electronic musical devices is a time-consuming and error-prone process, as it involves manual operations like establishing local configurations and carrying cables for each device. Such is the case of MIDI (musical instrument digital interface), which is widely used in the context of musical applications. On the other hand, the capabilities of WSN (wireless sensor networks) allow developers to build up more complex applications, since nodes have the ability of autoidentifying, autoconfiguring, and establishing associations with other nodes, behaving in a smarter way than other networks. In this paper, we propose the use of an optimized WSN network for interconnecting MIDI devices. This network has been named collaborative musical wireless network (CMWN): it eases device configuration, enables musical collaboration, and allows artists to explore new ways of expression. The paper also presents the hardware and performance results of a prototype able to create CMWNs.
1. Introduction
The field of professional audio production refers to all those activities in some way related with the processing of sound using electronic means [1], that is, musical performances, composition and arrangement [2], artistic performances, and multimedia spectacles [3]. Technology plays an important role in the world of professional audio production. Several types of device are involved, such as microphones to record voices and musical instruments, audio processors to apply special effects (echo, delay, etc.), and multitrack mixers to synchronize audio and video in soundtracks for documentaries and cinema, amongst others. The quality of the artistic results depends on the selection of the device settings, which is usually done by a technician with special skills in music and technology, either by using cables or software.
Communication between musical devices is possible thanks to musical protocols, which can be classified into two categories: wave-oriented protocols and control-oriented protocols [4]. Wave-oriented protocols carry the sound data, either in analog or digital format, and contain all the information needed to play the sound. In contrast, control-oriented protocols are used to intercommunicate musical devices internally, usually to cause an action to be performed in response to the occurrence of an event. This is the case of the MIDI protocol, in which the information transmitted needs to be interpreted by a synthesizer before an audible waveform can be obtained [2]. The combination of both wave-oriented and control-oriented protocols allows artists to create a complete multimedia performance through the interaction of various multimedia nodes, which form a multimedia network. Control-oriented protocols have the advantage of being much easier to process because the musical events are explicit. For instance, with wave-oriented protocols a signal processing analysis algorithm like the one needed to detect sound frequency [5] can work incorrectly due to the presence of different musicians playing simultaneously in live concerts, as the harmonics of their instruments are mixed with the noise. However, control-oriented protocols like MIDI detect the sound immediately, simply by reading the corresponding value of a MIDI message.
Wireless sensor networks (WSN) are mainly used to collect data from a set of sensors distributed over a wide area without the need for a physical structure [6]. By design, these sensors are inexpensive and low-powered and they are applied to fields like agriculture, health, industry, and so forth. Typically, the sensors form ad hoc communication networks able to self-organize and auto-configure their nodes. These features can be taken into account for professional audio production, where, as we have seen, it is necessary to interconnect and configure a large number of musical devices.
By using WSN to provide connection support for control-oriented protocols (like MIDI) we can obtain many benefits and new possibilities. For example, in the traditional MIDI connections devices are classified as masters or slaves, with masters being responsible for initiating a communication. However, in WSN any device can initiate communications, providing greater flexibility. Additionally, WSN offers the possibility of autoidentifying, auto-configuring, and associating devices, allowing designers to provide smarter musical applications. The main limitation of WSN could be the low bandwidth usually available, but this does not represent a real problem for this application, since control-oriented protocols are not highly demanding in terms of bandwidth.
Therefore, this article proposes the use of WSN technology for interconnecting musical MIDI devices. This kind of network, named musical collaborative wireless network (MCWN) allows nodes not only to communicate wirelessly, but also to auto-identify and associate with other nodes in order to perform collaborative activities. By collaboration we refer to any association between two or more devices, such as a score digitalization using a musical instrument and a score editor, or a light show for a live concert, where instruments and lighting system have to be synchronized. As a proof of this idea, the article also introduces a preliminary prototype of an adapter that enables any MIDI standard device to join an MCWN network easily.
The article is structured as follows. Section 2 defines the requirements of the system to be developed. Section 3 gives a detailed description of the background of multimedia protocols, MIDI devices that can form part of MCWN networks and commercial wireless MIDI transceivers that could be used to implement some of the ideas put forward in this article. Section 4 is devoted to describing the MCWN network architecture, explaining how to extend a standard WSN network to support collaborative activities. Section 5 presents a prototype that allows any standard MIDI device to join an MCWN network and test basic collaborative activities. Section 6 describes real scenarios that could benefit from the inclusion of collaborative features in their normal operations. Finally, Section 7 is dedicated to conclusions and future work.
2. Problem Statement
This article describes a WSN-based system that enables electronic musical devices to establish collaborative communications. Our system has been designed according to the following requirements, which should be present in an ideal collaborative wireless interface.
MIDI compatible. The ideal system must be compatible with the multimedia protocols implemented in most academic/commercial devices, otherwise its adoption in real scenarios would be limited. MIDI is a good option, as it has been widely adopted within the musical industry. Transparent operation. The communications system should be able to transmit the data through a wireless network as if the musical device was using its regular nonwireless interface. Collaborative activities. Communications among all the nodes are possible, allowing the intelligent association of two or more of them in order to carry out collaborative activities, such as a light show for a live concert. This requirement contrasts with the limitations associated with the master-slave communications model used by multimedia protocols such as MIDI. Wireless communications. The use of cables can be problematic in certain cases, such as when assembling and disassembling musical devices on stage. Moreover, wireless devices are more user friendly, since they allow for automatic configuration and greater freedom of movement for the musician. License-free band. The use of a license-free band is almost mandatory, since it reduces the expenses associated with the use of the technology. Coverage extension. In some scenarios it would be necessary to extend the coverage to a larger area using relay nodes. Bidirectional communications. Commercial wireless adapters only allow one-way communications, thus limiting the range of the activities to be performed within the multimedia network (e.g., devices could report problems in the configuration or malfunctions). Auto-identification and Auto-configuration. Traditionally, the setup of multimedia systems has been performed manually using cables and local configurations. With a system of the characteristics proposed, the automatic configuration of the devices is possible, since devices can have enough knowledge about other devices within the network to make intelligent decisions as they are switched on. Low latency. Musical performances require low latency devices since they occur in real time. Furthermore, sound delays can lead musicians to reject the use of the interface.
In the following section (Subsection 3.2) currently available interfaces are analyzed in order to determine whether they fulfill the previous requirements.
3. State of the Art
3.1. Traditional Multimedia Protocols versus WSN Protocols
MIDI, which stands for musical instrument digital interface, is one of the most widespread protocols found in the field of professional audio production [2]. It was designed by a group of manufacturers under the name of MMA Association [7] in the late 80s with the main aim of ensuring compatibility between their products, something that until then had proved almost impossible because each manufacturer had proprietary connectors, cables, and protocols. As time went by, MIDI became very popular among musicians and was extended to support more complex scenarios. Although nowadays several academic and commercial alternatives can be found (e.g., OSC [8], mLAN [9], or HD Protocol [10]), MIDI remains the most popular, as it is the protocol being used by most of the musical instruments, devices, and software currently available.
MIDI is a multimedia protocol characterized by being event oriented, state based, unidirectional, and master-slave. MIDI does not carry any sound, but it does carry the information that a synthesizer can use to produce sound. MIDI was initially designed to solve the basic problem of intercommunicating one keyboard with one or more synthesizers. Events are represented by MIDI messages, which are created whenever a musician generates an event during his/her performance.
MIDI can be regarded as a multimedia network protocol. Figures 1 and 2 show typical connections between one sequencer and several synthesizers. MIDI network topologies can be said to act like unidirectional one-to-one communications: data travel only from the master to the slave. To achieve bidirectional communications, two MIDI cables are required. It is possible to interconnect several slaves to one master by using the so-called “daisy-chain” technique (shown in Figure 1) or a MIDI Hub (shown in Figure 2). In the latter case, the data sent by the master is received by all the slaves, while the same MIDI cable is shared by all the devices. Also, note that MIDI defines channels, which allows synthesizers to respond to only one specific channel.
Network topologies available in MIDI technology (the synthesizer is an E-Mu Xboard49, while the sequencer is an Akai MPC2500).
Network topologies available in MIDI Technology (the synthesizer is an E-Mu Xboard49, while the sequencer is an Akai MPC2500. The MIDI hub is an M-Audio MidiSport 4 × 4).
Although bidirectional communications are not strictly necessary, they can improve user experience, as the master can receive feedback from its slaves to facilitate maintenance operations (e.g., slaves can report hardware failures), to enable device-discovery within a MIDI network or even to solve configuration problems (e.g., to fix incorrect MIDI channel configurations or to set up a synthesizer). It must be said that this feature is already used by other multimedia protocols, for example, DMX512 [11].
The difference between WSN and MIDI network topologies can be observed when comparing Figures 1 and 2 with Figure 3, where examples of star and mesh network topologies are shown. WSN networks try to maximize the number of possible connections between network nodes, leading to more complex topologies such as the aforementioned star and mesh topologies. In star networks, all nodes interact with one central node, which is called the coordinator. Unfortunately, the area covered by star networks is restricted by the transmission power of the coordinator. This problem does not exist in mesh networks, as all nodes within the network can provide network connectivity, not only the coordinator. Thus, the area covered by mesh networks is wider than in a star network.
Network topologies available in WSN technology: star network and mesh network.
WSN technologies offer new possibilities for music applications: high interconnection capacity, bidirectional low-powered communications, low cost, and a communication range in 2.4 GHz of up to 120 meters (depending on the environment, but usually enough for a live concert stage). Due to these characteristics, WSN-based music applications emerge as an interesting field, whose optimization is critical due to tight restrictions on bandwidth and latency, especially when performing live music.
Although MIDI has its own specific connectors (the well-known DIN-5), the fact is that MIDI protocol is independent from the transmission media being used to interconnect the musical devices. In fact, the MMA Association released documents dealing with USB, IEEE 1394 (or FireWire) [12], Ethernet [13], and RTP (Real-Time Protocol) [14] transmission media to use with MIDI. As detailed in Section 3.2.2, there are also wireless MIDI interfaces available on the market. Such interfaces are wireless replacements of one-to-one MIDI communications and therefore do not allow the one-to-multiple node communications that would support collaborations between several musical devices.
The features of multimedia WSN networks also include the fact that the intercommunication and integration of devices become easier. Moreover, a multimedia WSN network would extend communication to other devices, since it removes the natural boundary of cables and connectors, and thus facilitates collaboration between musical and nonmusical devices. For instance, it is straightforward to translate musical notes into light movements and colors (e.g., [15]), helping artists find new ways of expression.
3.2. MIDI Devices
3.2.1. Generic MIDI Devices
The MIDI protocol was initially conceived to support musical keyboards and synthesizers but, as time went by, the range of devices that could be connected to a MIDI network increased, extending to a high proportion of all the currently manufactured electronic musical devices.
Available MIDI devices can be divided into several groups. The first group is called “instrument-like” devices, and includes any MIDI device that emulates, at any level of detail, the physical appearance and touch feeling of an existing instrument. Examples of such group are the Yamaha WX5 [16] (a monophonic wind controller with a fingering similar to a flute, clarinet, or saxophone), the AKAI EWI4000S [17] (another wind controller), the Steiner MIDI EVI [18] (a trumpet-style wind controller), and the Morrison Digital Trumpet [19] (a brass-style controller designed by Steve Marshall with the Australian multi-instrumentalist James Morrison). The first group also includes novel and creative musical instruments that do not emulate physical instruments, but nevertheless bear some resemblance to traditional instruments. Examples are the Zendrum ZX [20] (a percussion instrument) or the Sonalog Gypsy MIDI [21], a performance instrument for controlling MIDI music through motion capturing.
The second group is the “Music-processing” group, which includes any device that deals with music as a piece of information, producing some sort of music as an output. One typical example is a sequencer, which is able to record musical events (a sequencer assigns a timestamp to each event and stores it into a file and vice versa, being able to reproduce previously recorded time-stamped events). A score editor can be regarded as a kind of sequencer with graphical abilities that shows a graphic representation of a score in conjunction with additional information. Examples of score editors are MakeMusic Finale [22], Sibelius [23], or GVOX Encore [24].
The second group also includes devices that are able to compose music by following several composition rules. During the Baroque period the keyboard player was given a score with a bass line with certain numbers annotated with the objective of completing the given line with additional notes, respecting the harmony [25]. Nowadays, technology allows us to build a harmonizing device able to complete the bass line automatically in the same way live musicians used to.
Finally, there exists a third group known as “synthesizers,” which includes any device that transforms music information into something that can be perceived by one or more human senses. The most obvious example is music synthesizers: devices that generate audible signals as a response to MIDI events. Light synthesizers, firework firing systems, and water machines fall into this category as well.
Of course, all these categories are not mutually exclusive: in practice most devices implement characteristics of two or even three of the categories mentioned.
3.2.2. Wireless MIDI Devices
In this section, the features of some of the most relevant wireless devices currently on the market are analyzed in order to determine whether they are able to satisfy the requirements for constituting a musical collaborative wireless network such as that proposed in this article.
The first of these is the MIDIJet Pro Wireless MIDI [26], a low-latency 2.4 GHz wireless transmission system with a range of up to 20 meters. With a set of alkaline batteries it lasts about 30 hours, giving it excellent autonomy, but it does not fulfill two of the requirements mentioned in Section 2 (3 and 7).
It can only work as a transmitter or as a receiver, but not at the same time. It only admits up to 31 transmitter/receiver pairs at the same time, so would be inappropriate for use with large orchestras.
Furthermore, the device is quite bulky, making it hard to use with certain instruments.
Other examples of 2.4 GHz transmitters are the M-Audio MidAir Wireless MIDI Interface [27] and the CME WIDI-X8 Wireless MIDI System/USB Interface [28]. The M-Audio MidAir Wireless MIDI Interface works in half-duplex mode and has a range of up to 10 meters, which is quite small compared to the range achieved with the CME WIDI-X8 Wireless MIDI System/USB Interface (up to 80 meters with LOS (line of sight)). The latter system is only powered by two AA batteries, and allows full-duplex communications with up to 64 MIDI channels. There is another version of the CME transceiver, the CME WIDI XU Wireless MIDI Interface [29], which has been designed to work with computers acting as a wireless USB-MIDI adapter. The main drawback of these three transceivers is that they are limited to acting as a mere wireless replacement of a one-to-one connection, so there would be no possibility of collaborative work among multiple devices (requirement 3 in Section 2). Moreover, such devices are not able to extend the network coverage using mesh techniques (requirement 6).
M-Audio also sells the M-Audio Mid Air 25 Midi Wireless 25-Key Keyboard [30], a small music keyboard MIDI Controller with an embedded wireless transmitter. This product offers basic MIDI-editing features, although its high current consumption (6 AA-batteries that last an average of 2 hours) is a significant limitation.
Another commercial system worth mentioning is the MIDIStream Wireless MIDI System [31], a specialized UHF (ultra high frequency) communications module for transmitting from a musical instrument to another device, reaching a maximum distance of 80 meters outdoors and about 30 meters indoors. The transmitter is relatively small (the size of a pack of cigarettes) and is powered by a 9 V battery. Unfortunately, this module acts as a replacement of one-to-one MIDI communication, and thus collaborative intercommunications among multiple devices are not possible (so it therefore fails to fulfill requirement 3 of Section 2). Moreover, this device does not offer mesh network capabilities, so it is not possible to extend coverage through this mechanism (requirement 6).
A comparison of the most relevant features of the interfaces previously described is given in Figure 14, which also shows the main characteristics of our prototype (detailed in Section 4).
It is important to note that none of the wireless MIDI devices studied allows one-to-multiple node communications, as they all have been designed as transparent replacements of the original MIDI one-to-one communications. Only WIDI XV-8 permits a restricted one-to-multiple communications, allowing the user to switch among different slaves by pressing a button. Observing such lack of support for collaborative musical activities, we decided to implement our own prototype for performing the required one-to-multiple node communications. This prototype is called wim, making reference to “wireless musician,” and can be defined in a nutshell as a WSN-based MIDI interface.
4. Architecture Description
4.1. Architecture Description
Figure 4 presents the architecture of a musical collaborative wireless musical network (MCWN), a specific WSN especially designed to provide optimized communications between musical devices, thereby enabling collaborative activities among multiple participants. Our definition of collaborative activity includes any association between two or more devices, regardless of its complexity. For instance, a communication between a MIDI controller and a score editor is a good example of a simple collaboration. More complex applications are described in Section 5.
Collaborative wireless musical network.
A star topology seems to be the most appropriate for musical applications since they typically use one-to-multiple (e.g., one keyboard to multiple synthesizers) or multiple-to-one (e.g., multiple instruments and one sequencer) communications schemes. This poses no problem for most WSN transceivers, because star and mesh topologies are in general available by default. For instance, the IEEE 802.15.4 standard [32] supports the star topology, whilst ZigBee [33] supports the mesh topology.
Our MCWN network is composed of two types of nodes.
The terminology we use is inherited from the field of WSN, where a central node (coordinator) is able to communicate with all the other nodes that form the network (end devices). It is useful to distinguish between both types of node in order to improve wireless communications performance. Also note that, at this point, the terms master and slave make no sense since one node can freely initiate communications with any other node.
Figure 4 also shows the complete network stack after adding a new network layer called collaborative support layer (CSL) at the top of a typical WSN stack. This special layer deals with device identification, communication, and association and, therefore, the ability to perform collaborative activities. To enable all these features a software library has to be developed. Such a library has to offer a list with several device categories (score editing, sound generating, etc.) and the corresponding set of operations each device is able to perform. The idea is similar to the ZDO Library used in ZigBee [33], which provides an easy configuration of wireless devices. In addition, devices could be programmed to behave in a smart way to help users detect hardware malfunctions or configuration errors (e.g., an incorrect MIDI channel setup).
The CSL is an extra layer explicitly added by us since it is specific for the application and, therefore, is not included in the default WSN stack. The CSL layer is placed between the network layer (NWK) and the host application. Ideally, communications with the host application are transparent, regardless of being connected to an MCWN network. Thus, it is possible to reuse existing designs with minor changes.
The operation of the network from a node perspective is as follows.
Network connection. The musical device is switched on and joins the wireless network. Musical device discovery. The musical device searches for other musical devices within the network. Definition of collaborative activities. The musical device determines which collaborative activities are feasible depending on the potential collaborators found in step 2. The device asks the user which collaborative activity it would like to perform. The device associates with one or several musical devices within the musical network to perform the collaborative activity. The collaborative activity is performed. When the collaborative activity ends, go back to step 2.
4.2. Collaboration Paradigm
Collaborative activities among different musical devices are not usually offered as a feature by MIDI devices due to the previously mentioned limitations in MIDI communications. Section 5 shows the benefits of performing such collaborations in several real scenarios. In contrast, this section presents a new musical instrument based on the concept of collaboration and WSN, which could be regarded as a paradigmatic application of the ideas expressed in this article.
The instrument uses concepts of location algorithms and modular synthesis. The idea consists in deploying a set of simple nodes, which collaborate with other nearby nodes to play music in real time. The performers could be, for example, visitors to a science museum or a contemporary art museum. Visitors would move the nodes within the exposition room, thereby changing the associations between the nodes. Also, the nodes could include some sort of basic controllers (buttons, sliders, touch screens, etc.) in order to allow real time control of synthesis parameters, for example, general tempo, filter depth, vibrato amplitude, or just note frequency.
Figure 5 shows a simple deployment of such an instrument. For the sake of simplicity, only three different types of node are used. The red node is the general input of the whole network and is the only node that cannot be moved by the visitors. The other two types are as follows.
Wave-node, which performs a note selected by another node. The waveform can be customized and, thus, the resulting sound will have a different timbre. Sequencer-node, which generates a random sequence of notes as output, with different duration and pitches.
WSN-based collaborative musical instrument.
When a sequencer-node associates with a wave-node, the notes generated by the sequencer-node are played in real time with the sound configuration selected at that moment in the wave-node. The association between nodes only occurs when visitors move two nodes and thus reduce the distance by up to half a meter, for example. Distances can be estimated by using the correlation between distance and the received signal power.
Figure 5 shows a scenario where the sequencer is moved from one point to another, so the sequencer-node is first associated with wave-node A, and then with wave-node B. Visitors can notice the change of association easily, as the notes being played have changed in timbre. Of course, more advanced types and interactions can be implemented, but the general idea of collaboration is still valid.
In [34] four scenarios are shown where WSN-based creative art was previously exhibited in two museums, namely, the National Museum of History and National Palace Museum, both in Taiwan.
Smart Museum [35]. It uses a ZigBee WSN tag-based system to identify museum visitors (age, sex, day of the visit, etc.), adapting the content of the presentations in order to encourage learning by challenging them with interactive games. Also, the data collected by the WSN network are used to grow an “e-flower,” and thus make the visitor part of the artwork. One Million Heart Beats [36]. As in the case of the Smart Museum, visitors are given a ZigBee tag, which also acted as a game controller. In the first phase, visitors took control of virtual sperm fertilizing ova; the sex, blood type, and Chinese astrological birth sign was determined based on the combined action of multiple participants. In the second phase, users held a bottle with an embedded WSN node that collected a heartbeat for the growing fetus. The final resolution of the artwork is determined after one million heartbeats are collected. Interactive WSN-bar [37]. A WSN network receives environmental data from outdoor sensors: brightness, temperature, and CO2 density. This information is used to make interactive flowers on a bar. Also, a WSN locating system detects the participant's movement, being used to animate the flight of butterflies among different bushes, according to the participant's moving between different rooms. iFurniture WSN in a community [38]. In this exhibition there are three types of furniture: iBoxes, iChairs and iTables. Each user that joins the artwork holds an iBox, which is a tag that stores his or her profile (gender, personal hobbies etc.) and mood. When the user sits on the iChair, the iBox transmits his or her profile and mood, so the iChair displays LED colors and images matching the user's information. When people gathered together in the same place have common interests, the iTable not only displays a visual representation, but also plays music, allowing people to meet and start a conversation.
5. Prototype
Figure 6 shows the prototype we have built to test the WSN-based collaborative paradigm proposed in this paper. This patent-pending design [39] is based on the popular Arduino board [40], uses a Digi XBee ZigBee module [41] and has three types of connectors: MIDI-in, MIDI-out, and USB. Additionally, a special serial-to-MIDI driver was developed in order to connect the prototype to a PC that ran professional music software. The prototype was used to perform different tests.
Multiuser tests. The implemented scheme is showed in Figure 7, where a small MCWN network was deployed. All nodes are Arduino-based with an XBee module. Nodes A and B played the role of musicians, each playing two different musical excerpts using the MIDI protocol, making it necessary to develop a small program. On the other hand, node C was configured as Power consumption tests. Table 1 shows prototype consumption, measured with a multimeter. As can be seen, power consumption depends on the node state, for which there are three different possibilities: sleeping, awake and idle, and awake and transmitting. It is worth mentioning that the consumption values shown can be easily reduced since some of the electronic components of the prototype are not strictly necessary. For instance, the design includes two processing units, an AVR microcontroller at the Arduino board [40], and an EM250 microcontroller at the XBee ZigBee transceiver [41].
Power consumption test.
Prototype design.
Multiuser test.
Given the above results, battery duration can be easily calculated. For instance, in the case of a 1,200 mAh lithium battery of 9 V, node autonomy is 10.93 hours in the worst case. It is important to note that power consumption in the awake and transmitting state is not constant: each transmission causes a peak of power consumption that lasts a very short time, thereby extending the calculated autonomy. Also, if an application does not need a continuous stream of music data, it is possible to allow the node to enter the sleep state, thus saving even more energy.
Coverage tests. This set of tests consisted of detecting the mean RSSI (received signal strength indicator) value obtained during the reception of 100 messages. Each measure was averaged five times for each experiment, to counteract the signal fluctuations caused by indoor fading. Both modules transmitted with a power of 3 dBm and boost mode enabled [41]. The results obtained are shown in Tables 2 and 3. Note that the total attenuation is the sum of the losses due to free space propagation and the existing obstacles [42]. Latency tests. Latency in music systems is defined as the amount of time elapsed between the emission of a note execution request and the actual execution of such note. Latency is especially important in activities that demand real-time responses, such as live concerts. In these tests, we have focused mainly on analyzing the latency caused by wireless communications (i.e., the total communication time) when sending MIDI messages of different sizes. Although the measurements performed are significant, we should point out that further study is required in order to take into account all the factors that may influence the data transmission.
Signal loss in free space.
Signal loss with different obstacles.
Figure 8 represents the testing scenario for the latency tests. An Arduino Mega board [43] (which is completely compatible with the Arduino Diecimila we used for the prototype) was used because it facilitates the measurement process: it has four serial interfaces, more than enough to interconnect the two XBee modules needed to perform the latency experiment. A USB connection was used to allow wireless data to be sent from and received by a computer, where a Java-based application was run to measure the communications latency. The results obtained are shown in Tables 4 and 5. Depending on the test the communication direction varied (from a ZigBee relay to a ZigBee coordinator, and vice versa), as did the number of bytes transmitted (from one byte to three bytes, the latter being the typical length of a MIDI message). The distance between the two nodes was one meter, the wireless modules were configured in AT mode and each test consisted of sending 100,000 messages of one of the two lengths specified.
Total latency values measured (1 byte).
Total latency values measured (3 bytes).
Latency test.
One of our concerns was that the underlying WSN protocol could introduce timing differences, as ZigBee was not initially thought to be a real-time protocol, but the results show that there are no significant differences between the measurements in both communication directions. Values are very stable, although sometimes there is a large delay in certain measures, thus causing a high variance. The cause of this behavior is being studied by the authors, and may be motivated by external factors, such as intentionally delayed transmissions as a part of optimization techniques, or even issues related with Java timing management.
Finally, although latency values are not very high for simple applications, they would definitely have to be lowered when working in more complex scenarios. To optimize such latency, it is important to take into account the configurations available in the XBee modules and the inner workings of the ZigBee protocol. Similarly, it is important to note that the results shown have been obtained without performing any optimization, as this was not their initial objective.
6. Applications
In this section five different applications for the proposed scheme are discussed.
Score management: the Stage rigging: a series of multimedia devices (lights, smoke machines, etc.) placed on a stage communicate with each other through the Computer-aided learning: technology can help music students to improve their musical skills by means of electronic musical instruments. Sensors can also be useful in fields where traditional education is not very effective, such as music theory, harmony, counterpoint, ear training, and certain aspects of instrument playing. Computer-aided score edition: this application is aimed at digitalizing manuscript scores. The notes can be obtained in real-time from a group of musicians, speeding up the score edition process (the inputting of notes using a keyboard and mouse is a really time-consuming task). Another application is Virtual Musician, where we would build up a musician that is equivalent in functions to a human musician, but with the advantage of being available any time, without agenda restrictions. It can play with other human musicians, adapting its performance as required by the musical discourse (tempo, dynamics, etc.).
6.1. Score Management in Musical Ensembles
One typical problem in large musical ensembles is the management of musical scores. Wireless nodes can collaborate among themselves to facilitate the management of such musical scores. This idea is represented in Figure 9, where an interactive device, like an iPad, is placed on each musician's stand and on that of the conductor. Such devices integrate the Score annotation exchanged between musicians. For example, violins must follow the bow indications marked by the leader. Thus, the leader would write the bowing in the electronic score and would share it with the rest of the violinists. Individual score annotations. Each music stand can save annotations that, although having a personal meaning for a particular musician, may be shared with other colleges for their personal use. For example, a flautist could add a “more piano” entry to a given passage, because he or she considers that it is important to not to break the sound balance with the other sections. These annotations could be shared with other musicians within the orchestra, or even with the flute students in their charge. General score annotations. They are particularly useful to the conductor, as he or she can mark rehearsal sections, tempos, and dynamics relating to the whole orchestra directly on to the score. The collaborative network will take responsibility for announcing those annotations. Additional features. As the score is in digital format, it is possible to offer additional features that may help the musicians or the conductor in several situations. For instance, a musical excerpt could be displayed in different clefs, which is very useful when working with transposing instruments. Transposing instruments are those for which the written and played sound does not match, as happens with the B-flat clarinet, for instance, which sounds one tone lower than written. Visual transposition is especially useful for conductors, music arrangers and composers, and sometimes even for instrument players.
Score management in musical ensembles.
6.2. Stage Rigging
Figure 10 shows an MCWN network where several multimedia devices (stage lights, lasers, smoke machines, etc.) are interconnected in a live performance. In this scenario, it is necessary to synchronize a large number of multimedia devices in order to make them “dance” with the music, which means that even musical instruments can join the MCWN network in order to set the main tempo (e.g., guitar, bass guitar, or battery). It is particularly easy to determine the tempo by monitoring percussion instruments.
Stage rigging (The software shown is Capture Polar [44]).
In this specific application, it is important to emphasize the advantage of using wireless interfaces, as it greatly facilitates the assembling/disassembling of the orchestra stage, which can be repeated many times if the orchestra is on tour. In addition to making it easier for stagehands to deploy the multimedia network deployment, configuration is much simpler because devices can self-configure and even report abnormal situations to the technicians. Finally, musicians also have greater freedom of movement on stage, thus benefiting the show.
6.3. Computer-Aided Learning
Computer-aided learning can be greatly benefited by the close relationship between music devices and computers, which leads to a wide range of possibilities for interaction between students and smart virtual teachers (SVT). One advantage of computer-aided learning when compared to traditional systems is the possibility of a more effective training in practical subjects such as music theory, harmony, counterpoint, ear training, or instrument playing. For instance, there are pianos with a special circuit capable of emitting light in the correct keys to guide students, helping them to play difficult passages. Such a piano could also be able to interact with software that would challenge the student to perform short musical excerpts, as if it was a video game (shown in Figure 11).
Computer-aided learning (The software shown is Synthesia [45] (left) The pictures shown in are under Creative Commons Attribution license. Their author is woodleywonderworks (right)).
6.4. Collaborative Computer-Aided Score Transcription
The transcription of music scores to electronic versions using keyboard and mouse is a very slow process due to the abstract nature of music notation. Figure 12 proposes a scenario where an MCWN network is used to collect notes played by a string ensemble using pickups and sensors capable of detecting the music performance. All those notes are sent to music transcription software, which is able to translate the musical performance to an electronic score, with a certain error rate. This collaborative activity enormously accelerates the transcription of musical scores. Score editing software like MakeMusic Finale [22], Sibelius [23], and GVOX Encore [24] could easily integrate music transcription through a collaborative network, saving the musical data using their own score formats.
Collaborative computer-aided score transcription (The software shown GVOX Encore [24] (left). The pictures shown in are under Creative Commons Attribution license. Their author is Ravpapa (right)).
6.5. Smart Virtual Musician
Technology allows us to have a virtual substitute for a musician, either for individual rehearsals, or even for public concerts. That is the idea represented in Figure 13, where a virtual musician plays a trumpet concerto with a human pianist. The idea of using a robot as a metaphoric representation of a synthesizer is very interesting, since we visualize a synthesizer as a smart musician that can be asked to play any instrument.
Smart virtual musician (The pictures shown in are under Creative Commons Attribution license. Their authors are Fundação Cultural de Curitiba (left) and Chris 73 (right)).
Devices proposed to create the prototype (*no information given by the manufacturer).
The advantages of using virtual musicians are clear for anyone accustomed to the scheduling problems of musicians. For instance, music students often have agenda issues, even in small ensembles (duets, quartets, etc.) and sometimes parts of orchestras are unbalanced due to the lack of musicians of a certain instrument. These problems can be partially solved by using smart virtual musicians (SVM), who can play any musical instrument and are available at any time of day. Although the use in professional environments requires the work of skilled technicians, their use for group music rehearsal is possible and relatively easy. The benefits of SVM musicians are clear, as they serve to get an overview of the general composition with no human musicians, thus saving time.
7. Conclusions and Future Work
This article has studied the possibility of using WSN as the technological basis for interconnecting various MIDI-compatible musical devices. WSN topologies make it possible to increase the number of communication links and enable musical collaborations to take place among different MIDI devices. Such collaborations lend themselves to activities related with self-identification and self-configuration, or even to exploring new ways of artistic expression. In addition, wireless communications facilitate the connection of MIDI devices with other multimedia protocols, such as DMX512, making it possible to achieve a more complex level of integration between lights and music. The proposed architecture offers the possibility of developing more advanced applications for smart musical devices. A number of the many applications possible have been put forward: score management, stage rigging configuration, computer-aided learning, score transcription, and the use of a virtual musician.
Moreover, it has been observed that, despite the benefits that can be obtained from collaborative activities in the music field, none of the musical interfaces studied is valid for this purpose. As a result we have developed a preliminary prototype based on the popular Arduino platform and which makes use of ZigBee communications. Our prototype was aimed at testing some of the ideas described in this article, but it was also valid for discovering the technical issues relating to their real-life implementation. In this way, we performed multiple-user, power consumption, coverage, and latency tests.
It has been shown that it is necessary to optimize wireless communications in order to decrease latency, probably by simplifying the protocols of the lower layers of the ZigBee stack. The study of such optimizations can constitute an interesting topic for further research.
Further interesting future work consists in the implementation of a demonstration platform, which can be used as a paradigm of the concepts described in this article. Such a platform could be designed as a collaborative sensor-based musical instrument with indoor location features that could be used in contemporary art and science museums.
Finally, another interesting area for future research is that relating to the development of distributed algorithms that could be used in MCWN-based collaborations among musical devices like those described in this article.
Footnotes
Acknowledgments
This paper has been supported by Xunta de Galicia (10TIC003CT) and Ministerio de Ciencia e Innovación of Spain with FEDER funds of the European Union (IPT-020000-2010-35).