Single node repair algorithm for a multimedia cloud storage system based on network coding

Theorem 1.

For the multicast problem, if any data receiver can receive the original file M sent by the source node, the maximum stream minimum cut by the source node to any data receiver must be greater than or equal to the size of the original file M.

Theorem 2.

If the maximum flow minimum cut of the source node to any data receiver is greater than or equal to the original file M, when the encoded domain value is large enough, then there is a random linear coding mode, so that any data receiver can restore the original file.

According to the Theorems 1 and 2, the network coding technology is applied to the node repair model of the distributed storage system, and a new coding strategy is created to maintain the nature of the erasure code. And through the setting and analysis of the parameters, it is shown that when repairing the failed node, the data download amount of the surviving node can be greatly reduced compared to the erasure code.

MBR is located in 0 track 0 cylinder 1 sector of the entire hard disk. The MSR partition is the Microsoft Reserved (MSR) partition. It is a partition required by every Windows operating system on the GUID Partition Table (GPT). System components can allocate parts of MSR partitions to new partitions for their use.

Any node data repair strategy can only tolerate the failure of a limited number of nodes, and in order to ensure the persistence of the original file data, the system needs to constantly replenish the data lost due to node failure. There are currently two main repair strategies for failed nodes: Eager repair and lazy repair. Once a single node is detected to be damaged or fails, the strategy for immediate repair is an active fix. For possible node failures, the system delays the repair process as much as possible, and when the unavailable node reaches or exceeds a certain fixed value, the system starts repairing as delayed repair. For these two different repair strategies, the researchers established corresponding repair models according to different setup requirements. Next, we give a specific repair model. In general, by comparing the data stored by the original node with the data obtained by repairing the failed node, there are three types of repair modes: exact repair, functional repair, and accurate repair of only system node data. The data recovered by the new node may be different from the data stored by the failed node, as long as the repair mode that can guarantee the same reliability is called function repair. The repair of the node by the random network coding is a function repair, and the implementation generally needs to perform operations in a large finite field, and the computational complexity is high. The exact repair requirement can accurately recover the data of the failed node, so there is no need to update the data repair and reconstruction rules after the node is repaired, so the overhead is relatively small and more research significance. Accurate repair of system node data only requires accurate repair of system nodes, which ensures that there is always a system part of the storage system after the node is repaired, that is, unencoded raw data.

4. Implementation and discussion

With the rapid development of cloud computing, the concept of cloud storage came into being, which is an extension of cloud computing. Cloud storage is a virtualization technology that relies on the network. Customers can upload their own resources to the cloud and save them through the network. As long as there is a network, users can access them. Cloud storage treats servers and storage devices such as USB flash drives as a data source and supports multiple working modes. It can automatically and reasonably allocate resource groups according to requirements. With the continuous development of network technology, the use of cloud disk storage information has become the only way for many apps to store internal materials and resources. At present, there are many kinds of common cloud storage technologies, such as Amazon cloud storage. The storage service provides three different products according to the user’s data characteristics: high-read and write hot data has SSDstorage, ordinary read and write data has S3, cold backup data. There are Glacier and others; Microsoft’s cloud storage technology is recognized by the AzureBlobStorage service, which is also the first storage service in the AzurePaaS service. In addition, OpenStack storage and HDFS are more popular in the open source world. OpenStack is the most mature cloud computing project in open source cloud computing technology. OpenStack provides Swift object storage, which can realize distributed PB-level standardized server storage access, and storage access is highly parallel and scalable, and can also guarantee good. The security performance is suitable for realizing the application of frequent reading and writing while ensuring high availability and unlimited horizontal expansion capability. HDFS is a distributed file system of the Hadoop big data platform. It has the advantages of horizontally expanding storage capacity, high reliability, and suitable for distributed computing of data, and can process massive amounts of data. HDFS is very suitable for managing a large number of small files. During the access to files, its focus is on data throughput, and there is a certain data access delay. First, the cloud storage splits and refines the huge data, and then passes the refined small data module to the computer cluster system, and the system stores the small data modules in a distributed manner. The core of cloud storage is the perfect combination of storage devices and application software to achieve free access to storage devices. Cloud storage is a relatively complex system consisting of network devices, storage devices, servers, application software, public access interfaces, access networks, and client programs. Among them, the storage device plays an important role in every part of the system. The data and service access in the storage device are realized by an application software. The structure of the system consists of an access layer, an application interface layer, a basic management layer, and a storage layer, which are arranged from top to bottom. Figure 1 shows a schematic diagram of multimedia cloud storage.

OPEN IN VIEWER

Fig. 1.

Multimedia cloud storage.

The cloud multimedia classroom virtualization management platform is an application cloud storage and virtual disk storage technology, which mainly solves the problem of operating system startup and end user data storage, and has the advantages of convenient maintenance management and low cost. Figure 1 contains mobile hardware device terminals, cloud services, global mobile communication services, and protocol support for communication processes. Based on the platform, teachers can store data information and install related software under the requirements of their own courses, solving the problem of repeatedly installing software in each classroom. It is not necessary to use U disk to copy courseware every time. Through the services provided by the cloud storage system, teachers do not have to consider which campus they are in, which classrooms they attend, and they can access their own resources and teaching environment through the network. Therefore, the problem of delaying the lecture due to a software installation error or a virus infection is solved. The operating system of the terminal is started by the PXE service. The local device can have a disk or a diskless disk. The local operating system is started in a soft restore mode, which solves the problem that the system operation is affected by the data operated by the user. On this platform, it is operated once on the workstation and is available to all clients, whether it is installing the system or software. Maintenance personnel can install or update the software at any time according to the teaching needs, which greatly facilitates the maintenance of the system and application software. Management server business logic is the core of the entire system, PXE boot and data write back service are the two engines of the system. The PXE boot-up part is mainly used when the client is powered on, and after the network card is initialized, the operating system is downloaded from the server through the network and started. The PXE startup service module is responsible for the booting of the client, and is mainly composed of the client powering up, initializing the network card, and the DHCP service and the TFTP service on the server side. The server’s DHCP service is mainly responsible for the management operations such as the assignment and recovery of the workstation’s IP address, and provides relevant information to the client. The TFTP service can send a file to the workstation to boot the workstation and then download the operating system and boot through the network. The data required for the service terminal to run is stored on the server side, and is obtained through the network using the TCP/IP protocol. For wood or disk, the data is written back locally or on the server.

In software design, the idea of using hardware close to the network is generally more efficient than network storage. In the local storage, three strategies for writing back to the server, such as write-back, real-time writeback, and post-writeback, are designed. On the centralized management platform, the user can set the system startup base parameter, obtain the client online status and network real-time traffic, perform remote restart, remote shutdown, remote operation, etc. for the client, and also validate the operating system image management (Fig. 2).

OPEN IN VIEWER

Fig. 2.

Future trajectory prediction based on the classification.

Initialized IO input and output. In order to simplify the input operation and save cost and resources, this article uses a simple GBK file containing the content of “ABC”. The interface reflects the data server startup, cross-network segment and super workstations in different colors. Multi-system startup, the cloud storage platform is preferentially set to network virtual startup. Only when the network fails, in order to ensure the normal operation of the system, the system can be switched to start from the local hard disk to avoid the occurrence of teaching accidents. Network boot support for Windows XP, Win7 and Win10. The classroom BIOS setting is required to support the remote wake-up function, and the CMOS is set to the network boot priority option. Before setting up the data server and workstation software installation, you should disable the QS protocol deletion and firewall of the machine to prevent the DHCP packet from being filtered out. If the client and server are not in the same VLAN, you need to perform DHCP relay on the switch. For the data server to be used, set the default working directory in the Server Basics tab. The image file directory and the restored file are mainly read operations, generally placed under the same disk partition, the temporary file is mainly a write operation, and the directory is placed in another partition. Then add a 5-15G diskless boot disk in the Disk Management tab. After the data server is established, the workstation parameters must be set. When the workstation is not booted for the first time, it will be started according to the default parameters of the workstation. Use a server to set up a workstation in the IP segment. If there is a disk, start the workstation and copy the FLCL ENT file package to the hard disk. After the installation, install the corresponding service file, configure the configuration network card model, and shut down the computer. On the server side, set the default setting of this workstation to the specific IP address of the current data server, and then set this workstation as the super workstation. At this point, the disk starts the super workstation, executes the FLCLIENT client program, selects the system upload, and closes the corresponding SERVER end after the upload is completed to close the super station function of the workstation. Then all workstations choose network boot, and you can achieve diskless boot. If you want to install the program later, repeat this process, after installing the required software on it, upload the system image, and all workstations will be mirrored after the new installer file. When the cloud storage platform obtains the administrator synchronization command, it will automatically penetrate the difference data between the virtual system and the local system into the local hard disk system. The so-called penetration, in layman’s terms, is to update the disk system with a diskless system. The cloud virtualization management platform has a multi-server hot standby function. Here, the server can be any node in the cloud. Through the cloud storage, all the multimedia classrooms of the school can be incorporated into the cloud multimedia classroom virtualization management platform. Multi-server hot standby is used in combination with multiple I/O servers. When an I/O server is down, the client automatically switches the I/O server to achieve hot standby, thus avoiding the emergence of an I/O server. In the event of a failure, it affects the entire I/O service.

For complex multimedia data types and the large difference in question, in order to reduce the number of nodes connected to a single node during the repair, i.e., reduce disk I / O overhead herein, all storage nodes are grouped, so that the same type of a data packet stored in the same the encoding algorithm: 1) a cloud storage system provided multimedia total number of nodes is n, and m are divided into groups, there are four nodes in each group, then n = 4  m, the ith node of the jth packet labeled node j i , 1 ⩽ i ⩽ m , 0 ⩽ j ⩽ 3 ; 2) Raw data blocks stored in the ith packet B i = { b 0 i , b 1 i , b 2 i , b 3 i } , Are stored on four corresponding to the node number in the packet; in GF (2) finite field 3) in the ith packet, taking the raw data from the j + first 2 successive nodes a node block, Perform an exclusive OR operation to obtain the encoded data block of the jth node c j i = b ( j + 1 ) mod 4 i ⊕ b ( j + 2 ) mod 4 i And so on, to obtain coded data block 4, wherein the node is a next node 0 of node 3; 4) 3) obtained in c j i Stored in node node j i On this point, the memory on each node of two data blocks, an original data block, a coded data block.

OPEN IN VIEWER

Fig. 3.

Number of surviving nodes to connect when repairing a single node.

OPEN IN VIEWER

Fig. 4.

Storage overhead and repair bandwidth of different algorithms.

In a distributed storage system, the disk I/O cost during node repair is usually measured by the number of surviving nodes connected by the new node. Figure 4 is the number of surviving nodes connected in the single node repair in different algorithms, and also represents the single node of each algorithm. Disk I/O overhead during repair. As can be seen from Fig. 3, the disk I/O overhead of the algorithm in the node repair process is stable, which is less than the traditional MSR code and the traditional MBR code algorithm under the same conditions, and the total number of nodes in the system. Increase, its performance advantage is more obvious, this is because the algorithm in this paper repairs the data on the failed node, as long as the other three nodes of the group where the node is connected, regardless of the total number of system nodes. When n = 64 , k = 32 , store M = 300  MB multimedia video files, storage overhead of different algorithms and single node repair bandwidth overhead are shown in Fig. 3. As can be seen from Fig. 4, the storage overhead of different algorithms is not much different; but the repair bandwidth is quite different, which is based on MBR algorithm.

OPEN IN VIEWER

Fig. 5.

Stateful model.

In the Fig. 5, the repair bandwidth is the smallest, and the repair bandwidth based on the erasure code algorithm is the largest. In short, the storage overhead and repair bandwidth overhead of MSR, MBR and this algorithm are relatively balanced, which is better than the erasure code algorithm. The size also has an impact on storage overhead and single-node repair bandwidth. The algorithm has the same node storage overhead as the MSR algorithm, and is smaller than the MBR algorithm. This is because the original data block B = 4 in each packet of the algorithm and the number of nodes required for decoding. k = 2 , the node stores the data amount α = B / k = 2 , that is, the MSR algorithm is adopted in the group, so the storage overhead reaches the lower boundary point. Compared with the MSR algorithm, the algorithm significantly reduces the node repair bandwidth and its node repair bandwidth. Only about 13% more than the MBR algorithm.