Abstract
Under the influence of novel corona virus pneumonia epidemic prevention and control, it has put forward higher requirements for data storage and processing for personnel management system. The distributed asynchronous data aided computer information interaction system can solve the problem of multi node concurrent data processing. The traditional computer information interaction system has poor real-time performance, low precision and asynchronous data processing ability. The invocation features of message queuing asynchronous caching mode are combined with the standardization of Web services and cross language with cross platform access features in this paper. Through the combination of the two technologies, a flexible and universal asynchronous interaction architecture of distributed system is established. Based on Web service technology and system to system access, the call and response of tasks between modules are carried out in the system, which makes the interaction between the whole system have the characteristics of message driven. The test result shows that the system proposed in this paper has good real-time performance and strong data processing ability. It is suitable for the data interaction of distributed personal management system under the influence of novel corona virus pneumonia epidemic prevention and control.
Keywords
Introduction
Since December 2019, surveillance of influenza and related diseases has been continuously carried out in Wuhan City, Hubei Province. A number of cases of viral pneumonia have been found, all of which are diagnosed as viral pneumonia/pulmonary infection. On January 12, 2020, the World Health Organization officially named it 2019-nCoV [1]. The novel corona virus pneumonia can be identified as direct transmission, aerosol transmission and contact transmission. Therefore, the government has introduced personnel control measures in response to the spread of new corona virus pneumonia [2, 3]. The floating population supervision departments at all levels manage and control the floating population by classification and industry. The management and control departments of various industries have strengthened the supervision on the employment departments of floating population. There are many personnel management and control points. When the computer-aided management system is used, there are many network nodes and large amount of data concurrency. Therefore, higher requirements are put forward for data storage and processing of personnel management system. The distributed asynchronous data aided computer information interaction system can solve the problem of multi node concurrent data processing.
In this paper, The invocation features of message queuing asynchronous caching mode are combined with the standardization of Web services and cross language with cross platform access features in this paper. Through the combination of the two technologies, a flexible and universal asynchronous interaction architecture of distributed system is established. Based on Web service technology and system to system access, the call and response of tasks between modules are carried out in the system, which makes the interaction between the whole system have the characteristics of message driven.
Architecture and its implementation of Message Queuing and Web Services
Features of message queuing
Message Queuing (MQ) is a loosely coupled and reliable network communication service based on transaction model [4]. Among them, queue is a place where routing messages are temporarily stored in the network, and it is the container where messages are stored during the transmission of messages [5–7]. The main purpose of a queue is to provide routing and ensure the delivery of messages. If the receiver is not available when a message is sent, the message queue retains the message until it can be delivered successfully. Message queuing makes it easier to integrate applications and develop reliable applications on unreliable but cost-effective networks. Message processing provides a powerful and flexible mechanism for inter-process communication of server based application components. Compared with the direct call between components, it has the advantages of stability, message priority, offline ability, transactional message processing, etc. [8–10].
Features of web services
Web services provide a distributed computing technology, which is used to present application services through standard XML protocol and information format on the Internet or intranet. Using standard XML protocol makes web service platform, language and publisher independent of each other.
Web services use standard protocols (such as HTTP, XML, XSD, soap and WSDL) to provide message exchange in scalable, loosely coupled and stateless environments. Web services make it possible to interact and integrate different software modules in a heterogeneous environment. Because Web services use standard protocols and are platform independent, they can communicate with a wide range of soap implementations, platforms, and devices.
Web services usually use HTTP protocol as its communication carrier [11], WSDL (Web service description language) as its interface and service, UDDI (general description, discovery and integration) specification as its basic protocol for publishing and discovering information about web services, and soap (Simple Object Access Protocol) as its basic protocol for remote object access [12]. All specifications are based on XML (Extended Markup Language), and all elements in the interaction process are described in the form of XML, so as to ensure the independence with the specific system platform and language to the maximum extent, and ensure the most extensive accessibility of application services through web services.
Architecture of message queuing and Web services
The combination of message queuing technology and web service technology can effectively solve the problem of asynchronous interaction between different application systems, and realize the synchronization of Web Services interaction through the stable and flexible asynchronous interaction mechanism provided by message queuing, At the same time, we use web services technology to expand the application scope of asynchronous message effectively, which provides an effective way for asynchronous message based interaction between any platforms, and also makes it possible for collaborative work between different message queuing products.
The following is the interaction system between different enterprise information systems in our spatial information mobile user application system, and the architecture and implementation methods of this system are described and explained in detail. The specific cases are as follows:
The central system platform provides users with a wide range of available information. It needs to automatically interact with geographic information providers, transportation departments, meteorological departments and other enterprise information service systems to collect information. At the same time, it also provides system level access interface for enterprise level users. These system interfaces often exist as key task servers in the whole integration system. For example, effective integration and full use of these servers are also issues to be considered in architecture design.
Different enterprises use different systems and adopt different technologies. If we provide an interface for each system platform, it will greatly increase the complexity of system interaction. At the end of the central platform, we need to integrate various interface technologies, which is unrealistic and difficult to control. Therefore, we define the interaction interface between systems as a unified Web Services interaction mode. Each system only needs to provide its own system services in the form of Web services, and the specific interaction data is expressed in XML. In this way, at each end of the system interaction, only one interface schema of Web services and one data format of XML need to be processed. However, due to the limitations of the protocol, Web services bound to the HTTP protocol respond in a synchronous RPC manner. If the interaction between systems is also in the synchronous mode, each end will be blocked by synchronous requests for the services of the other system, which is not conducive to the batch of system service tasks and the completion of large throughput. Therefore, we use message queuing technology and multithreading technology in the back end of its interface. In this way, although the inter system interface is a synchronous call mode, the execution of the whole task is completed asynchronously. At the same time, each web service call is initiated by an independent thread, and the execution of one task does not affect the execution of other tasks. The architecture based on this model can greatly improve the concurrency and throughput of the whole integrated system. A simplified architecture of the system interaction part is shown in Fig. 1.
The framework of system interaction.
In Fig. 1, the task core is the integrated information processing module, which needs to request the external system to update information from time to time. When the information from different data sources is obtained, the integrated processing becomes the system core information base for other service program analysis. In order to meet the needs of enterprise level system interaction, a series of modules are added between service request, response and return, such as message queue, task listener, SOAP interaction interface, information collector, etc. It will be described its function and implementation one by one. For example, Windows 2000 platform and.Net technology is used. The Message Queuing service uses MSMQ, but this does not mean that this architecture is only applicable to Windows platform. There are corresponding services on UNIX platform, such as MQ Series of IBM and JMS of Java, which are similar to message queuing services and the web services technology is supported by all platforms.
The following is the definition of XML documents involved in information exchange and the representative code to implement this asynchronous interaction architecture. XML representation of the message entity used to request the figure information update:
< ? XML version = “1. 0”encoding = “U TF - 8”?>
<Message Type = “request”>
<Transaction>
<Key> 8 E34 # 9 I - OA2 @V IOE < /Key>
<Start Time > 2002 - 8 - 12 00:00:00 < /Start2 Time>
<Over Time > 2002 - 8 - 12 07:00:00 < /Over2
Time>
< /Transaction>
<ServerName> GISinfoCenter< /ServerName>
<Service>
<Name> GetArea Points < /Name>
<Area ID > 51D< / Area ID>
<Info ID > 00010113 < /Info ID>
< /Service>
Web service definition for callback:
[WebMethod]
public Result GetArea PointsCallback (string Server2 Name; string Transaction Key, List packagelist)
[WebMethod]
public Result GetArea Traffic InfoCallback (st ring ServerName; string Transaction Key, List package2 list)
It is worth mentioning that we add the concept of transaction to the whole process of distributed interaction. By dividing the cross server transactions into different processes according to the service control domain, the rollback operation of the whole transaction can be completed independently without affecting each other, thus greatly reducing the coupling between systems and ensuring the feasibility of cross system transactions. Transaction definitions are already included in the new soap specification, but we do not use them here. The reasons are as follows:
1) The transaction processing of web service is not supported by the system platform, so the transaction code must be processed by the transaction itself.
If a web service transaction is implemented by self-processing, the soap head must be processed, and such processing can only be completed after strict and consistent definitions are made at both ends of the platform participating in the transaction. Such an interface call requires additional coding in the system target system, rather than a simple web service reference. Such an implementation will make the interaction between systems unenforceable.
The method of passing transaction key with function parameters can make transaction processing very flexible. The target system can complete the interaction between systems even without transaction processing. The key here is that the whole transaction is handled separately in different systems.
So far, we have understood the main process and implementation means of cross system interaction under such a system architecture, as well as the flexible and diverse interaction modes supported by this architecture. The asynchronous multitask processing of message queue and the non blocking call between modules in the system are realized by multithreading technology. Through the web service call, the service request and callback between systems and the transactions processed by stages and across systems are realized. It can be said that the system based on this architecture has been able to meet most of the needs of distributed asynchronous system to system interaction over the WAN.
The LDB storage subsystem is implemented based on the event driven model and programmed with the object-oriented C++ language. Using the Epoll API of reusing I/O provided by Linux, on LDB, including network read and write, task scheduling, timer response are all based on Epoll. Epoll can not only listen to network read-write events, but also listen to pipeline read-write events. The operation interface of Epoll is simple and clear, including three operations: registering new I/O events, deleting I/O events, and modifying the collection of monitored events on I/O events. In the implementation of LDB, these operations are encapsulated in the Epoll event class, which encapsulates the data operations required by the Epoll event driven model, as shown in Fig. 2.
Epoll event class definition.
The most critical function in the Epoll event class is run(). When the system initialization is completed, the main thread will execute the run() function. In the run() function, the main thread will enter the cycle of the event driven model. The specific logic is shown in Fig. 3.
Logical implementation of run() function.
After executing the run() function, execute the function Epoll first_ Wait (), the main thread enters a blocking state. When an I/O event occurs, the main thread is no longer blocked, and the following code is executed. Check whether there is a readable event. If there is one, handle it. If the readable event comes from a network socket, the recvdata() function of the agent corresponding to the socket is called. If the read event comes from the pipeline read end of the completed task queue, the function recvdata() function of thread pool dispatcher, the thread pool scheduling module, is called. Then check whether there is a writable event, and handle it if there is one. If the writable event is from a network socket, the senddata() function of the agent corresponding to the socket is called. If the writable event comes from the pipeline writer of the task queue to be completed, the function senddata() of thread pool dispatcher, the thread pool scheduling module, is called. The timer’s timeout event is then processed. Finally, it will check whether the shutdown signal from the system is received. If the shutdown signal is received, exit the loop. Otherwise, execute epoll_Wait () enters the block again. During the initialization of LDB system, SIGINT will be shielded to prevent the interruption of the key operation. After receiving the signal, a flag will be set. In this step, the administrator will check whether he wants the program to exit. If so, he can exit safely.
For distributed database, cross node table connection is not only the key but also the difficulty. In the design of LDB subsystem, LDB can execute the sub plan sent by DTE. Therefore, if a cross node table connection is involved, the connection strategy is determined by DTE. At present, the research focuses are the strategies based on direct connection algorithm and the strategies based on semi connection algorithm. In CRDB system, the design of LDB subsystem is based on the execution strategy of DTE. For customer query, either direct connection strategy or semi connection strategy can be used. In this section, we will design tests, test direct connection and half connection respectively, and analyze the test results.
Test method
In the experiment, the design test encountered two one to many relationship types, student and comment. The operation is to find the left connection between the comment and the student. The direct connection algorithm and the semi connection algorithm are applied to the connection operation of these two relationships to test the time cost of communication and the amount of data transmitted by the network. There will be six groups of use cases for these two relationships in this test. In each group of use cases, the data amount of the two relationships is the same. The only change is that the data in the relationship comment is projected to stu_no. The result set after no Πstu _ no(comment). This will lead to the result student ¡Þ Πstu _ no(comment) after the subset is semi connected with the student_ The amount of no (comment) data is different. The test results will show the cost results of direct connection algorithm and semi connection algorithm applied to six cases respectively. It is predicted that the result of direct connection is the same in six groups of use cases, while the result of semi connection will be the same because of student ¡Þ Πstu _ no(comment) is different.
Test cases
The attributes of student include (stu_no, name, gender, birth, addr, tel). The attributes of comment include (stu_no, teacher_no, date, comment), where stu_no is the primary key related to student and the foreign key to comment. The query sent by Gateway to DTE is:
"select stu_no, name, gender, birth, addr, tel, teacher_no, date, comment from comment left join student on student.stu_no = comment.student_no".
Among the six sets of use cases, the same parts include: the relationship student has a tuple of 100,000 and the amount of data is 22.9MB, the comment has a tuple of 100,000 and the amount of data is 50.73MB, and the stu_no in the studnet ranges from 1 to 100,000, which is not repeated. The different parts are shown in Table 1.
The different parts
The different parts
Insert, update and delete operations are common basic operations of database. For the test of these operations, a fixed number of execution times is used to measure the execution time. For example, for insert testing, measure how much time it takes to perform 300000, 600000, and 900000 times, respectively. The tests for update and delete operations are similar. As a comparison, we did the same test for MySQL. Figure 4 is the test result of CRDB system.
Logical implementation of run() function.
The same operation is performed on MySQL and the result is shown in Fig. 5.
Logical implementation of MySQL.
First of all, it can be seen that in the two sets of data, when the number of execution is increasing, the time consumption basically increases linearly. This shows that the design and implementation effect of the system is better, so it will not lead to a sharp decline in performance in the face of increasing pressure.
In addition, compared with the two, CRDB system, as a distributed database, has two more network transmissions than MySQL for each operation compared with insert, update and delete: gateway sends query request to DTE and DTE returns query execution result to gateway. This is the main reason why CRDB system is slower than MySQL.
Generally speaking, because each time a command is executed, the former will have two more network transmission times than the latter. The test result of CRDB cannot be better than MySQL, but the closer it is, the more it shows that the design of CRDB system can reduce the extra cost of database changing from single node to distributed as much as possible. From the test result, CRDB has the same execution times as MySQL It takes about twice as long and basically meets the performance requirements of distributed database.
The time cost of the direct connection algorithm in the use case includes: the time cost of gataway sending the query request to DTE (t1); the time cost of DTE sending the query request to LDB_ 1 time cost of sending query plan (t2); LDB_ 1 send local relationship comment to LDB_ Time cost of 2 (t3); LDB_ 2. Calculate the time cost (t4) of comment ∞ stud_net sent to the gateway locally. So there are:
The time of t1 and t2 in local area network is about 1 ms, which is ignored in the whole time. Therefore, the time cost model can be written as follows:
In addition, t4 includes the time of connection operation and network transmission. It is usually to connect two relationships one by one locally, instead of inefficiently connecting and putting them in memory before sending. Therefore, this part of time must not be calculated separately. Considering that the access speed of the local cache is much faster than that of the network, the time cost of the t4 part actually depends on the time cost of the network transmission. In the test, the direct connection scheme gets the same time cost in six groups of use cases, and the results are shown in Table 2. The result shows that it takes 2.1 seconds for the relationship comment sent to LDB_ 2. In LDB_ 2, it takes 6.7 seconds to connect the table and send it to the gateway, and the total time is 8.8 seconds.
Time cost of direct connection algorithm in six groups of use cases
The time cost of applying semi join algorithm includes: gateway sends query request to DTE (t1), DTE sends query request to LDB_ 1(t2); LDB_ 1 Πstu_no (comment) sent to LDB_ Network transmission (t3); LDB_ 2 sent to LDB_ Network transmission overhead(t4); LDB_ 1 network transmission cost sent to gateway (t5). The proportion of t1 and t2 is ignored, which is calculated as:
The data transmitted includes: the data sent by the client to query (d1); DDBMS sent to LDB_ 1 (d2); LDB_ 1 Πstu_no (comment) sent to LDB_ Data (d3); LDB_ 2 sent to LDB_ Data (d4); LDB_ 1 ¡Þ (student ¡Þ Πstu_no (comment) sent to the client (d5). d1 and d2 are negligible. So there are:
The test results of applying the semi join algorithm are shown in Tables 3 and 4.
Time cost of semi join algorithm in six groups of the cases
Semi join algorithm transfers data in six groups of the cases
From the data in Tables 3 and 4, it can be seen that in terms of time consumption, in the first group of use cases, the time consumption of applicable semi join algorithm is slightly higher than that of applicable direct join algorithm. In the second group of use cases, the time consumption of applicable semi join algorithm is reduced by 1.8 seconds. In the latter four groups, the time-consuming of applying semi join algorithm is close to that of the second group and there is no improvement.
It can be seen from this that the key to reduce network overhead is to connect tables across nodes in a distributed database. In the protocol design of LDB, bandwidth utilization has been taken into account. If you want to improve, you should consider data compression. In this way, by increasing CPU overhead and reducing network overhead, the total delay may be reduced. In addition, from the delay results of LDB applying two connection algorithms to different six groups of use cases, semi connection algorithm is superior to direct connection algorithm in many times (case 1–5). This has a high reference value for DTE designers.
Message queuing and web service technology are two very practical and common technologies in the field of distributed computing. They have been widely supported by almost all system platforms, and products providing similar services have been widely used in various application systems. Message queuing products such as MQ Series of IBM, MSMQ of Microsoft, JMS service under J2EE platform, etc. The services provided by these products are completely similar, which provide support for asynchronous message interaction between different application modules. However, due to the different internal protocols adopted by different products, it is difficult to cross use these products with each other. This is also the main obstacle of interaction between different platforms and systems.
Web service technology effectively solves the problem of seamless connection and interaction between different systems, and well solves the problem of cross firewall. Its emergence provides a unified framework for mutual access between systems, and has become the industry standard.
By combining the two technologies, we build a flexible and universal asynchronous interaction architecture of distributed system. Web service technology can be used for inter system access, and the inter module task call and response can be carried out within the system combined with message queuing technology. This makes the interaction between the whole systems have the characteristics of message driven. At the same time, it is not required that the two ends of the system use the same system platform and the same message queuing product, which is quite different from the previous system implementation based on the same message queuing service. This also makes our system architecture have a wide range of adaptability, it can be built on most of the system platform and language environment. At the same time, based on this architecture, a variety of flexible system interaction modes can be realized, such as inter system function callback, cross system scope transaction processing, batch processing mode of critical task system, etc.