Web Intelligence

Web data extraction has seen significant development in the last decade since its inception in the early nineties. It has evolved from a simple manual way of extracting data from web page and documents to automated extraction to an intelligent extraction using machine learning algorithms, tools and techniques. Data extraction is one of the key components of end-to-end life cycle in web data extraction process that includes navigation, extraction, data enrichment and visualization. This paper presents the journey of web data extraction over the last many years highlighting evolution of tools, techniques, frameworks and algorithms for building intelligent web data extraction systems. The paper also throws light into challenges, opportunities for future research and emerging trends over the years in web data extraction with specific focus on machine learning techniques. Both traditional and machine learning approaches to manual and automated web data extraction are experimented and results published with few use cases demonstrating the challenges in web data extraction in the event of changes in the website layout. This paper introduces novel ideas such as self-healing capability in web data extraction and proactive error detection in the event of changes in website layout as an area of future research. This unique perspective will help readers to get deeper insights in to the present and future of web data extraction.

In order to overcome the problems of poor data classification accuracy and effectiveness of traditional data monitoring methods, this paper designs a data security monitoring method based on narrow-band Internet of things. Firstly, the model of network data acquisition and sensor node’s optimal configuration is established to collect intranet data. Based on the analysis of data characteristics, dynamic intranet data analysis indexes are designed from three aspects: establishing security incident quantity index, establishing address entropy index and data diversion. According to the above-mentioned narrow-band data aggregation rate, the security index of the Internet of things is calculated to realize the security of monitoring data. The experimental results show that: whether the network attack exists or not, the accuracy rate of the method is always higher than 90%, the classification time is less than 4 s, and the energy consumption of monitoring process is always less than 150 J, which fully proves that the method achieves the design expectation.

In order to overcome the problems of high data storage occupancy and long encryption time in traditional integrity protection methods for trusted data of IoT node, this paper proposes an integrity protection method for trusted data of IoT node based on transfer learning. Through the transfer learning algorithm, the data characteristics of the IoT node is obtained, the feature mapping function in the common characteristics of the node data is set to complete the classification of the complete data and incomplete data in the IoT nodes. The data of the IoT nodes is input into the data processing database to verify its security, eliminate the node data with low security, and integrate the security data and the complete data. On this basis, homomorphic encryption algorithm is used to encrypt the trusted data of IoT nodes, and embedded processor is added to the IoT to realize data integrity protection. The experimental results show that: after using the proposed method to protect the integrity of trusted data of IoT nodes, the data storage occupancy rate is only about 3.5%, the shortest time-consuming of trusted data encryption of IoT nodes is about 3 s, and the work efficiency is high.

In order to overcome the problems of high energy consumption and low execution efficiency of traditional Internet of things (IoT) packet loss rate monitoring model, a new packet loss rate monitoring model based on differential evolution algorithm is proposed. The similarity between each data point in the data space of the Internet of things is set as the data gravity. On the basis of the data gravity, combined with the law of gravity in the data space, the gravity of different data is calculated. At the same time, the size of the data gravity is compared, and the data are classified. Through the classification results, the packet loss rate monitoring model of the Internet of things is established. Differential evolution algorithm is used to solve the model to obtain the best monitoring scheme to ensure the security of network data transmission. The experimental results show that the proposed model can effectively reduce the data acquisition overhead and energy consumption, and improve the execution efficiency of the model. The maximum monitoring efficiency is 99.74%.

In order to overcome the problems of traditional link network sensitive data anti tampering operation, such as long time-consuming and low data security, a tamper proof model of link network sensitive data based on blockchain technology is proposed. Calculate the evenly distributed random variables of sensitive node data and the difference of running distance to obtain the probability of meeting the sensitive data with other neighbor nodes, and determine the sensitive data in the link network; obtain the frequency domain of the sensitive data of the infected link network through the square difference function, and calculate the membership mean value of the infected data samples in the sensitive data; analyze the working principle of blockchain technology, Set the master key and public key of sensitive data, generate the encryption key of sensitive data of link network, and use blockchain technology to complete the design of tamper proof model of sensitive data in link network. The experimental results show that the shortest time-consuming of the proposed method is about 1 s, and the maximum tamper proof security factor is about 9.7.

The traditional channel encryption method is interfered by noise signal, which leads to long encryption time and poor security. This paper proposes a communication channel security encryption method based on cloud computing technology. In order to obtain the minimum benefits of the same channel and orthogonal channel in the frequency conversion receiver, the dual input and dual output model is constructed; the influence of time domain and frequency domain on the transmission signal is analyzed, and the nonlinear transmission characteristics of the channel signal are obtained. Through cloud computing technology, the channel transmission characteristics are divided into dynamic parameters and static parameters to realize the key distribution of communication channel under cloud computing technology; the balance factor is used to provide the key for the sorted generators to realize the secure encryption of communication channel. The results show that the average error probability of the proposed method is about 0.275, and the signal-to-noise ratio is between 115 and 118 SNR.