The impact of big data market segmentation using data mining and clustering techniques

2.1 Customer life time value (CLTV) model

Customer Lifetime Value (CLTV) is an important metric to measure the total worth or profit to a business obtained from a customer over the whole period of their relationship with the retailer [8]. The literature defines the customers churn as the extinction of the contract between the firm and the customer, where customer retention refers to the collection of activities organizations take to reduce the number of customer’s defections.

Churn rate and retention rate critical matrix for any company and considered primary components of the future CLTV. Where CLTV is an estimation of the average profit, a customer is expected to generate before he or she churn [48]. The concept of retention and churn is often correlated with industry life-cycle. When the industry is in the growth phase of its life-cycle, sales increase exponentially. However, customer churn is the most challenging task for the retailer industry. In this perspective, more insight is needed to know the reason for customer churn in a dynamic industry.

The three main components of CLTV are customer acquisition, customer expansion, and customer retention [46]. Nevertheless, it is crucial to consider COGS (Cost of Goods Sold) and acquisition cost to square off the real CLTV. The basic model to calculate CLTV is presented in Equation (1). CLTV = ∑ t = 1 n ( r ) t p t ( 1 + d ) t(1)

The above CLTV formula is more of a proxy for an average customer who stays for X period of time and pays Y total amount of money. The t represents a specific period of time, while (t = 1) represents the first year, and (t = 2) denotes the second year. The n represents the total time period the customer will stay with the retailer before churn occurs. The r represents the month over retention rate. Pt is the profit that the customer/customers will contribute or generate to the Retailer in the Period t, and finally, d refers to the churn rate. Additionally, the customer’s loyalty can be calculated using the Retention Rate formula, as illustrated in Equation (2). Based on the Retention Rate formula, CE denotes to the number of customers at the end of each time period, where, CN is the total number of new customers acquired in the chosen time period, and CS denotes to the number of customers at the start of the time period. Retention rate = ( ( CE - CN ) CS ) × 100(2)

Management of consumer retention requires the tools that allow decision-makers to assess the risk of each consumer to defect and understanding the factors that trigger consumers’ defection [47]. Customer retention strategy also known as a loyalty rate is the collection of activities a retailer uses to maintain on a long-term relationship basis by engaging existing customers to increase profitability by increasing the number of repeat customers. CLTV represents a greater improvement compared to the traditional RFM analysis as the frequency of customer’s purchases, and the amount of customers’ average purchase is used for segmenting customers-base. CLTV matrix classifies customer purchase behavior using different segments, namely Best, Spender, Frequent, and Uncertain classified by Marcus (1998). Table 1 illustrates the criteria of each segment that were classified by Marcus (1998).

RFM is a standard statistical marketing model for customer behavior segmentation assess consumer lifetime value. The model is very popular in the retail industry as it groups customers based on their shopping power history –how recently, how often, and how much did the customer buy. RFM model helps retailers group customers into various segments or categories to identify customers who are more likely to respond to marketing promotions and future customer personalization services [17]. The R symbolizes recency refers to the interval between the time since last purchase the customer made. The F symbolizes the frequency of consumer behavior in a time period, and the M symbolizes monetary referring to the amount of money consumption in a period [18]. Quintiles scoring is the most commonly used scoring in the RFM method in arranging customers in ascending or descending order or (Best to Worst). Customers are grouped into five equal groups where the best group receives the highest score of (5), and the worst receives the lowest score of (1) [1]. The RFM score is the weighted average of its individual components and is calculated as portrayed in equation 3 and 4 to derive a continuous RFM Score. Finally, these scores can be re-scaled to the 0 –1 range [17].

RFM score = ( recency score × recency weight ) + ( frequency score × frequency weigh + ( monetary score x monetary weight ) )(3)

Rescaled RFM score = ( RFM score - minimum RFM Score ) / ( Maximum RFM score - minimum RFM score )(4)

Market segmentation helps to differentiate and customize marketing strategies into segments. Market Segmentation is a significant key in data mining, where data mining is used to interrogate segmentation data to create data-driven behavioral information segments that are applied to detect meaningful patterns and rules underlying consumer behavior [19]. Furthermore, [26] and [27] were among the studies that performed market segmentation using data mining, RFM, CLTV, and clustering technique to form a decision-making system. [28] proposed clustering and profiling of customers using customer relationship management (CRM) and RFM for recommendations were proposed. On the other hand, data mining was conducted on historical data of customer’s sales using the RFM model with K-Means algorithm where results have outlined recommendations to perform customer relationship strategy. Also, f (2016) proposed a three-dimensional market segmentation model based on customer lifetime value, customer activity, and customer satisfaction. For more accuracy, the author grouped customers into several different groups. RFM, Kano, and BG/NBD models obtained the corresponding variables.

Furthermore, the market segmentation model helps enterprises to maximize their profits. In [29–35], customers were classified into various clusters using RFM technique and association rules were mined to identify high-profit customers. RFM statistical Techniques and Clustering methods for Customer Value Analysis were combined in [26] for a company’s online selling. As far as the methodology is concerned, there is no standard convention for measuring customer purchase behavior as each literature differs in examining customer purchase behavior. Nevertheless, the K-Means algorithm is noted as an extensively used clustering algorithm in previous research due to its simplicity and speed in working with a large amount of data [36]. Despite its strength, it has been recorded that the K-Means method uses a random distribution for the seeding positions and does not maintain the same result each time that it is run [37]. New, improved K-means algorithm called K-means++, which uses sophisticated seeding procedure for the initial choice of the center positions and often twice as fast as the standard k-means. Contrastingly, Neha, Kirti, and Kanika (2012) noted that K-means and (EM) Expectation-Maximization algorithms are the two most commonly used based algorithms for the identification of growth patterns. Even though EM is similar to the K-Means algorithm, this algorithm is based on two different steps iterated until there are no more changes in the current hypothesis [29]. Expectation (E) refers to computing the probability that each datum is a member of each class. Maximization (M) refers to altering the parameters of each class to maximize those probabilities. Eventually, they converge, although not necessarily correct. Furthermore, EM algorithm is embedded with a significant feature where it can be applied to problems with observed data that provide “partial” information only [30]. Based on several comparative studies of EM and K-Means methods [31–34], it was observed that EM outperformed K-Means and results were improved when they were hybridized. The current study integrates two dynamics models, namely CLTV and RFM models, with the addition of new RFM variants, i.e., P, Q and T to cater the weakness and inaccuracy of consumer modeling that are caused by the limitations RFM. In addition, this study applies K-means++ and Expectation Maximization (EM) clustering algorithms to offer the retail industry with effective analysis of customer buying behavior through the combination of customer profitability and product profitability in creating a strategic marketing campaign as explained previously in the introduction [38–40].

2.4 Mining big data

Data mining is the process of extracting information from large data sets and transform it into an understandable form for further use. Data mining can be used in such a case where the database is large, and the classification of such data is difficult [35]. The term Big data is often used for very large databases whose size in terabytes to many PETA bytes and it is beyond the ability of commonly used Relational Database Management (RDBM) to process the data within a tolerable elapsed time. Patel, Birla, & Nair (2012) have done a lot of experiment on the big data problem. The result was the finding Hadoop Distributed File System (HDFS) for storage and map-reduce method for parallel processing on a large volume of data. However, the research in Big Data analysis using data mining especially with clustering methods is still considered to be young, and therefore attracts many researchers to conduct further research in this potential area [37, 38] proposed a fast-parallel k-means clustering algorithm based on Map Reduce, which has been widely embraced by both academia and industry. They used to speed up, scale-up, and size up to evaluate the performances of their proposed algorithm. Their finding showed that the proposed model could process very large dataset on commodity (Low-cost) hardware effectively.

Hadoop is becoming a commodity for every data-driven organization, where data is larger and comes in many formats, mining and extracting intelligence from data has always been a challenge [39]. The new dynamic in the database has brought new challenges to the current analytical models and traditional databases and emphasize the need for a paradigm shift in data extraction and data analysis. Such challenges are the performance of the data retrieval and the varieties of data sources for which the format of the relational databases may no longer be the best option. [39] stated that Traditional database systems fall short in handling scalability to boost the performance efficiency and dealing with Big Data effectively and thus the adoption of based systems such as Hadoop is increasing. Hadoop is an open-source framework for data-intensive distributed system processing of large-scale data, based on Map Reduce programming model and a distributed file system called Hadoop Distributed File system (HDFS).

Map Reduce programming model is a methodology that deals with implementation and generating large datasets, making Hadoop the preferred as a solution to the problems in the traditional Data Mining [41].

The main components of Hadoop are Hadoop distributed file system (HDFS) a high bandwidth clustered storage allows writing an application that rapidly processes massive data in parallel, which is vital for large files. Map Reduce, is the heart of Hadoop. HDFS is high bandwidth clustered storage, while Map-Reduce processing enormous pieces of data and divide the input dataset into independent smaller pieces and be distributed amongst multiple machines referred to as nodes to parallel process them [42].

In this study we are using Apache Hadoop on single-node Hadoop cluster using Ubuntu Linux 12.04 64 Bits Server Edition was preferred as the operating system and KVM (virtual memory) was selected as virtualization environment. Hadoop (HDFS) node was accessed via Secure Shell (SSH). In this study, no parameter or optimization adjustment was made on the operating system to cause performance improvement. This type of Hadoop implementation serves the purpose and sufficient to have a running Hadoop environment in order to conduct our experiment. The market segmentation model uses retail POS data acquired from a medium-sized retailer from the State of Kuwait. The POS data contains Three years (2012 –2015) of customers initial and repeated purchases who made their purchases at different geographical branches. Each transaction represented a product purchased, with each line consists of a cashier number, store-code, item-code, brand- code, product (quantity) sold, product price, date and time of the transaction, sub-total, grand total as well as the customer’s demographic information. Since the data was in separate compilations based on the stores’ geographic locations, a common data format with consistent definitions for more descriptive keys and fields was developed to merge the information. In this phase, the string variables are converted to numeric variables, and subsequently, missing values were checked and replaced default or mean values manually. In this research, the PQ variants will be used to describe customer’s purchase power in different demographic and behavioral eras and customer’s attractiveness to a specific product and service. For the T variant customers from the best segment are used to identify customer purchase curve, and we calibrate the market segmentation model using repeated transactions for 3220 customers over two years’ period.

The age attribute was grouped into four (i.e., ages from 1–17, ages from 18–24, ages from 25–34, ages from 35–44, ages from 45–54 and ages above 55). The age group analysis was based on the premise that a typical customer’s needs would change as they age. The customer’s age was classified into six categories, where each category was identified using a unique number. Category 1 = (1–17), Category 2 = (18–24), Category 3 = (25–34), Category 4 = (35–44), Category 5 = (45–54), and senior Category 6 = (55 +). The Gender attribute was encoded as 1 for Male customers, 2 for Female customers, and 3 for Companies. Furthermore, the demographic concept hierarchy method such as city and country was replaced by higher-level concept nationality. Citizens of Middle Eastern nationalities, Asian nationalities, USA, and Canada, were assigned unique numeric (binary) value. Other nationalities were grouped based on continents, namely Europeans and Africans. One exception was made for British nationalities due to the high volume of purchases. To ensure the maximum accuracy of RFM scores, the values of five-dimension attributes from the POS Data were necessary.

The attributes are described in Table 2 as follows:

The next step involved the calculation of RFM scores as well as the newly proposed variation PQ and T. The implementation of CLVT, retention rate, and RFMPQ and T are developed using advanced PL/SQL programming language. It must be noted that the RFMPQ score refers to the weighted average of its individual components in which the scoring analysis typically involves grouping customers into equal buckets (quantiles) sizes. As far as this study is concerned, the grouping procedure was applied independently to the five RFMPQ component measures.

Customers were grouped according to the respective measure into classes of equal sizes. The derived R, F, M, P, and Q groups became the components for the RFMPQ cell assignment. RFMPQ groups are aggregated, with appropriate client-defined weights, and the scores were the weighted average of its components.

The next step involved where values and scores of RFMPQ variables were determined and used as inputs of clustering algorithms. According to this research and the client’s requests, RFMPQ variables had equal weights (1:1:1).

The second variant proposed in this research was T variable, which represented the trend of customer purchase behavior using change rate. This study proposes a combination of two analytical data mining steps. For finding T, the change in consumer purchase behavior trend, it uses an enhanced best-fit regression algorithm. Then T dataset is then put into the unsupervised clustering algorithms, to split the consumers into different groups based on patterns dissimilarities. The variable T should answer a very important question if the consumer is at high risk of shifting his or her service to another retailer. One of the most common indicators of high-risk consumers is a drop off in purchase power and a decrease of visits. One of the most common indicators of high-risk consumers is a drop off in purchase power and a decrease of visits. Major limitations of market segmentation and RFM models is ignoring the behavioral changes of consumers during the time period of analysis. Although the recency variable acted as one of the indicators of such consumer behavior, it was affected by customers’ transient behavior, and it was only based on the last purchase date. Therefore, introducing a new analysis variable was essential for the retailer to narrow down high-risk consumers.

Based on the perspective of a retailer, average values change according to customers’ purchase and their satisfaction with the retailer in terms of price and services. If these average purchase amounts of a customer decrease continuously, it could be concluded that this customer is or was on the verge of shifting his business to another retailer. A similar conclusion can also be derived from a customer who had an increased average purchase value, showing that the customer was very profitable and should be ranked accordingly. Therefore, these customers with a variety must be treated accordingly. A computable parameter was developed through the introduction of some advanced programming. Firstly, the purchase amounts of all customers in each selected time period of analysis were required.

The drawback of the classic k-means algorithm is that the user needs to define the centroid point and offers no accuracy guarantees. This has become more critical when clustering documentation because each center point is represented by a word and the distance calculation between words is not a trivial task. To overcome this problem, a k-means++ was introduced in order to find a good initial center point. K-means++ is a simple probabilistic means of initializing k-means clustering that not only has the best known theoretical guarantees on the expected outcome quality but works very well in practice. According to [43–51] k-means++ algorithm is another variation of the k-means algorithm, a new approach to select initial cluster centers by random starting centers with specific probabilities are used. The steps used in this algorithm are described below. In this regard, the essential component required is the preservation of the diversity of seeds while ensuring that the outliers remain robust. The primary concern of the k-means problem is to identify cluster centers that minimize intra-class variance by reducing the distances from each clustered data point. This can be achieved through an effective and well-designed cluster-initialization technique. k-means++ was proposed in 2007 by [42] for choosing initial values (seeds) for the classic k-means clustering algorithm to avoid poor results found by the k-means clustering algorithm. K-means++ algorithm initializes means more intelligently so that there is a distribution of cluster means that is roughly even relative to the data. K-means++ accomplishes this by selecting the first cluster center at random and then drawing the next sample from a distribution that puts a heavy probability weight where there are data and no close-by cluster center. The execution of K-Means++ and EM algorithms is carried out using WEKA tools, then the generated results are exported into excel for easy comparative analysis. We evaluate the performance of the market basket based on the mean calculated across three years forecast customers’ sales transaction. The K-means++ algorithm equation’s method is explained below. The K-means++ algorithm comes with a theoretical guarantee to find a solution that is O (log k) competitive to the optimal k-means solution. It is also fairly simple to describe and implement.

Expectation-maximization (EM) algorithm is an iterative estimation algorithm, a method similar to the K-Means algorithm introduced by Dempster, Laird, & Rubin (1977). The Expectation-Maximization algorithm is an important tool of a statistical and powerful method for obtaining the maximum likelihood estimation of the parameters of an underlying distribution when data contains null and missing values to generate an accurate hypothesis. There are three steps involved in EM technique, and the first was the EM clustering initialization. Every class j, of M classes (clusters), is formed by a vector parameter (θ), composed by two parameters the mean (μj) and the covariance matrix (Pj) which defined the Gaussian probability normal distribution as features used to describe or classify the observed and unobserved entities of the data set x. The Expectation Maximization (EM) algorithm was aimed to approximate the parameter vector (θ) of the real distribution. Cluster Convergence was the third step in M-step. After every iteration was performed, a convergence inspection was conducted to verify whether the difference of the attributes vector of an iteration to the previous iteration was smaller than an acceptable error tolerance given by parameter.

3.3 Quality assessment

One type of cluster quality assessment as suggested in [41] was performed in this study is to compare between the size (diameter) of the clusters versus the distance to the nearest cluster, the inter-cluster distance versus the size of the cluster is conducted in this study. This process can also be understood in terms of the distance between members of each cluster and the center of the cluster, and the diameter (size) of the smallest sphere containing the cluster. If the inter-cluster distance was much larger than the size of the clusters, then the clustering algorithm is considered to be trustworthy. The formula is explained as; min (Dij)/max (di) where Dij was the distance between cluster I and cluster j, where both i and j were from (1 to 6) and di is the diameter.

Figure 2 shows the quality can be assessed simply by looking at the cluster diameter. Therefore, the cluster can be formed by heuristic even when there is no similarity between clustered patterns. This is occurring because the algorithm forces K clusters to be created.

OPEN IN VIEWER

Fig. 2

Cluster quality accessed by ratio of distance to the nearest cluster and cluster diameter: source (Draghici, 2003).

Results in Fig. 5. shows the summary for all four segments, namely best, spender, frequent and uncertain where most customers were females and their average age according to the cluster. In general, K-means++ is seen as the best clustering method except in the Best segment; the accuracy is very close due to the less number of customers. Each cluster will be discussed in further details with respect to the nationality in the following sections.

OPEN IN VIEWER

Fig. 5

Summary for all segments in demographic clustering (Gender).

Figure 6 shows the four clusters for Uncertain segment as generated by K-means++ and EM Clustering. Total customers in this segment are 39624. The analysis in both clustering algorithms showed that cluster 1 was the most beneficial cluster (segment) because it was more superior to the other clusters in terms of the demographic variables.

OPEN IN VIEWER

Fig. 6

Four clusters generated by K-Means++ and EM clustering for uncertain segment.

Average age results portrayed that customers were between the age of (45–54) are the most beneficial customers, and the average gender indicated most customers were male customers. In terms of the average nationality, citizens of the Philippines were considered as the best customers in average monetary when using K-Means++ and citizens of India were the best customers based on EM Clustering.

With regards to the Market Segmentation on RFMPQ dataset, it can be summarized that Best segment was the most profitable segment with its Total Average Spending Per Segment of (1109.37) compared to other segments. Cluster 2 was the most profitable cluster in the Best segment with most female customers and the age groups of (25–34) and (35–44). Meanwhile, the result for average nationality portrayed that citizens of Kuwait and UAE were the best customers in average monetary and it was discovered that both K-means++ and EM showed similar accuracy. The Spender segment was the second most profitable segment with the Total Average Spending Per Segment of (834.01). Cluster 4 of this segment was the most profitable cluster with an average monetary of (924.19).

Females customers, those between the ages of (25–34) and (35–44) as well as the citizens of Kuwait and Qatar were the best customers, and it was also shown that K-Means++ algorithm was the most accurate. Next, the Frequent segment was the third most profitable segment with the Total Average Spending Per Segment of (349.57), and it was noted that cluster 1 was the most profitable cluster with an average monetary of (397.81). Female customers, the age group of (35–44) and citizens of the UAE and Bahrain were the best customers. K-means++ algorithm was regarded as the most accurate algorithm for this cluster in Frequent segment.

The Uncertain segment was the least profitable segment with its Total Average Spending Per Segment (168.59). Cluster 1 was the least profitable cluster with an average monetary of (239.14). Female customers, those in the age group of (45–44) and citizens of the Philippines and some Arab nationalities were the best customers, and K-means++ algorithm was the most accurate algorithm for this cluster in Uncertain segment. Based on the comprehensive information extracted from the RFMPQ dataset, it is concluded that (a) the newly proposed PQ method can fulfill both robust classification and robust segmentation for market segmentation model, even in the dataset that has noisy data. Also, it is noted that (b) The (PQ) is the most important variable, and it has also been proven as a crucial parameter for clustering.

4.3 Predictive customer lifetime value

Predictive CLTV Matrix projects the new customers’ expenses over their entire lifetime with the retailer. Most retailers measure CLTV in dollars or based on the retailer’s local currency spent by the customer over their entire relationship with the retailer, i.e., from the first to last transaction. Nevertheless, it is essential to consider COGS (Cost of Goods Sold) and acquisition cost to square off the real CLTV. Predictive Customer Lifetime Value (CLTV): Estimation of the customer’s future value also considers the customers’ expected lifetime and potential monetary value from new purchases. Furthermore, based on the clients’ request, the static discount rate is set 25% in, of which 25% is minced from the total sales. The acquisition cost, which is also set based on clients’ request, which is 400 becomes the estimated figure or percentage of how much is spent to attract new customers.

4.4 Predictive customer lifetime value

Predictive CLTV Matrix projects the new customers’ expenses over their entire lifetime with the retailer. Most retailers measure CLTV in dollars or based on the retailer’s local currency spent by the customer over their entire relationship with the retailer, i.e., from the first to last transaction. Nevertheless, it is essential to consider COGS (Cost of Goods Sold) and acquisition cost to square off the real CLTV. Predictive Customer Lifetime Value (CLTV): Estimation of the customer’s future value also considers the customers’ expected lifetime and potential monetary value from new purchases. Furthermore, based on the clients’ request, the static discount rate is set 25% in, of which 25% is minced from the total sales. The acquisition cost, which is also set based on clients’ request, which is 400 becomes the estimated figure or percentage of how much is spent to attract new customers.

4.5 Customer lifetime value ranking matrix

CLTV ranking groups customers into quadrants according to their profitability and retention propensity. CLTV ranking can assist marketers in performing more effective market segmentation [17]. This is accomplished through the incorporation of RFMPQ and RFMT values with the CLTV to rank the total sales quarterly and yearly using the ranking matrix analytical function, as shown in Fig. 7.

OPEN IN VIEWER

Fig. 7

The result from the strategies implemented using ranking matrix.

Experimental results on CLTV using Ranking Matrix, which is presented in Fig. 7 reflects the success of the implementation of the market segmentation data mining model. Important to note, the test POS dataset is from 2012 to 2015. The client noticed a drop in sales and a decline in foot traffic in the second half of 2011. However, the development of our Market Segmentation model started in 2013, and our market segmentation model was implemented in the fourth quarter of 2014. The analysis indicated that the client had lost customers and sales immediately dropped in quarter 1 of 2015 as a direct result from the shift of mass marketing strategy to target marketing strategy using market segmentation methods. This is considered as a healthy result due to the shifting from mass marketing to more customized market segmentation. Nonetheless, in quarter 2 and onwards, the client has gained a small increase in sales, and a new group of customers has been retained. Results also revealed that the retailer started to turn profitable, starting with quarter 3 and big growth in quarter 4 of 2015.

A consistent result was observed from each quarter in 2015. Thus, the client strengthened its market position and made slow the effects of a rapidly weakening overall retail market. It is generally portrayed that the average lifetime of the customer is only two years, and the churn rate is 52% (see Table 6), resulting in the development of a comprehensive marketing strategy. Results have shown that the incorporation of RFMPQ model and CLTV.

Matrix was the best method to categories and classified different consumer segments and different potential consumer segments by long term profitability. Each of the market segments will be further investigated and analyzed to provide specific characterizations, better understanding and identification of opportunities, as well as the profitability and the existence of risks as each segment no longer encompasses the entire market. Based on the segments identified, the retailer should create a tailored campaign for each segment and offer separate services that distinguish these segments to maximize responses according to each segment’s behavior. Finally, based on the experiment results, it can be positively concluded that the newly proposed RFM (PQ) and (T) were far more efficient than the traditional RFM in classifying and clustering the segmentation of customer value via RFM attributes. Kindly note that the implementation of this work was done in 2015; however, for some circumstances, our documentation and publication process was delayed for quite some time.