The maximal information coefficient (MIC), a measure of dependence for two-variable relationships, can be used to discover the relationships between two variables in big data. This paper proposes a new mathematical program model for calculating the value of MIC. A corresponding efficient algorithm is designed to solve the model in big data environment. In order to illustrate the validity of the proposed algorithm, the proposed algorithm is applied into the analysis of railway accidents data. Experimental results show that the proposed algorithm could find important relationships between two variables from big data. And some factors influencing accidents are identified from many factors. In addition, compared with the algorithm proposed by Reshef et al. in 2011, the proposed algorithm has lower time complexity and needs less computation time. Hence the proposed algorithm is more suitable for big data environment.
AlbanesD., FilosM., VisintainerR., RiccadonnaS., JurmanG. and FurlanelloC., Minerva and minepy: A C engine for the MINE suite and its R, Python and MATLAB wrappers, Bioinformatics29(3) (2013), 407-408.
2.
AlelyaniS., TangJ. and LiuH., Feature selection for clustering: A review, in: Data Clustering: Algorithms and Applications, AggarwalC. and ReddyC., eds, CRC Press, London, 2013, pp. 1-33.
3.
BallG.H. and HallD.J., ISODATA: A novel method of data analysis and pattern classification, in: Technical Report of Stanford Research Institute, Stanford Research Institute, Menlo Park, 1965, pp. 6-45.
4.
BlumA.L. and LangleyP., Selection of relevant features and examples in machine learning, Artificial Intelligence97 (1997), 245-271.
5.
ChandrashekarG. and SahinF., A survey on feature selection methods, Computers & Electrical Engineering40(1) (2014), 16-28.
6.
ChongM., AbrahamA. and PaprzyckiM., Traffic accident analysis using machine learning paradigms, Informatica29(1) (2005), 89-98.
7.
CoverT.M. and ThomasJ.A., Entropy, relative entropy, and mutual information, in: Elements of Information Theory, John Wiley & Sons, New Jersey, (2012), 12-49.
8.
DepaireB., WetsG. and VanhoofK., Traffic accident segmentation by means of latent class clustering, Accident Analysis & Prevention40(4) (2008), 1257-1266.
9.
EilerA., Zaremba-NiedzwiedzkaK., Martínez-GarcíaM., McMahonK.D., StepanauskasR., AnderssonS.G. and BertilssonS., Productivity and salinity structuring of the microplankton revealed by comparative freshwater metagenomics, Environmental Microbiology16(9) (2014), 268-298.
10.
FayyadU.M., Piatetsky-ShapiroG., SmythP. and UthurusamyR., From data mining to knowledge discovery: An overview, in: Advances in Knowledge Discovery and Data Mining, FayyadU.M., Piatetsky-ShapiroG., SmythP. and UthurusamyR., eds, AAAI/MIT Press, Cambridge, 1996, pp. 1-36.
11.
FanJ. and LiuH., Statistical analysis of big data on pharmacogenomics, Advanced Drug Delivery Reviews65(7) (2013), 987-1000.
12.
FigueiredoM.A.T. and JainA.K., Unsupervised learning of finite mixture models, IEEE Transactions on Pattern Analysis and Machine Intelligence24(3) (2002), 381-396.
13.
GierlH. and SchwanenbergS., A comparison of traditional segmentation methods with segmentation based upon artificial neural networks by means of conjoint data from a Monte Carlo simulation, in: Classification, Data Analysis, and Data Highways, BalderjahnI., MatharR. and SchaderM., eds, Springer, Berlin, 1998, pp. 386-392.
14.
GorfineM., HellerR. and HellerY., Comment on ``Detecting novel associations in large data sets'', Unpublished (available at http://emotion.technion.ac.il/∼gorfinm/files/science6.pdf on 11 Nov. 2012), (2012).
15.
HalevyA., NorvigP. and PereiraF., The unreasonable effectiveness of data, Intelligent Systems, IEEE24(2) (2009), 8-12.
16.
JainA.K., Data clustering: 50 years beyond k-means, Pattern Recognition Letters31 (2010), 651-666.
17.
JavedK., BabriH.A. and SaeedM., Feature selection based on class-dependent densities for high-dimensional binary data, Knowledge and Data Engineering, IEEE Transactions on24(3) (2012), 465-477.
18.
KohaviR. and JohnG.H., Wrappers for feature subset selection, Artificial Intelligence97 (1997), 273-324.
19.
LohrS., The age of big data, New York Times, 11 February 2012.
20.
LynchC., Big data: How do your data grow? Nature455(7209) (2008), 28-29.
21.
MacQueenJ., Some methods for classification and analysis of multivariate observations, in: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability1(14) (1967), 281-297. %
22.
MirabadiA. and SharifianS., Application of association rules in Iranian Railways (RAI) accident data analysis, Safety Science48(10) (2010), 1427-1435.
23.
Mayer-SchönbergerV. and CukieK., Big data: A revolution that will transform how we live, work, and think, Houghton Mifflin Harcourt, Boston, 2013.
24.
PellegD. and MooreA., X-means: Extending K-means with efficient estimation of the number of clusters, in: Proceedings of the 17th International Conference on Machine Learning, Morgan Kaufmann (2000), 727-734.
25.
RasmussenC., The infinite gaussian mixture model, Advanced Neural Information Processing Systems12 (2000), 554-560.
26.
RamzeR.M., LelieveldtB.P.F. and ReiberJ.H.C., A new cluster validity indexes for the fuzzy c-mean, Pattern Recognition Letters19 (1998), 237-246.
27.
ReshefD.N., ReshefY.A., FinucaneH.K., GrossmanS.R., McVeanG., TurnbaughP.J. and SabetiP.C., Detecting novel associations in large data sets, Science334(6062) (2011), 1518-1524.
28.
ReshefD.N., ReshefY.A., MitzenmacherM.M. and SabetiP.C., Equitability analysis of the maximal information coefficient, with comparisons, arXiv preprint arXiv:1301.6314, 2013.
29.
ReshefD.N., ReshefY.A., MitzenmacherM.M. and SabetiP.C., Cleaning up the record on the maximal information coefficient and equitability, PANS111(33) (2014), EE336-E3362.
30.
SchadtE.E., LindermanM.D., SorensonJ., LeeL. and NolanG.P., Computational solutions to large-scale data management and analysis, Nature Reviews Genetics11(9) (2010), 647-657.
31.
SimonN. and TibshiraniR., Comment on ``Detecting Novel Associations In Large Data Sets'' by Reshef et al., Science Dec 16, 2011, arXiv preprint arXiv:1401.7645, 2014.
32.
SteinleyD., K-means clustering: A half-century synthesis, British Journal of Mathematical and Statistical Psychology59(1) (2006), 1-34.
33.
SzékelyG.J. and RizzoM.L., Brownian distance covariance, The Annals of Applied Statistics3(4) (2009), 1236-1265.
34.
TibshiraniR., WaltherG. and HastieT., Estimating the number of clusters in a data set via the gap statistic, Journal of the Royal Statistical Society: Series B (Statistical Methodology)63(2) (2001), 411-423.
35.
XieY., LordD. and ZhangY., Predicting motor vehicle collisions using Bayesian neural network models: An empirical analysis, Accident Analysis & Prevention39(5) (2007), 922-933.
36.
XuX., LiK., YangL. and YeJ., Balanced train timetabling on a single-line railway with optimized velocity, Applied Mathematical Modelling38(3) (2014), 894-909.
