Two Step Cluster

 In two-stepclustering algorithms, the first that comes to mind is the classical two-stage clustering algorithm proposed by Punj and Steward (1983). This algorithm is a hybrid approach consisting of Ward’s minimum variance method and “Kmeans” method. The advantage of such a mixed approach is that Ward’s minimum variance method calculates the number of clusters required by the “K-means” method [1].

The two-step clustering algorithm is designed to analyze large databases as primary purpose. This algorithm groups the observations in the clusters using the trait approach. Compared to classical clustering algorithms, two-step cluster analysis provides both more continuous and more featured categories. In addition, this method automatically determines the optimum number of clusters.

The two-step clustering algorithm performs the stages of pre-clustering, parsing typical data types, and clustering. During pre-clustering, each data is examined and it is decided whether each data can be included in the previous cluster or another cluster should be started. This decision is taken according to the distance between the data. There are two measures of distance, Euclidean distance and log-probability distance. In the typical data analysis phase, data that cannot be included in any cluster is evaluated. If the inclusion is still not achieved after all attempts have been made for the addition process, this data is separated as external data. In the cluster stage, a tree structure is created. All data starts to be distributed from root to leaves. Each data is attached to a branch close to it, if the number of groups to be included has reached the optimum number of group members, it is attached to another cluster in another branch that is most suitable according to the distance criterion.

To automatically determine the most suitable number of clusters, BIC (Schwarz’s Bayesian Information Criterion) or AIC (Akaike’s Information Criterion) methods are used.

REFERENCE

[1] Taşkın, Ç. ve Emel, G.G., “Veri madenciliğinde kümeleme yaklaşımları ve kohonen ağları ile perakendecilik sektöründe bir uygulama”, Süleyman Demirel Üniversitesi İktisadi ve İdari Bilimler Fakültesi Dergisi, 15(3): 395–409 (2010).

Data Mining Methods

Models used in data mining are examined under two main headings as predictive and descriptive [1].

In predictive models, it is aimed to develop a model based on the data with known results and to estimate the result values for the data sets with unknown results by using this model. In descriptive models, the patterns in existing data that can be used to guide decision making are provided.

Methods used in data mining [2]

There are also those that divide data mining methods into two main categories as supervised and unsupervised. In data mining, when there is a well-defined or precise target, the term supervised is used. If there is no specific definition for the desired result or there is uncertainty, the term unsupervised is used.

Controlled and unsupervised statements are the opposite of each other. When supervised and unsupervised methods are evaluated in terms of the whole process;

• Unsupervised methods mostly aim to give an idea about the methods used for understanding, recognizing and discovering the data and which will be applied next.
• It can be said that supervised methods are used to extract information and conclusions from the data. For this reason, it is important for the accuracy and validity of the findings to confirm an information or result obtained by an uncontrolled method, if possible, with a controlled method.

Factor Analysis, Principle Component Analysis, Hierarchical Clustering, K-Nearest Neighbor, K-Means Clustering, Two-Stage Clustering (Two Step Cluster), Kohonen Networks (Kohonen Nets) or Self-Organizing Maps, Anomaly Detection and Feature Selection algorithms can be counted [3].

Chi-Square Automatic Interaction Detector / CHAID, Detailed Chi-Square Automatic Interaction Detector (E-CHAID), Classification and Regression Tree (Classification and Regression Tree / CRT), Fast, Unattended, Efficient Statistical Tree (QUEST), Artificial Neural Networks with C5.0, Linear Regression Analysis and Logistic Regression Models and Association Rules (Association Rules), Generalized Rule Induction (GRI), Apriori and CARMA algorithms can be cited as examples [3].

New methods and algorithms are being added to the many methods used in data mining almost every day. Some of these are mainly statistical methods, which we can call classical techniques that have been used for decades. Other methods are usually based on statistics, but mostly machine learning and artificial intelligence supported next generation methods.

Data mining models are basically divided into 3 groups according to the functions they see. These:

1- Classification and Regression,
2- Clustering,
3- Association Rules.
Classification and regression models are predictive, clustering and association rules models are descriptive models.

You can find the descriptions of the models in the next Post …

REFERENCES

[1] Zhong, N. — Zhou, L., “Methodologies for knowledge discovery and data mining”, Third pacific-asia conference Pakdd-99, Beijing China, (1999).

[2] Kaya, H. ve Köymen, K., “Veri madenciliği kavramı ve uygulama alanları”, Doğu Anadolu Bölgesi Araştırmaları, 159–164 (2008).

[3] Albayrak, A.S. ve Akbulut, R., “Sermaye yapısını belirleyen faktörüler: ĠMKB sanayi ve hizmet sektörlerinde işlem gören işletmeler üzerine bir inceleme”, Dumlupınar Üniversitesi Sosyal Bilimler Dergisi, 22: 22 (2008).

Data Mining Process

It is very difficult to detect and access interesting and valuable information in large databases. Stages of Knowledge Discovery in Databases play a huge role in reaching this valuable, previously unknown, usable information by applying certain methods. The algorithm used to implement data mining enables us to realize the information discovery process in databases. In this process, it is necessary to know the properties of the data on which the model will be applied.

Data mining is also a process. In addition to uncovering the data by making abstract excavations between data stacks, separating and filtering patterns in the information discovery process and making them ready for the next step is also part of this process. If the properties of the work and data under investigation are unknown, it is not possible for any data mining algorithm to benefit, regardless of how effective it is. For this reason, before entering the data mining process, the first condition of success is a detailed analysis of job and data characteristics.

The steps followed in the data mining process are generally as follows [1]:

1. Defining the problem,
2. Preparation of data,
3. Establishment and evaluation of the model,
4. Using the model,
5. Watching the model.

Data mining in the knowledge discovery process

Defining the problem

The most important condition for success in data mining studies is to define the business purpose of the project and how the success levels of the results to be achieved will be measured. In addition, the costs to be incurred in incorrect estimates and estimates of the benefits to be gained in correct estimates should be included at this stage. At this stage, how the current business problem will be solved if the result is produced, the cost-benefit analysis of the result to be produced, in other words, the value of the produced information for the enterprise should be analyzed correctly.

This stage is the most important stage of the data mining process. The stage of defining the research problem (subject) includes the purpose of the research, the assessment of the current situation, the objectives of data mining and the determination of the project planning process.

Preparation of data

Problems that will arise during the establishment of the model will cause frequent return to this stage and rearrangement of the data. This causes an analyst to spend 50% -85% of his energy and time in the total data discovery process for the stages of data preparation and model establishment [2].

At this stage, it should not be forgotten that the numerical information produced by the company on the existing information systems should be analyzed well, and there should be a relationship between the data and the current business problem. With which business processes the digital data to be used within the scope of the project were created, it should be analyzed without using this data, so that the analyst can have an idea about the data quality.

Data quality is a key issue in data mining. To increase reliability in data mining, data pre-processing should be done. Otherwise, incorrect input data will lead the user to incorrect output. Data preprocessing is a data mining phase that is semi-automatic in most cases and takes time as mentioned above. The increase in the number of data and the necessity of preprocessing of a large number of data accordingly made effective techniques for automatic data preprocessing important [3].

Preparation of data consists of the following stages:

• Collection,
• Appraisal,
• Assembling and cleaning,
• Sample selection,
• Conversion.

Establishment and evaluation of the model

Finding the most suitable model for the defined problem is possible by installing and testing as many models as possible. For this reason, data preparation and model building stages are an iterative process until the best model is reached. The model establishment process differs according to the models in which supervised and unsupervised learning is used.

In supervised learning, which is also called learning from example, the relevant classes are separated according to a predetermined criteria by an auditor, and various examples are given for each class. The aim of the system is to find the features of each class based on the examples given and to express these features with rule sentences. When the learning process is completed, the defined rule sentences are applied to the given new examples and the class of the new examples is determined by the established model [4].

In unsupervised learning, as in cluster analysis, it is aimed to observe the relevant examples and to define the classes based on the similarities between the characteristics of these samples.

After the relevant data is prepared in accordance with the algorithm selected in supervised learning, in the first stage, part of the data is reserved for learning the model and the other part for testing the validity of the model. After learning the model using the learning set, the accuracy level of the model is determined with the test set [5].

The simplest method used to test the accuracy of a model is simple validity test. In this method, typically between 5% and 33% of the data is separated as test data, and after the learning of the model on the remaining part, the test is performed on these data. The error rate is calculated by dividing the number of events classified as false in a classification model by the number of all events, and the accuracy rate is calculated by dividing the number of correctly classified events by the number of events.

Another method that can be used in case of limited amount of data is cross validation test. In this method, the data set is randomly divided into two equal parts. In the first stage, model training on one part and testing on the other part, in the second stage, model training on the second part and the average of the error rates obtained by testing on the first part is used [5].

In small databases consisting of a few thousand rows or less, an n-fold cross validation test in which the data is divided into n groups may be preferred. In this method where data is divided into 10 groups, for example, the first group is used for testing and the other groups are used for learning. This process is continued by using one group for testing and the other groups for learning. The average of the ten error rates obtained as a result will be the estimated error rate of the established model.

Bootstrapping is another technique used to estimate the error level of the model for small data sets. As with cross validation, the model is built on the entire data set. Then, a large number of learning sets, at least 200, sometimes more than a thousand, are formed from the data set with repeated sampling and the error rate is calculated [6].

Before starting the model establishment studies, it is difficult to decide which technique is the most suitable. For this reason, it is useful to make countless trials to find the most suitable model according to their accuracy by establishing different models.

Risk matrix

The risk matrix, which is a simple but useful tool, is used to evaluate the accuracy of the models established for classification problems. In this matrix shown on the side, there are actual classification values in the columns and estimated classification values in the rows. For example, it is easily seen in the matrix that 46 elements that should actually belong to class B are classified as 2 A, 38 as B, and 6 as C by the model established.

Another important evaluation criterion is the understandability of the model. While small increases in accuracy are very important in some applications, in many enterprise practice it may be even more important to be able to interpret why the decision was made. Although they are rarely too complex to be interpreted, decision trees and rule-based systems in general can very well reveal the underlying reasons for model estimation.

Leverage ratio and graph are an important aid in evaluating the benefit provided by a model. For example, in an application aimed at determining the customers who will probably return the credit card, if 35 of the 100 people determined by the model used actually return their credit card after a while and only 5 of the 100 randomly selected customers return their credit cards in the same time period, the leverage ratio will be found as 7 [7].

Another criterion used in determining the value of the established model is the rate of return of the investment to be obtained by dividing the income to be obtained from the application proposed by the model by the cost to be incurred for the implementation of this application.

No matter how high the accuracy of the installed model is, it is not possible to guarantee that it fully models the real world. The main reasons why a valid model is not correct as a result of the tests performed are the assumptions accepted in the model establishment and the incorrect data used in the model. For example, the change in the assumed inflation rate over time during the establishment of the model will significantly affect the purchasing behavior of the individual [5].

Using the model

The established and validated model can be a direct application or can be used as a sub-part of another application. Established models should create meaningful patterns and evaluations for problem solutions intended in the area of use.

Monitoring the model

Changes that occur in the properties of all systems and therefore in the data they produce will require continuous monitoring and rearrangement of established models. Graphs showing the difference between predicted and variables are a useful method for monitoring model results [3].

Data mining process steps

REFERENCES

[1] Shearer, C., “The crisp-dm model: the new blueprint for data mining” Journal of Data Warehousing, 5(4), 13–23 (2000).

[2] Piramuthu, S., “Evaluating feature selection methods for learning in data mining applications”, Thirty-First Annual Hawai International Conference on System Sciences, Hawai, 5: 294 (1998).

[3] Kayaalp, K., “Asenkron motorlarda veri madenciliği ile hata tespiti”, Yüksek Lisans Tezi, Süleyman Demirel Üniversitesi Fen Bilimleri Enstitüsü, Isparta, 1–45 (2007).

[4] Akbulut, S., “Veri madenciliği teknikleri ile bir kozmetik markanın ayrılan müĢteri analizi ve müĢteri segmentasyonu”, Yüksek Lisans Tezi, Gazi Üniversitesi Fen Bilimleri Enstitüsü, Ankara, 1–25 (2006).

[5] Albayrak, M., “EEG sinyallerindeki epileptiform aktivitenin veri madenciliği süreci ile tespiti”, Doktora Tezi, Sakarya Üniversitesi Fen Bilimleri Enstitüsü, Sakarya, 56–70 (2008).

[6] Aldana, W.A., “Data mining industry: emerging trends and new opportunities”, Yüksek Lisans Tezi, Massachusetts Institute of Technology, Massachusetts, 11 (2000).

[7] İnan, O., “Veri madenciliği”, Yüksek Lisans Tezi, Selçuk Üniversitesi Fen Bilimleri Enstitüsü, Konya, 1–50 (2003).

 

History of Data Mining

 Nowadays, information is proliferating very rapidly and it is difficult to distinguish between what will and what will not. Data mining is the method used to get the ones that will work out of these chunks of information.

From the past to the present, data have always been interpreted, information has been sought and hardware has been created for this. In this way, information has become carried from the past to the present.

In the 1950s, the first computers were used for counts.
In the 1960s, the concept of database and data storage took its place in the technology world. In addition, scientists were able to develop computers with simple learning in the late 1960s.

In the 1970s, relational Database Management Systems applications were started to be used. Computer experts, on the other hand, have developed expert systems based on simple rules and provided simple machine learning.
In the 1980s, Database Management Systems became widespread and used in scientific fields, engineering etc. started to be applied in areas. In these years, companies have created databases of data about their customers, competitors and products. These databases contain large amounts of data and can be accessed using SQL database query language or similar languages.

In the 1990s, it has begun to be considered how to find useful information from databases whose amount of data has increased exponentially. 1989, the meeting of the Working Group of Knowledge Discovery in Databases of KDD (IJCAI) -89 and 1991, the concluding declaration of KDD (IJCAI) -89 ‘’ Knowledge Discovery in Real Databases: A Report on the IJCAI-89 Workshop ‘’ The process accelerated with the introduction of definitions and concepts. In 1992, the first software for data mining was developed.

In the 2000s, data mining has developed continuously and has been applied to almost all areas. As the benefits of the results are seen, studies and practices in this field are constantly increasing.
The years of 2010 took place in history as the time period in which the concept of Big Data became more widespread. The biggest impact on this was the spread of social media sites and mobile life.
2020 is now; In parallel with the increasing data diversity with disciplines such as Internet of Things, Cloud Computing, and Deep Learning, data processing algorithms have also diversified.

Areas Where Data Mining Is Used

It is possible to use data mining wherever there is a large volume of data. Today, data mining applications are widely used in many areas where decision-making is needed. For example, education, biology, finance, stock market, genetics, health, insurance, industry, intelligence, etc. Successful applications are seen in many branches. It is known that successful Data Mining applications have been made in all these sectors for the last 30 years.

The fields and applications that use data mining can be specified as follows:

Marketing

• Determining the purchasing patterns of customers,
• Finding links between demographic characteristics of customers,
• Increasing the response rate in mail campaigns,
• Retaining existing customers, gaining new customers,
• Market basket analysis,
• Customer relations management,
• Customer evaluation,
• Sales forecast,
• Customer distribution,
• Various marketing campaigns,
• Creating marketing strategies,
• Cross selling analysis [2].

Banking, Insurance and Stock Exchange

• Hidden correlations between different financial indicators,
• Detection of credit card frauds,
• Determining customer groups according to credit card expenditures,
• Evaluation of loan requests,
• Customer distribution,
• Detection of irregularity,
• Risk analysis,
• Risk management,
• Estimating customers who will request new policies,
• Detection of insurance frauds,
• Stock price prediction,
• Detecting fraudulent accounts and frauds [3].

Retail

• Point of sale data analysis,
• Shopping cart analysis,
• Supply and store layout optimization,
• General market analysis,
• Optimization of trading strategies.

Telecommunication

• Quality and improvement analysis,
• The density estimates of the lines,
• Profile analysis of website visitors.

Science and Engineering

• Making sense of large amount of scientific data produced during the simulation and analysis of systems in laboratory or computer environment [4].

Health, Biology and Genetics

• Gene research,
• Discovery and classification of new virus types,
• Facilitating diagnoses by determining the characteristics of diseases,
• Investigating the side effects of drugs used together,
• Estimation of test results,
• Product development,
• Determination of the treatment process,
• Medical diagnosis.

Criminology

• Identifying criminal tendencies,
• Intelligence units.

The benefits it provides to researchers, company managers and administrators in the above-mentioned areas where data mining is used can be listed as follows [3]:

• Medicine is one of the fields where data is kept the most. Especially in recent years, diseases have begun to be classified with the gene maps resulting from the incredible rapid progress of genetics. It is now possible to carry out studies on which genes are likely to develop which diseases.
• Thanks to the collaboration with genetics, a lot of criminology information can be obtained from the prevention of these events before they occur by making predictions about which individuals are prone to commit crimes, to the calculation of many possibilities based on the spelling characters of the users.
• Using data from simulation environments, predictions and solutions can be generated in engineering, production or problem solving.
• Evaluations made with data mining algorithms can provide reliable and shorter time-consuming results without the need for long-term experiments and test cases.
• In banking activities, marketing strategies can be developed with sales packages to be created in cooperation with distributor companies that sell machinery and equipment for small businesses.
• It can make existing customers better known by managers.
• Especially in the financial sector, existing customers can be divided into segments and credit risk behavior models can be created to minimize the risk to new customers.
• New risk management policies can be created for all customers with similar characteristics by examining the payment performance of existing customers and determining the common characteristics of customers with poor payment performance.

The most used areas in Turkey can be listed as banking, insurance and stock exchange. When the resources are examined, it is seen that the fields where data mining is mostly used are medicine, biology and genetics.

REFERENCES

[1] Aldana, W.A., “Data mining industry: emerging trends and new opportunities”, Yüksek Lisans Tezi, Massachusetts Institute of Technology, Massachusetts, 11 (2000).

[2] İnan, O., “Veri madenciliği”, Yüksek Lisans Tezi, Selçuk Üniversitesi Fen Bilimleri Enstitüsü, Konya, 1–50 (2003).

[3] Albayrak, M., “EEG sinyallerindeki epileptiform aktivitenin veri madenciliği süreci ile tespiti”, Doktora Tezi, Sakarya Üniversitesi Fen Bilimleri Enstitüsü, Sakarya, 56–70 (2008).

[4] Akgöbek, Ö. ve Çakır, F., “Veri madenciliğinde bir uzman sistem tasarımı”, Akademik Bilişim 09, Harran Üniversitesi, ġanlıurfa, 801–806 (2009).

Data Mining

 Many terms in the literature refer to the process of automatically or semi-automatic analysis of a large amount of data and transforming it into usable information. One of them is data mining [1]. Data mining is the job of accessing confidential information caught in data piles. The reason why the term mining is used here is that it is analogous to the process of extracting suitable data in very large data environments.

Today, daily and historical data are stored in data warehouses. The size of these stored data is increasing day by day. For this reason, there are difficulties in making decisions by using increasingly increasing databases. Additionally, data may come from many sources. Ultimately, it is necessary to analyze the data to support planning and other functions of the institution [2]. Data mining is the search and analysis of meaningful and useful connections and rules through computer programs that will help predict the future of large amounts of data [3].

The application area of data mining is quite wide. Among these areas, There are disciplines such as: Database Systems, Data Visualization, Artificial Neural Networks, Statistics, Artificial Learning, etc…

Data mining and disciplines

Many terms are used instead of data mining. Some point out that data mining means extracting uncertain information and some means drawing a conclusion from the collected information. It is difficult to realize and determine whether a particular technique is a data mining technique. For example, some discuss statistical analysis techniques by saying that they belong to data mining techniques, others disagree. For example, with data mining, companies selling medical goods can increase sales of their products by pressing certain scores in advertising, and a credit bureau can limit its losses by selecting candidates who are eligible and free of defects in payment. Data mining can also be used to find abnormal behavior. For example, a spy agency can use this technology to identify abnormal behavior among its employees [2].

By using data mining tools, it is possible to reveal the trends and behavior patterns required in decision support systems for businesses to make more effective decisions. Unlike the tools in which classical decision support systems were used in the past, there are many different features in data mining for much more comprehensive and automated analysis.

The most important feature that data mining offers to businesses is the determination of similar trends and behavior patterns between data groups. This function is used extensively in marketing activities especially for target markets [1].

Another feature of data mining is that previously unknown information can be revealed. Thanks to data mining, it is possible to reveal information that is in data warehouses but cannot be seen in the first place. For example, by analyzing the products it sells, a company can shape its future campaigns or discover the links between the products it sells. The aim here is to find data sets that were not noticed before.

Although many methods in the field of statistics are used in data mining to investigate the probabilities of data sets in databases, it differs from known statistical methods in making inferences based on the qualitative values of objects [1]. Considering the level reached by data dimensions, it can be said that using statistical methods is more difficult when compared to data mining algorithms.

Data mining can actually be considered as the result of the natural development process of information technologies, because, in this development process, the increased use of computer networks and the Internet accelerated the development of databases and it became difficult to reach the purposeful information level of the collected data. In today’s economic conditions and fast changing environments, the risk of making wrong decisions is very high in decisions made based on work experience and hunches. The only way to reduce risk is decision support solutions that anticipate knowledge-based management. Data mining tools are indispensable tools for building a true decision support system. At this point, it has become inevitable to benefit from information technologies.

Based on all this information, it is possible to make the following definition for data mining: Data mining is the job of accessing and using data that is meaningful, which will enable us to make predictions about the future, from databases where a large amount of information is stored.

REFERENCES

[1]: İnan, O., “Veri madenciliği”, Yüksek Lisans Tezi, Selçuk Üniversitesi Fen Bilimleri Enstitüsü, Konya, 1–50 (2003).

[2]. Thuarisingham, B.M., “Web Data Mining and Applications in Business Intelligence and Counter Terrorism”, CRC Press LLC, Boca Raton FL USA, 35 (2003).

[3] Akpınar, H., “Veri tabanlarında bilgi keĢfi ve veri madenciliği”. İ.Ü. İşletme Fakültesi Dergisi, 29(1): 1–22 (2000).