How do you choose the best data mining technique for your analytics project? (2024)

Key takeaways from this article

• Define your problem:

  Before diving into data mining, pinpoint the exact issue you're looking to solve. This helps you choose a technique that aligns with your goals, be it understanding customer behavior or improving operational efficiency.

• Test and refine:

  After selecting a method that suits your data and objectives, rigorously test its effectiveness. Refining based on performance metrics ensures the technique you've chosen is truly enhancing your analytics project.

This summary is powered by AI and these experts

• Giovanni Dicandio Senior Analytics Leader and BI…
• Sankalp Saoji Machine Learning Engineer @ Stealth…

• Giovanni Dicandio Senior Analytics Leader and BI specialitst | Advancing Corporate Strategy with Actionable Intelligence insights | Tableau | Python | QlikSense

  • Report contribution
  Often, the combination of both approaches (ML and Data mining) may be the most effective solution to extract valuable insights from the dataset, however, the choice is dependent on the project goals (predictive vs. exploratory), data type (structured vs. unstructured), data availability (labeled vs. unlabeled), interpretability, and resource constraints.

  Like

  2

• Sankalp Saoji Machine Learning Engineer @ Stealth Startup | LinkedIn Top Analytics Voice | L'Oreal | Target | Fidelity Investments | University of Rochester '23 | IIT Madras '20

  • Report contribution
  From my experiences, what you should do is this. Start by clearly defining your problem and understanding your data. Select a technique that fits your data type and objectives, such as clustering, classification, or regression. Finally, test and refine the chosen method based on its performance metrics to ensure it effectively meets your project's goals.

  Like

  5

• Afnan Khan ML Expert and Marketing analyst

  • Report contribution
  In my experience, one tends to forget the actual purpose of data mining by focusing too much on the data and their inability to link it to the business objectives. While machine learning is an extension of data mining, without an understanding of the business and the data, the extension is just a waste of time. Yes, you can run the data and make forecasts, but then you will still have to do a debrief. Although ML can save thousands of human hours when analyzing complex data sets.

  Like

  5

• • Report contribution
  While it's true that data mining and machine learning share similarities, their distinction lies in how they're applied within analytics projects. Data mining is more exploratory in nature, focusing on extracting patterns or hidden relationships within vast datasets, often without a predefined outcome. On the other hand, machine learning brings in the ability to automate and enhance this process by learning from the data and making predictions. The choice between the two often hinges on whether the goal is discovery (data mining) or prediction and optimization (machine learning). It’s important to understand that integrating both can significantly improve the depth of insights and efficiency, depending on the project’s objectives.

  Like

  5

• • Report contribution
  Yet the key question remains; Will AI replace the majority of data mining methods? The response is "NO" as AI and data mining are likely to work together. AI has the ability to automate operations and enhance the effectiveness of data mining. Nevertheless data mining methods are often customized for sectors/industries or uses whereas AI/ML models may need domain adjustment. AI/ML excels, in detecting patterns and constructing models while data mining provides a set of tools for creating features.

  Like

  4

• • Report contribution
  In my experience data as a product is something organisations should focus on. Getting a high quality data is a debatable topic because in machine learning some of the records are dropped due to presence of invalid or garbage data but sometimes the impact of those records is overseen. I believe data quality plays very important role to trust data driven decisions.

  Like

  2

• Enola Picardo Data Analyst at ICF | Python | SQL | Tableau | R | Power BI | MS - Information Systems - Data Analytics from University of Cincinnati | Ex-Accenture | Ex-Wipro

  • Report contribution
  Absolutely, data needs to be wrangled and cleaned before using it to fit any model. Also, it is very important that the data is free from any bias so that the predictions made by the model are accurate.

  Like

  2

• John Poppin Senior Business Intelligence Architect

  • Report contribution
  I have found that a strictly defined hierarchy (or maybe set of hierarchies) is key to working well with data. And while many out here might reply "Well, duh...", I've found it surprising to see such a lack of good hierarchical data structures in the professional world. If your hierarchy is solid, the size of the data almost doesn't matter. And while quality can play havoc with analyses, data quality can usually be solved with adherence to proper data governance and a good ETL layer. I would never try to reduce my data, but instead extrapolate to fill in blanks. In almost every case, more data is better data. And a great ETL implementation is pure gold.

  Like

  1

• Ashish Singh 🇮🇳 Senior Analytics Associate @ Novartis | Ex ZS Associates | Big Data | Content Creator | Writing to 50k+ Linkdlen💜 | Go Getter | Brand Collaboration 🤝 | 20 million + Content Impressions 📈| Retail Investor | HBTI

  • Report contribution
  Understanding data characteristics is crucial for selecting the right analytical approach. Structured data lends itself well to traditional statistical models, while unstructured data may require more advanced techniques like natural language processing. Large datasets might benefit from distributed computing frameworks, whereas small datasets can be handled with simpler tools. Data quality is paramount; poor quality can lead to misleading insights, necessitating rigorous preprocessing steps. The choice of technique is not one-size-fits-all but should be tailored to the specific data at hand.

  Like

  1

• • Report contribution
  The type of data you are working with plays an important role in selecting data mining or machine learning.Volume: Large datasets might benefit from more sophisticated machine learning models that can automatically process and learn from vast amounts of data.Variety: Diverse data types (text, images, numerical) may require specific preprocessing techniques and algorithms. For instance, text data might use natural language processing (NLP), while numerical data could be analyzed with regression analysis or neural networks.Veracity: The accuracy and consistency of your data can affect your choice. If data quality is an issue, you'll need to start with cleaning and normalization. After that, the appropriate technique can be applied.

  Like

• James Brown Market Intelligence at Amazon

  (edited)

  • Report contribution
  A critical part of this process is to make sure you aren't making these determinations in a monologue with yourself, but rather a dialogue with the business. Make sure you are familiar with the business case and context of the question, either through interviewing stakeholders/customers or having domain experience yourself. Before concluding that particular approaches, dimensions, or datasets are in or out of scope, you need to ensure you have a clear problem statement (or statements) in place.

  Like

  3

• John Poppin Senior Business Intelligence Architect

  • Report contribution
  First, ask yourself: Is my query addressing an 'actionable' business question? Second, remain realistic in your expectations of what your data can answer, as well as efforts required. Too many times I've seen these two facets of data work utterly ignored. There are plenty of methods for messing about with your data, and they all have their places. But if you're not chasing 'actionable' results or you're blowing time and resources on too granular a prize, you're not where you should be.

  Like

  3

• Giovanni Dicandio Senior Analytics Leader and BI specialitst | Advancing Corporate Strategy with Actionable Intelligence insights | Tableau | Python | QlikSense

  • Report contribution
  In each analytics project, maintaining a focused approach is the gold rule; it allows teams to allocate their time and resources efficiently, leveraging the most appropriate techniques and strategies. That also leads to quicker insights and more effective decision-making, while the pursuit of numerous techniques can scatter efforts and impede progress.

  Like

  2

• Ashish Singh 🇮🇳 Senior Analytics Associate @ Novartis | Ex ZS Associates | Big Data | Content Creator | Writing to 50k+ Linkdlen💜 | Go Getter | Brand Collaboration 🤝 | 20 million + Content Impressions 📈| Retail Investor | HBTI

  (edited)

  • Report contribution
  The article section rightly emphasizes the importance of aligning data mining techniques with the specific goals and scope of an analysis. From my experience, a common challenge is the temptation to use complex models for simple problems, which can lead to unnecessary complications. A solution is to start with simpler models to establish a baseline and then gradually move to more complex algorithms if needed, ensuring that the model complexity is justified by the data and the analytical goals. This approach is both efficient and cost-effective.

  Like

  1

• • Report contribution
  Your project's objectives significantly influence the choice of data mining technique:Descriptive Analysis: If your goal is to understand existing data and find patterns or relationships, techniques like clustering or association rules might be ideal.Predictive Analysis: For projects aiming to predict future trends or behaviors, supervised learning techniques such as regression analysis, decision trees, or support vector machines (SVM) are more suitable.Prescriptive Analysis: When the goal is to provide recommendations on actions to take, techniques that offer insights into causal relationships or optimization algorithms might be required.

  Like

• James Brown Market Intelligence at Amazon

  • Report contribution
  While it's great to have competency in these and other tools, I would stress that tools are simply a means to an end: delivering business value through data analysis and insights generation. I have seen hugely impactful solutions delivered using Excel spreadsheets and a Word document; conversely, I've seen analytics projects lose trust and fail to delivery useful insight even when backed by hundreds of lines of code and elaborate dashboards. If you apply the most cutting-edge tools to the wrong business questions, you are going to end up with disappointed stakeholders, no matter how impressive your technical skills.

  Like

  6

• Ashish Singh 🇮🇳 Senior Analytics Associate @ Novartis | Ex ZS Associates | Big Data | Content Creator | Writing to 50k+ Linkdlen💜 | Go Getter | Brand Collaboration 🤝 | 20 million + Content Impressions 📈| Retail Investor | HBTI

  • Report contribution
  The resource and tool consideration is pivotal in analytics, as it directly impacts the feasibility and sophistication of the project. Leveraging open-source tools like Python or R can significantly reduce costs while providing robust capabilities. However, it's essential to balance the allure of advanced techniques with the practicality of project constraints. Investing in skill development or selecting scalable tools can offer long-term benefits, enabling teams to adapt to evolving data challenges and maintain a competitive edge in the analytics landscape.

  Like

  3

• John Poppin Senior Business Intelligence Architect

  • Report contribution
  I often see organizations with far too powerful data and analytics packages for the amount of data they have or the analyses they need. Much more rarely have I seen organizations with data and analytics packages not powerful enough to yield actionable results. Most of the time, organizations lack the experience or vision to see outside their data silos to find the actionable results. Big tools and sophisticated, complex, and expensive techniques all have their places. But without understanding the results you need, no tool is of any help.

  Like

  2

• Odiljon Khudoyberdiev Top Voice | Executive of Business Relations | NEWPORT

  • Report contribution
  Consider the resources available to you, including budget, time, expertise, and computational resources. Some techniques may require specialized software, hardware, or domain expertise. Open-source libraries like scikit-learn, TensorFlow, and PyTorch offer a wide range of data mining and machine learning algorithms that are accessible and often free to use. Additionally, cloud-based platforms like AWS, Google Cloud Platform, and Azure provide scalable infrastructure for data analysis and model deployment.

  Like

  2

• Andre Nader Ex-Meta. Upleveling financial literacy across tech. Product Growth Leader.

  • Report contribution
  Always focus on the goal of any analysis. Rarely will anyone care the tool that was used to derive the insights. Ive seen even the most advanced analysis pulled from complex pipelines end up just getting summarized via SQL with the final calculations/visualizations happening in Google Sheets.Tools are just tools.

  Like

  1