I would like to find out how to combine gene set enrichment analysis with hierarchical clustering. The motivation for this combination is that potentially too many gene-set symbols for leukemia may pass the p-value significance threshold. We are using an R-package for fast preranked gene set enrichment analysis (GSEA) with the URL, https://github.com/ctlab/fgsea.
Subsequent to the fast gene set enrichment analysis on a ranked list of gene symbols with different expression t-statistics, we specifically need to identify gene-set symbols belonging to a common functional group using the R language hclust function intended for supervised hierarchical clustering with a Euclidean distance between features in a timely manner, for example 15 or fewer minutes.
Quoting from Alan Moses 2017 book, "Statistical Modeling and Machine Learning for Molecular Biology" , "Clustering is meant for exploratory data analysis and therefore doesn't really have a strong framework for hypothesis testing".A new R package ClusterProfiler provides enrichment analysis of gene clusters as reported in this URL, http://guangchuangyu.github.io/2015/05/use-clusterprofiler-as-an-universal-enrichment-analysis-tool/
The fact that most quality measures found in the literature have been conceived to evaluate non-overlapping clusterings, even when most real-life problems are better modeled using overlapping clustering algorithms is analyzed in detail in the following paper and University of Texas Ph.D thesis
Academic paper: CICE-BCubed: A New Evaluation Measure for Overlapping Clustering Algorithms. Available from: https://www.researchgate.net/publication/260421976_CICE-BCubed_A_New_Evaluation_Measure_for_Overlapping_Clustering_Algorithms [accessed Apr 16, 2017].
As I am a novice to this type of bioinformatics research, please correct any inaccuracies in my problem statement.