Scalability issue: Massive output size (>300GB) with >1000 genomes

[chengwbTOT]

Hi there,

I am running Spacedust on a dataset of >1,000 bacterial genomes, but the output file size is exploding (over 300GB).

Since the output format seems to be pairwise, it is very hard to parse this massive file to find gene clusters that are widely present across the whole dataset.

Do you have any recommendations for handling such large datasets? Are there any upcoming updates or existing scripts that can help condense these pairwise results into a population-level summary (e.g., identifying core/prevalent clusters)?

Thanks for your help!