Recommendations for handling large datasets

[leeanapeters]

Hi, thank you for creating this great tool!

I was wondering if you could offer some guidance on handling large datasets in the unsupervised workflow? In particular this seems to be a problem with the clustering/KNN classification steps as it seems to be prohibitively memory-expensive.

I think that downsampling is interfering with the classification accuracy so I would like to use all the data if possible.

Thanks so much for your help!

Leeana

[sophiachen1]

Hi I am also using this tool with large datasets (~150k sequences). The KNN classification returns empty knn_seq.pkl and an error like below. I am wondering if you have ever encountered this error? and I suspect it may be an out-of-memory issue of KNN?

---

ValueError Traceback (most recent call last)
/tmp/ipykernel_15992/968723552.py in
----> 1 DTCRU.KNN_Sequence_Classifier(metrics=['AUC'],plot_metrics=True,n_jobs=-1, Load_Prev_Data=True,by_class=True)

~/deeptcr/lib/python3.7/site-packages/DeepTCR/DeepTCR.py in KNN_Sequence_Classifier(self, folds, k_values, rep, plot_metrics, by_class, plot_type, metrics, n_jobs, Load_Prev_Data)
2429 if plot_metrics is True:
2430 if by_class is True:
-> 2431 sns.catplot(data=df_out, x='Metric', y='Value', hue='Classes', kind=plot_type)
2432 else:
2433 sns.catplot(data=df_out, x='Metric', y='Value', kind=plot_type)

~/deeptcr/lib/python3.7/site-packages/seaborn/_decorators.py in inner_f(*args, **kwargs)
44 )
45 kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
---> 46 return f(**kwargs)
47 return inner_f
48

~/deeptcr/lib/python3.7/site-packages/seaborn/categorical.py in catplot(x, y, hue, data, row, col, col_wrap, estimator, ci, n_boot, units, seed, order, hue_order, row_order, col_order, kind, height, aspect, orient, color, palette, legend, legend_out, sharex, sharey, margin_titles, facet_kws, **kwargs)
3801 # so we need to define palette to get default behavior for the
3802 # categorical functions
-> 3803 p.establish_colors(color, palette, 1)
3804 if kind != "point" or hue is not None:
3805 palette = p.colors

~/deeptcr/lib/python3.7/site-packages/seaborn/categorical.py in establish_colors(self, color, palette, saturation)
317 # Determine the gray color to use for the lines framing the plot
318 light_vals = [colorsys.rgb_to_hls(*c)[1] for c in rgb_colors]
--> 319 lum = min(light_vals) * .6
320 gray = mpl.colors.rgb2hex((lum, lum, lum))
321

ValueError: min() arg is an empty sequence

---