$n$-gram profiles have been successfully and widely used where long sequences of potentially differing lengths are analysed for clustering or classification. Mostly, machine learning algorithms have been used for this purpose but, despite their superb predictive performance, these methods cannot discover hidden structure or provide a full probabilistic representation of the data. That is why in this paper we centre our attention on a novel class of Bayesian generative models designed for $n$-gram profiles used as binary attributes. The flexibility of our modelling allows us to consider a straightforward approach to feature selection in this generative model. Furthermore, we derive a slice sampling algorithm for a fast inferential procedure which is applied to both synthetic and real data scenarios and shows that feature selection can improve classification accuracy.