The Dirac delta “function” (really it’s a distribution) is super useful as a model of various phenomena in physics, math, and as I recently learned, even in data science.
One way to make a Dirac delta distribution is to take a normal distribution and shrink its width smaller and smaller, while preserving area under the curve so the distribution gets taller and taller in height. Eventually you’ll end up with an infinitesimal-width, infinitely-tall distribution centered on zero, with \[ \int_{-\infty}^\infty \delta(x) \: dx = 1 \]. This odd distribution can be used to model things like impulses, point masses, and other weird things that are useful but may not really exist.
Gaussian mixture models consist of a bunch of normal distributions superimposed and added together, meant to represent different underlying features of a dataset. However, as I just mentioned, a Dirac delta is really just an extremely thin and tall version of the Gaussian distribution. So what can we get from a Dirac delta mixture model?
The answer is pretty much anything. If you mix as many dirac deltas as you have points in your dataset, you can exactly recreate the dataset in what might be the worst possible case of overfitting. If you allow yourself to loosen the integral-must-be-one requirement on the delta function, you can represent any one-dimensional data with less than (or worst case equal to) the number of datapoints, using the integrals of your deltas as a representation for how many points in your dataset lie at that particular coordinate.
If you let yourself get even crazier, you can reconstruct basically any function as a sum of infinitely many delta functions. Essentially all you need to do for that is to reverse engineer the integral, adding a delta function with the same height as each point on the graph.
Dimensionality isn’t a huge deal for these kinds of things either. For example, you could use two variations of the Kronecker delta to form a basis in 2d on which you can recreate any surface. In 3d you have a weird form of overextended Democritean physics, with infinite sums of infinitely many point particles coming together as atoms in a new kind of universe based on infinities.
It’s hard and often pointless to think in terms of infinities, but the tools you can build with them are super flexible.