Announcements

A little note about what I meant under "general correct treatment of probability distribution". It's not really about the course content and more about the classical probability theory, but I think it's something important to understand.

In the exercise, we define the PDF of the Gaussian mixture model, $p(x)$. While it most often arises from the contexts of modelling data coming from multiple underlying sources, this function is mathematically just a PDF on $\mathbb{R}^D$, and it does not automatically assume latent variables representing the choice of component. We can indeed consider a joint distribution $p(x, z) = p(x \vert z) p(z)$ where $z$ plays the role of a mixture component such that $p(x)$ would be its marginal distribution. And it's totally ok to introduce this model to find the moments of this distribution conveniently. But it was not specified in the problem statement, and you have to do this yourself. Besides, not all random $X$ with the PDF $p(x)$ are generated this way, though indeed they would be equal in distribution.

More rigorous example when $Z$ cannot be properly defined: we can take a probability triple of the sample space $\Omega$ being $\mathbb{R}^D$, event space being $\sigma$-algebra on $\mathbb{R}^D$, and probability measure induced by  $p(x)$. Then, $X(\omega) = \omega$ is a valid random variable with probability density $p(x)$.

Now, say $Z$ is a random variable representing a component from which $X$ came. And naturally, different components should produce the same values of $X$. But the space we introduced before is not rich enough: we cannot have $(Z(\omega_1) = 1, X(\omega_1) = x)$ and  $(Z(\omega_2) = 2, X(\omega_2) = x)$ because from how we built it, $\omega_1 = \omega_2$ so $Z$ should be the same as well.