Complete your answers on a separate sheet (handwritten or LaTeX) and submit on Gradescope.
Problem 1 (20 points)
Consider a causal AR(2) process of the form:
X_t = \phi_2 X_{t-2} + W_t
with 0 < \phi_2 < 1 and W_t \sim \mathcal{N}(0, \sigma_w^2).
(i) (6 points)
Assume that we have observations \{x_1,x_2\}. Derive the mean and variance of a future observation x_t with t>2. (Hint: you’ll need your solutions from Problem 4 of Homework 1.)
(ii) (2 points)
What is the mean for t=3 and t=4. Explain the intuition behind this result.
(iii) (2 points)
What is the covariance for t=3 and t=4. Explain the intuition behind this result.
(iv) (2 points)
What is the mean and variance of x_t as t \to \infty? Explain the intuition behind this result.
(v) (6 points)
Assume that we have observations x_1. Derive the mean vector and covariance matrix of a future set of observations \{x_t, x_{t+1}\} with t>1.
(vi) (2 points)
What is the mean vector and covariance matrix of \{x_t,x_{t+1}\} for t=2? Explain the intuition behind this result.
Problem 2 (10 points)
Consider the latent space model we presented in class defined by:
\begin{align} \mathbf{z}_t &= \mathbf{A} \mathbf{z}_{t-1} + \mathbf{w}_t \\ \mathbf{x}_t &= \mathbf{C} \mathbf{z}_{t} + \mathbf{v}_{t} \end{align}
where the latent space \mathbf{z} has dimension d and the data \mathbf{x} has dimension n. Our noise is being drawn from \mathbf{w}_t \sim \mathcal{N}(0,\mathbf{Q}) and \mathbf{v}_t \sim \mathcal{N}(0,\mathbf{R}).
(i) (4 points)
Assume that n=d and that we have C = \alpha \mathbb{I} and R = \sigma_v^2 \mathbb{I}. Write the mean \mu_{t|t} = \mu_{\mathbf{z}_t|\mathbf{z}_{t-1},\mathbf{x}_t} and covariance \Sigma_{t|t} in terms of the mean \mu_{t|t-1} = \mu_{\mathbf{z}_t|\mathbf{z}_{t-1}} and covariance \Sigma_{t|t-1}. Simplify as much as possible.
(ii) (2 points)
What happens in the limit \sigma_v \to 0? Demonstrate the answer quantitatively and explain the intuition behind this limit.
(iii) (2 points)
What happens in the limit \sigma_v \to \infty? Demonstrate the answer quantitatively and explain the intuition behind this limit.
(iv) (2 points)
What happens in the limit \alpha \to 0? Demonstrate the answer quantitatively and explain the intuition behind this limit.
Problem 3 (5 points)
Consider a modified version of our latent space model that depends on a set of observed values \mathbf{y}_t as follows:
\mathbf{z}_t = \mathbf{A} \mathbf{z}_{t-1} + \mathbf{B} \mathbf{y}_t + \mathbf{w}_t
otherwise all the other components are identical to those described in Problem 2.
(i) (5 points)
Derive how this new process changes the filtering step of our Kalman filtering.
Problem 4 (19 points)
Consider the latent space model we presented in class defined by:
\begin{align} \mathbf{z}_t &= \mathbf{A} \mathbf{z}_{t-1} + \mathbf{w}_t \\ \mathbf{x}_t &= \mathbf{C} \mathbf{z}_{t} + \mathbf{v}_{t} \end{align}
where the latent space \mathbf{z} has dimension d and the data \mathbf{x} has dimension n. Our noise is being drawn from \mathbf{w}_t \sim \mathcal{N}(0,\mathbf{Q}) and \mathbf{v}_t \sim \mathcal{N}(0,\mathbf{R}). Assume that n=d and that we have \mathbf{A} = \mathbb{I}, \mathbf{C} = \mathbb{I}, and \mathbf{Q} = 0 \cdot \mathbb{I}. Finally, assume that \boldsymbol{\Sigma}_0 \to \infty. We make no specific assumption for \mathbf{R} or \boldsymbol{\mu}_0. We want to show that the smoothed mean \boldsymbol{\mu}_{t|T} simplifies to the mean of the data \frac{1}{T} \sum_{i=1}^T \mathbf{x}_i. This problem will walk you through the steps:
(i) (5 points)
Show that \boldsymbol{\mu}_{1|1} = \mathbf{x}_1 and \boldsymbol{\Sigma}_{1|1} = \mathbf{R}. You will need to take the limit \boldsymbol{\Sigma}_0 \to \infty. Hint: it will be useful to know that (\mathbb{I} + \mathbf{M})^{-1} = \sum_{i=0}^\infty (-1)^i \mathbf{M}^i, where \mathbf{M} is an arbitrary matrix.
(ii) (2 points)
Explain the intuition behind the result for \boldsymbol{\mu}_{1|1} and \boldsymbol{\Sigma}_{1|1}.
(iii) (4 points)
Assume that \boldsymbol{\mu}_{t-1|t-1} = \frac{1}{t-1} \sum_{i=1}^{t-1} \mathbf{x}_i and that \boldsymbol{\Sigma}_{t-1|t-1} = \frac{1}{t-1} \mathbf{R} (this is consistent with your previous result). Show that \boldsymbol{\mu}_{t|t} = \frac{1}{t} \sum_{i=1}^{t} \mathbf{x}_i and that \boldsymbol{\Sigma}_{t|t} = \frac{1}{t} \mathbf{R}.
(iv) (6 points)
Finally, show that \boldsymbol{\mu}_{t|T} = \frac{1}{T} \sum_{i=1}^T \mathbf{x}_i and \boldsymbol{\Sigma}_{t|T} = \frac{1}{T} \mathbf{R} for all t.
(v) (2 points)
Explain the intuition behind the final result.