With the fundamentals in place, we can start digging into our first model, the Autoregressive Moving Average Model (ARMA).

Consider a random process of the form:

\begin{equation*} X_{t}=\phi X_{t-1}+W_{t}~, \tag{2.1} \end{equation*}

where W_{t} is drawn from \mathcal{N}\left(0, \sigma_{W}^{2}\right) and |\phi|<1. This is known as an autoregressive process because each value of the random variable in our series depends on another random variables in the series. This autoregressive process is of order 1 because the current random variable only has a direct dependence on the previous time-step’s random variable. Let’s calculate the statistics of this random process by expanding the recursion:

\begin{align*} X_{t} & =\phi X_{t-1}+W_{t} \tag{2.2}\\ & =W_{t}+\phi W_{t-1}+\phi^{2} W_{t-2}+\ldots \tag{2.3} \end{align*}

For now, we are ignoring the boundary conditions of our problem (i.e., what happens when we get to t=0 ). The mean of this process is given by:

We can show (by expanding the terms as infinite sums of white noise, recognizing that the covariance of two independent terms is zero, and applying the formula for a geometric series) that the covariance is given by:

Notice that this process is stationary. (Why?) We can generalize this into the broader concept of an AR(p) process:

X_{t} is stationary, and the mean of X_{t} is 0. For cases where the mean of X_{t} is nonzero, we can recast our AR(p) relation as:

Note that, by construction, we assume that our autoregressive process is causal: a random variable only depends on random variables that came before it. In the slides you can find some example of AR processes.

Consider a random process of the form:

\begin{equation*} X_{t}=W_{t}+\theta W_{t-1} \tag{2.13} \end{equation*}

where W_{t} is drawn from \mathcal{N}\left(0, \sigma_{W}^{2}\right) and |\theta|<1. This is known as a moving average process, with this particular process being order 1. Let’s once again calculate our statistics of interest, starting with the mean:

The covariance is given by:

The covariances are much more localized than in the AR case. The is also an example of a stationary process.

We can generalize this into the broader concept of an MA(p) process.

We can combine these two concepts together to build what’s known as an Autoregressive Moving Average Model (ARMA):

Now that we have our full model (of which we will introduce a further extension later in this lecture), we want to understand how to quantify the statistics, measure the model parameters on data, and make new data predictions given the parameters of the model (“forecasting”). In this pursuit it will be useful to reframe our AR and MA processes slightly. Let’s return the the definition of our AR(p) process:

\begin{equation*} X_{t}=\phi_{1} X_{t-1}+\phi_{2} X_{t-2}+\ldots \phi_{p} X_{t-p}+W_{t} . \tag{2.20} \end{equation*}

We can trivially rearrange the terms as follows:

We will also want to introduce the backshift operator, B. We will define our backshift operator such that:

\begin{equation*} B X_{t}=X_{t-1} \tag{2.22} \end{equation*}

This leads us to the autoregressive operator.

Before, we derived the expectation value and covariance of an \operatorname{AR}(1) process by expanding out the \operatorname{AR}(1) equation:

\begin{align*} X_{t} & =\phi X_{t-1}+W_{t} \tag{2.25}\\ & =W_{t}+\phi W_{t-1}+\phi^{2} W_{t-2}+\ldots . \tag{2.26} \end{align*}

The equation on the second line looks like an MA process. In fact, it is an MA process of infinite order. This means that we can write:

We found that for |\phi|<1 our AR(1) process was causal. It’s natural to wonder how we can extend this notion of causality to a higher order process or an ARMA process. As we will see soon, P(B)^{-1} will be vital to this definition. It also confuses the definitions of AR and MA to say that we can write one in terms of the other. To make matter more commplicated we can go down the same rabbit hole with our MA process:

\begin{equation*} X_{t}=W_{t}+\theta_{1} W_{t-1}+\theta_{2} W_{t-2}+\ldots+\theta_{p} W_{t-p} \tag{2.30} \end{equation*}

We can also write this in terms of backshift operators and introduce the moving average operator:

As with the AR process, we can reframe our MA process as:

It turns out that just like we were concerned with causality for our AR process, we can be concerend with invertability for our MA process. Specifically, consider two MA(1) processes:

\begin{align*} X_{t} & =W_{t}+\frac{1}{5} W_{t-1}, \quad p\left(W_{t}\right)=\mathcal{N}(0,25) \tag{2.35}\\ Y_{t} & =V_{t}+5 V_{t-1}, \quad p\left(V_{t}\right)=\mathcal{N}(0,1) \tag{2.36} \end{align*}

In every statistic we can measure on X_{t} and Y_{t} the two processes are indistinguishable^{1}. The only way we would know the difference is to have access to their white noise process, and we do not have that. The only difference is that one of these two processes can be written as an infinite AR process and the other cannot. Specifically, we can write:

But the term on the right-hand side is divergent if |\theta|>1. So only the model with \theta=\frac{1}{5} is invertable. We will also define this more broadly using \Theta(B)^{-1} in a moment. Before we do that, let me convince you of the value of these operators one more time. Imagine we have a simple white noise process:

\begin{equation*} X_{t}=W_{t} \tag{2.39} \end{equation*}

We can trivially also write:

This should be alarming, because I just made a white noise process look like an ARMA( 1,1 ) process.

Let’s resolve these issues in reverse order. First, we can write any ARMA process as:

\begin{equation*} P(B) X_{t}=\Theta(B) W_{t} \tag{2.43} \end{equation*}

We will demand that an ARMA ( p, q ) process has the property that P(B) and \Theta(B) do not share any roots. That is for all complex numbers z_{i} for which P\left(z_{i}\right)=0 and for all complex numbers z_{j} for which \Theta\left(z_{j}\right)=0 there does not exist i or j for which z_{i}=z_{j}. Returning to the white noise example we made look like ARMA( 1,1 ) we have that:

\begin{align*} & P(B)=1-0.5 B \tag{2.44}\\ & \Theta(B)=1-0.5 B \tag{2.45} \end{align*}

We can simplify both equation by dividing by a common factor of 1-0.5 B (eliminating the common root of 2 ) and get:

\begin{align*} & P(B)=1 \tag{2.46}\\ & \Theta(B)=1 \tag{2.47} \end{align*}

thereby demonstrating that this process is in fact white noise. Next we can return to the invertability issue for the MA process:

See Shumway and Stoffer for the proof of this statement.

We can make a similar definition for causality:

Again, see Shumway and Stoffer for the proof of this statement.

Let’s consider the process defined by:

\begin{equation*} X_{t}=0.4 X_{t-1}+0.45 X_{t-2}+W_{t}+W_{t-1}+0.25 W_{t-2} . \tag{2.52} \end{equation*}

We want to know what order ARMA process it is and whether or not it is causal and invertable. It looks like ARMA(2,2), but looks can be deceiving. Let’s get started by finding our P(B) and \Theta(B) operators:

If we factor both equation we find:

\begin{align*} & P(B)=(1+0.5 B)(1-0.9 B) \tag{2.55}\\ & \Theta(B)=(1+0.5 B)^{2} . \tag{2.56} \end{align*}

We can cancel out the shared root to get:

\begin{align*} & P(B)=(1-0.9 B) \tag{2.57}\\ & \Theta(B)=(1+0.5 B) \tag{2.58} \end{align*}

and find that our process is ARMA(1,1) and can be written as:

\begin{equation*} X_{t}=0.9 X_{t-1}+W_{t}+0.5 W_{t-1} \tag{2.59} \end{equation*}

The process is invertible and causal because both roots ( 10 / 9,-2 ) are outside the unit circle.