Time series analysis on TensorFlow and Edward: local level model:P.S. 1

Overview

On the article below, I tried to analyze time series data with local level model. On Stan, I could do it before without problem. But on Edward and TensorFlow, I have been struggling.

Time series analysis on TensorFlow and Edward: local level model

Deep learning and Machine learning methods blog

From the situation above, although it doesn’t work well yet, I got some progress.

Problem

Roughly, I tried to make local level model to time series data on Edward before. At that time, after inference.run(), all the sampled points are just nan.
About this point, I found similar situation and the answer to that.

• Inf and nan loss in Stochastic Volatility Models

Okay, I’ll change the Gamma() points to just tf.Variable().

Re-trial

To the same data, with same model except for the Gamma distribution points, I’ll try.
The code of model part is as following. Gamma distribution part was replaced with tf.Variable() and because of that from the variational feed-dict, I deleted that part.

import tensorflow as tf
import edward as ed
from edward.models import Normal,Gamma

x = [0] * len(data)
u = [0] * len(data)

s_x = tf.Variable(tf.nn.softplus(tf.random_normal([1])), name='s_x')
s_u = tf.Variable(tf.nn.softplus(tf.random_normal([1])), name='s_u')

# start point
u[0] = Normal(loc=tf.zeros([1]), scale=tf.ones([1]))
x[0] = Normal(loc=u[0], scale=s_x)

for i in range(1, len(data)):
    u[i] = Normal(loc=u[i-1], scale=s_u)
    x[i] = Normal(loc=u[i], scale=s_x)

# inference    
q_x = [Normal(loc=tf.Variable(tf.random_normal([1])), scale=tf.nn.softplus(tf.random_normal([1]))) for _ in range(len(data))]
q_u = [Normal(loc=tf.Variable(tf.random_normal([1])), scale=tf.nn.softplus(tf.random_normal([1]))) for _ in range(len(data))]

# variational feed
variational_feed = {}
variational_x_feed = {x_elem: q_x_elem for x_elem, q_x_elem in zip(x, q_x)}
variational_u_feed = {u_elem: q_u_elem for u_elem, q_u_elem in zip(u, q_u)}

variational_feed.update(variational_x_feed)
variational_feed.update(variational_u_feed)

# data feed
data_feed = {x_true: np.array(x_observed).reshape([1,]).astype(np.float32) for x_true, x_observed in zip(x, data)}

inference = ed.KLqp(variational_feed, data_feed)
inference.run(n_iter=1000)

1000/1000 [100%] ██████████████████████████████ Elapsed: 120s | Loss: 743.497

We can check the sampled points.

true_points = []
for elem in q_u:
    true_points.append(elem.sample(100).eval().mean())

To check the sampled points, let’s visually check it.

plt.plot(list(range(len(data))), data)
plt.plot(list(range(len(data))), true_points)
plt.show()

……From the plot, we can see that the sampled points don’t trace the data at all.
Model, Inference method etc.. Something is wrong and I need to check one by one.