Simple Bayesian Modeling on Edward by Hamiltonian Monte Carlo

Overview

On this article, I'll re-write by Edward the simple bayesian model that was written on Stan.
This time, my target is from the following article.

• Simple Bayesian modeling by Stan

Actually, I already wrote the article, Simple regression by Edward: variational inference. There, I re-wrote the model on Edward. But at that time, I used variational bayesian method for inference.
Stan uses Hamiltonian Monte Carlo. So, this time, I'll use Hamiltonian Monte Carlo on Edward and re-write the model.

Bayesian prediction interval from pystan output

On this article, I’ll leave the simple memo to get bayesian prediction interval from the sampled points of PyStan output and visualize it.
Basically, I’ll use the code from the article, Simple Bayesian modeling by Stan.

How to make the local level model with seasonal effect

Overview

On time-series analytics, we frequently need to think about a recurring pattern. In the context of time-series analytics, a recurring pattern is referred to as a seasonal effect.

For example, please see the image below. This image is the plotting of a time series data. As you can see, it has recurrent pattern.


On this article, I'll make the local level model with seasonal effects on Stan.

Simple trial of various types of time series analytics

By air passengers data, which is typical time-series data, I'll try some time-series analytics methods. Actually, about some points, I'm not sure if it is really appropriate or not. So, if you find some wrong or incorrect points, please let me know.

Anti-pattern of the local level model with explanatory variable

Overview

On this article, I'll try anti-pattern of the local level model with explanatory variable.
Before, on the article, Local level model with explanatory variable to time series data on Stan , I made the local level model with explanatory variable to time series data. From the article, I set the model as followings.
$u_t \sim Normal(u_{t-1}, \sigma_ξ^2)$
$β_t \sim Normal(β_{t-1}, \sigma_τ^2)$
$y_t \sim Normal(u_t + β_t \times x_t, \sigma_e^2)$
There, I set state disturbances $σ_τ^2$ as zero. That’s because on the book, An Introduction to State Space Time Series Analysis (Practical Econometrics), it is written that these state disturbances are usually fixed on zero to establish a stable relationship between $y_t$ and $x_t$. This time, as experiment, I’ll try the model without setting $σ_τ^2$ as zero, meaning I’ll set $β_t$ as changeable.

Edward modeling to an artificial data

Overview

On the article below, I switched method on Edward model from variational method to Hamiltonian Monte Carlo.

• Hamiltonian Monte Carlo on TensorFlow and Edward

As an another example, I'll try same thing to the model of the following article.

• Edward modeling to artificial data with random effects

In a nutshell, I'll make model for an artificial data and get sample by Hamiltonian Monte Carlo.

Hamiltonian Monte Carlo on TensorFlow and Edward

Overview

On this article, I tried Hamiltonian Monte Carlo algorithm to the simple data by TensorFlow and Edward.
Edward lets us use variational inference, Gibbs sampling and Monte Carlo method. And by relatively small changes, we can switch the methods. So I'll try simple HML model here.
About Hamiltonian Monte Carlo itself, I'll write another article for it.

Local level model to time series data on Stan

Overview

On the articles below, I tried local level modeling to time series data on Edward and am still struggling.

Time series analysis on TensorFlow and Edward: local level model

Deep learning and Machine learning methods blog

Time series analysis on TensorFlow and Edward: local level model:P.S. 1

On the article below, I tried to analyze time series data with local level model. On Stan, I could do it before without problem. But on Edward and TensorFlow, I have been struggling. Deep learning and Machine learning methods blog From the situation above, although it doesn't work well yet, I got some progress.

On this article, I’ll express by Stan what I wanted on Edward. In a nutshell, I’ll write local level model to time series data on Stan.

Time series analysis on TensorFlow and Edward: local level model:P.S. 1

Overview

On the article below, I tried to analyze time series data with local level model. On Stan, I could do it before without problem. But on Edward and TensorFlow, I have been struggling.

Time series analysis on TensorFlow and Edward: local level model

Deep learning and Machine learning methods blog

From the situation above, although it doesn’t work well yet, I got some progress.

Edward modeling to artificial data with random effects

Overview

By Edward, I’ll try to make the model with random effect.
There are some ways to fulfill that. On this article, I’ll follow the style that the Edward tutorial takes.

• Edward Tutorial

Fashion-MNIST exploring using Keras and Edward

Overview

On the article, Fashion-MNIST exploring, I concisely explored Fashion-MNIST dataset.

We can get access to the dataset from Keras and on this article, I’ll try simple classification by Edward.

Simple Baysian Neural Network with Edward

Overview

Edward can enable us to convert TensorFlow code to Baysian one. I’m not used to Edward. So for the training, I’m tackling with converting some TensorFlow code to Edward one. On this article, I tried to convert simple neural network model to Baysian neural network one.

The purpose of this article is to convert the TensorFlow code I posted before to Baysian one by Edward.

Baysian neural network model

By Edward, we can relatively easily convert the model using TensorFlow to probabilistic one.
The regression model for iris data is from the article below.

Simple regression model by TensorFlow

Neural network is composed of input, hidden and output layers. And the number of hidden layers is optional. So the simplest network architecture has just one hidden layer. On this article, I'll make the simplest neural network for regression by TensorFlow.

In a nutshell, the model is to predict one target value from three features. About the details, please check the article.

Simple regression by Edward: variational inference

Overview

Edward is one of the PPL(probabilistic programming language). This enables us to use variational inference, Gibbs sampling and Monte Carlo method relatively easily. But it doesn’t look so easy. So step by step, I’ll try this.

On this article, simple regression, tried on the article Simple Bayesian modeling by Stan, can be the nice example. So I did same things by Edward, using variational inference.

Hierarchical Bayesian model's parameter Interpretation on Stan

Usually, Hierarchical Bayesian model has many parameters. So apparently, the interception to the sampled point’s statistical information looks complex.

On the article below, I made a Hierarchical Bayesian model to the artificial data. Here, by using almost same but simpler data, I’ll make a model and try to interpret.

Hierarchical Bayesian model by Stan: Struggling

I'll try to make Hierarchical Bayesian model to the artificial data by Stan. Hierarchical Bayesian model lets us write the model with a high degree of freedom.

Hierarchical Bayesian model by Stan: Struggling

I’ll try to make Hierarchical Bayesian model to the artificial data by Stan. Hierarchical Bayesian model lets us write the model with a high degree of freedom.

From Wikipedia,

Bayesian hierarchical modelling is a statistical model written in multiple levels (hierarchical form) that estimates the parameters of the posterior distribution using the Bayesian method.[1] The sub-models combine to form the hierarchical model, and the Bayes’ theorem is used to integrate them with the observed data, and account for all the uncertainty that is present. The result of this integration is the posterior distribution, also known as the updated probability estimate, as additional evidence on the prior distribution is acquired.

Bayesian modeling to data with heteroscedasticity by Stan

Before, I wrote about the data with heteroscedasticity.

What is heteroscedasticity and How to check it on R

Linear regression with OLS is simple and strong method to analyze data. By the coefficients, we can know the influence each variables have. Although it looks easy to use linear regression with OLS because of the simple system from the viewpoint of necessary code and mathematics, it has some important conditions which should be kept to get proper coefficients and characteristics.

How to deal with heteroscedasticity

On the article below, I wrote about heteroscedasticity. Linear regression with OLS is simple and strong method to analyze data. By the coefficients, we can know the influence each variables have. Although it looks easy to use linear regression with OLS because of the simple system from the viewpoint of necessary code and mathematics, it has some important conditions which should be kept to get proper coefficients and characteristics.

This time, I’ll make the model again but with Python and Stan.

Bayesian multiple regression by Stan

Overview

On the article, Simple Bayesian modeling by Stan, I made a simple linear regression by Stan and PyStan. So, as an extension of it, I made multiple regression model on the same manner to show how to do Bayesian modeling roughly.

Simple Bayesian modeling by Stan

Overview

About Bayesian modeling, we can use some languages and tools. BUGS, PyMC, Stan. On this article, I made simple regression model by using Stan from Python.