Playing with Neural Networks / Boltzman Restricted Machine

For a long while I was interested in neural networks however I didn't have much experience nor knowledge on how to make a network that actually recognizes stuff.

So my first naive implementation involved using binary perceptrons where connections between visible and hidden layers were "evolved" using an evolutionary algorithm. Basically I had a suitability function that would report on how well the network responds to labeled input and then randomly changes the weights and connections of perceptrons to check if there was any improvement.

Obviously this naive approach worked just a little and couldn't learn to a great degree, probably gets stuck in local minimas. Then I lost most of my interest, which was in my teens.

Now, recently I had some spare time to waste, while on my vacation and figured maybe I could see if this recent rush for "deep learning" has some bits of research that I could personally try to replicate and see how it works.

I got hooked on this fascinating video from Google Talks, The Next Generation of Neural Networks from 2007 by Geoffrey Hinton. Although the video is dated by now, I figured I could still try to implement the algorithm that he demonstrated.

The neural network model he presents is based on stochastic binary perceptrons, a Restricted Boltzmann Machine. The idea is to have hidden and visible layers connected in a bi-partite graph and that way the network can project a visible layer on the hidden layer and backwards a well.

The algorithm then uses a relatively simple algorithm for learning that I don't have enough academic background to explain myself; – check wiki page for more details. But I have it implemented myself and it works (gasp!).

You can check the code I built at https://github.com/razzmatazz/nn-brm.

Learning data is contained in images.py, where an "image" is an ASCII picture of some 20 digits and letters. Pretty self-explanatory:

IMAGES_8 = [
    "#####"
    "#   #"
    " ### "
    "#   #"
    "#####",

    " ### "
    "#   #"
    " ### "
    "#   #"
    " ### ",
...

The tooling works so that you learn the network using main.py, where it periodically writes network to _layer1.pkl and periodically logs difference between a random image from learning set and it's reflection by the network. The idea of learning is to make sure the network has an "idea" of what "interesting" images looks like so it is able to recognize and reflect those back:

~/s/neuro/nn-brm :master $ ./main.py
----- -----

      ##
 #     #
 #     #
 #     #
----- -----

...

You can stop the process anytime with CTRL-C. Then you can run read.py to experiment how it works with input it has not learned from yet, – and it then emits 5 outputs that show how the network is recognizing and reflecting the input given:

~/s/neuro/nn-brm :master $ ./read.py
enter image, 5 lines:
  ##
    #
 ##
#
##
-----
  ###
    #
 ###

####
-----
  ###
    #
 ###
#
 ###
-----
  ##
    #
 ###

####
-----
  ###
    #
 ###

#####
-----
  ###
    #
 ###

####

You can see that it fluctuates between the representations of 2 and 3, the network needs more learning to be able to recognize this form of 2 with more certainty.

There is another fun tool there, dream.py that feeds the network a "blank" input and proceeds to run feedback loop on the network where you can see how the network "fantasizes" on the input to generate an image that then fluctuates between the images network has learned from.

This one-layer network model is not suitable to do any categorization, I probably want to build multi-layer network to do that. But coding & toying with this little network model is fun.

Published: 2015-08-06