Bayesian calculations using MySQL 5.0.1+ Views (ratings)

I’ve been incredibly lax about posting, but I’m going to try to write a new article every week. I have more than enough material, what I’m short on is time. That being said:

One of the sites I manage allows users to cast ratings for other members. It’s always been sort of a kludge but it worked. However, it was far from perfect. I recently had the time to improve it substantially by using Bayesian calculations and a feature of MySQL 5.0.1+ called a view.

For all intents and purposes, a View is a Stored Procedure (for those familiar with Oracle, Sybase or Microsoft SQL).

Bayesian calculations are pretty simple but they can be very CPU intensive. Take a simple example. Say two users have all ratings of 10. User A has six votes and User B has 7 votes. Basically, we want to see which user is more popular over all.

What we have is essentially a table of all votes and we need to track the average number of votes multiply by the average rating. We then add the individuals total votes and multiply that by their rating. We then divide that by the average number of votes plus the number of votes this individual has. The equation looks like this:

((avg_num_votes * avg_rating) + (this_num_votes * this_rating)) / (avg_num_votes + this_num_votes)

I created the view like this: (this assumes we have an existing table in the dB to track ratings, where the columns is “user_account”, the person the vote was cast for. “user” which is the person who cast the vote, and “rating” which is the rating value left by “user” for “user_account”)

CREATE 
    ALGORITHM = UNDEFINED 
    DEFINER = `jon`@`%` 
    SQL SECURITY DEFINER
VIEW `ratings_bayesian` AS
    SELECT 
        `ratings`.`user_account` AS `user_account`,
        ((SELECT 
                COUNT(`ratings`.`user_account`)
            FROM
                `ratings`) / (SELECT 
                COUNT(DISTINCT `ratings`.`user_account`)
            FROM
                `ratings`)) AS `avg_num_votes`,
        (SELECT 
                avg(`ratings`.`rating`)
            FROM
                `ratings`) AS `avg_rating`,
        COUNT(`ratings`.`user_account`) AS `this_num_votes`,
        avg(`ratings`.`rating`) AS `this_rating`
    FROM
        `ratings`
    GROUP BY `ratings`.`user_account`

You call views in MySQL just like you would an ordinary table. The difference is that all of the heavy lifting is handle as a background process.

Now the Display Query is simply

SELECT User_account, ((avg_num_votes * avg_rating) + (this_num_votes * this_rating)) / (avg_num_votes + this_num_votes)
as real_rating FROM `urnawm`.`ratings_bayesian` ORDER BY real_rating DESC

We now have a weighted ratings system that processes in milliseconds and provides an incredibly accurate real time snap shot of your ratings system.

For more information on Bayesian methods and theory, check out http://en.wikipedia.org/wiki/Bayesian.

Nano Syntax Highlighting (using git on Centos)

Having used linux for years, I’ve run the gamut of command line text editors. VI, VIM, EMACS, Pico and of course Nano. My get down and dirty editor, need to edit it fast editor of the moment is Nano.

Have you ever wished that Nano supported Syntax Highlighting? Well it does. All you need are the files. Thanks to git, creating a local repository of syntax highlighting definitions is pretty straight forward.

Just clone the repository of your choice: git clone git://github.com/serialhex/nano-highlight.git ~/.nano and then edit your .nanorc to include the highlighting definitions that you want.


## CSS
include "~/.nano/css.nanorc"

## HTML
include “~/.nano/html.nanorc”

That’s really it. Pretty easy eh?

Sublime Text 2, the best Text Editor you aren’t using.

images

If you’ve been living under a rock, then you haven’t heard about the Sublime Text Editor. You are probably using something like EMACS, TextWrangler or (gasp) Notepad++.

If you are, then you need to stop everything you are doing and run over to the Sublime Text 2 website and install this gem. It exists for pretty much every operating system out there, and out of the box, it’s feature set alone is worth the $80 price tag.

My recommendation is to hit up YouTube and look for Sublime tutorial videos. You will be amazed. Not only that, but you will find mundane repetitive tasks no longer take up a bulk of your editing time.

imgres

Probably the biggest selling point for me was multi-line editing. The ability to edit every single instance of a variable name for instance. A close second was auto complete followed by column select.

That being said, in addition to the already full featured list of capabilities, Sublime Text 2 allows for third party extensions. Need Lint? FTP or SFTP to a remote site? How about a robust Dif utility? All this and more are at your fingertips. Especially after you have installed package control.

I’m not going to waste eny more of your valuable time yammering on. Go, check it out. Thank me later.

Forcing html input type entry with jQuery

You should also do both client side and server side validation, never rely on just one method. Fortunately, jQuery makes the client side super easy. I won’t discuss validation per se, instead, we’ll focus on limiting the TYPE of data that can be entered in an input field.

Just add these two simple functions to your $(document).ready section. From now, on, any html input with the associated class assigned will limit the type of data that can be entered.

$(document).ready(function() {
    // common functions using class to control input entry
    $('.only_alpha').bind('keyup blur',function(){
        $(this).val( $(this).val().replace(/[^a-zA-z]/g,'') ); }
    );
 
    // Apply the class numericOnly to any html input to force it to only accept numbers
    $(".only_numeric").bind('keypress', function (e) {
        return ( e.which != 13 && e.which != 8 && e.which != 0 && (e.which < 48 || e.which > 57) && e.which != 46) ? false : true;
    });	
});

Now, to have an input that only accepts alpha characters, you just add the class “only_alpha” to it.

<input type="text" name="name_first" class"only_alpha" />

Likewise, to limit an input to numeric characters only, just add the class “only_numeric”.

<input type="text" name="name_first" class"only_numeric" />

For true compliance, you can also use the new HTML5 types. I won’t go into these here, but you can get a nice overview at http://diveintohtml5.info/forms.html.

Homework assignment, a Ulam Spiral Generator – Part One (Redux)

See the DEMO

Requires WebKit Browser, i.e. Chrome, Firefox, Safari

One of the things I enjoy about coding, is the ongoing development. Code is never really done, because I can always think of ways to improve it. This post was originally supposed to be the second half of the Ulam Spiral Generator (my experiments with writing an HTML, jQuery based Ulam Sprial, or Ulam Cloth generator), but certain improvements have been burning in the back of my mind and I decided to go ahead and make them now, rather then later.

The changes were mostly about the look and feel and some usability issues. I didn’t like having buttons on the page that didn’t do anything when the spiral wasn’t visible. Also, after playing with the interface, I decided that I didn’t like having to select “Go” every time I changed the row/column count. Now, the spiral adjusts live as you change the count.

I’m not going to go into much detail on the changes, they speak for themselves, and they weren’t too complex. I’ll just present the code and provide a link the revised code and you can decide for yourself.

<html>
    <head>
        <title>Ulam Spiral Generator with jQuery</title>
        <meta name="description" content="Ulam Spiral Generator written with jQuery. The Ulam Spiral is also know as the Ulam Cloth.">
        <!-- I like to use cdn's whenever possible -->
        <link href="http://ajax.googleapis.com/ajax/libs/jqueryui/1.8/themes/base/jquery-ui.css" rel="stylesheet" type="text/css"/>
 
        <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js"></script>
          <script src="http://ajax.googleapis.com/ajax/libs/jqueryui/1.8/jquery-ui.min.js"></script>
 
          <style>
              * {
                  font-family: Arial;
              }
              body {
                  background-color: #5872e8;
                  background-image: -webkit-gradient(radial, 50% 0%,0,50% 0%,200, from(#2c4160), to(#5872e8));
                  background-image: -webkit-radial-gradient(50% 0%, #5872e8, #2c4160);
                  background-image: -moz-radial-gradient(50% 0%, #5872e8, #2c4160);
                  background-image: -o-radial-gradient(50% 0%, #5872e8, #2c4160);
                  background-image: -ms-radial-gradient(50% 0%, #5872e8, #2c4160);
                  background-image: radial-gradient(50% 0%, #5872e8, #2c4160);
              }
 
              a {
                  color: #fff;
              }          
 
              #ulam ul {
                  list-style-type: none;
                  text-align: center;
              }
 
              #ulam li {
                  background-color: #fff;
                  position: relative;
                  float: left;
                  width: 40px;
                  height: 40px;
                  text-align: center; 
                  border: 1px solid black;
                  border-radius: 5px;
                  margin: 2px;
                  padding 5px;
                  line-height: 40px;
                  box-shadow: 2px 2px 3px #333;
              }
              #toolbar {
 
              }
 
              #output {
                  border-radius: 50%;
                  border: 2px solid white;
                  line-height: 40px;
                  text-align: center;
                  padding: 2px;
                  margin: 5px;
                  color: #fff;
                  background-color: red;
                  box-shadow: 3px 3px 5px #333;
              }
              .prime {
                  color: #fff;
                  background-color: #5872e8 !important;
              }
        </style>
    </head>
 
    <body>
        <script>
            $(document).ready(function() {
                // Apply the class numericOnly to any html input to force it to only accept numbers
                // this class function isn't currently used on this page, but I find it handy to keep around
                $("input.numericOnly").bind('keypress', function (e) {
                    return ( e.which != 13 && e.which != 8 && e.which != 0 && (e.which < 48 || e.which > 57) && e.which != 46) ? false : true;
                });
			    // we assign the class "true" to the table when we find prime numbers, we just toggle the class the reveal the prime numbers
                $("#showprimes").on('click', function () {
                    $(".true").toggleClass("prime");
			    });
 
                $("#animatespiral").on('click', function () {
                    cols = $("#cols").val();
                    maxsize = cols*cols;
                    var j = 1;
 
                    (function spiralnext() {
                        if (j < maxsize+1) {
                            $('#' + j).effect('pulsate', { times: 1 }, 200, spiralnext);
                            j++;
                        }
                    })();
			    });
 
			    // we'll update the div when the slider is updated so the user has feedback
                $('#cols').change(function() {
                    var x = $(this).val();
                    $('#output').text(x);
                });
                submit_num();
            });
 
            function submit_num() {
                // remove any td from previous 
                $("#ulam li").remove();
 
                var cells = [];
                var x = 0,
                y = 0,
                delta = [0, -1],
                cols = $("#cols").val();
                maxsize = cols*cols;
                for (i = 1; i <= maxsize; i++) {
                    cells[i] = {};
                    // I know I didn't have to add maxsize to the value, I just don't naturally visualize negative numbers
                    cells[i]['row'] = y + maxsize;
                    cells[i]['col'] = x + maxsize;
                    cells[i]['dat'] = i;
                    prime = isPrime(i);
                    cells[i]['pri'] = prime;   
                    //console.debug('POINT ', i, y + maxsize, x + maxsize);
 
                    if (x === y || (x < 0 && x === -y) || (x > 0 && x === 1-y)){
                        // change direction
                        delta = [-delta[1], delta[0]]            
                    }
 
                    x += delta[0];
                    y += delta[1];        
                }
                // Now we sort the array based on row, then column, this actually stumped me for a bit, I'm not used 
                // to sorting arrays on two fields, but fortunately, as usual, stackoverflow to the rescue
                cells.sort( function(a,b){ 
                    if (a.row != b.row) {
                        return b.row - a.row;
                    }
                    return a.col - b.col 
                });
 
                var trackrows = cells[1]['row'];
 
                for (j = 0; j < maxsize; j++) {
                    if (trackrows != cells[j]['row']) {
                        // we need to close previous rows and start a new one if the row number changes
                        // I'm thinking about rewriting this using ul, li
                	    trackrows = cells[j]['row'];
                    	$("#ulam ul").append('<div style="clear: both;">');
                    }
                    console.log('POINT ', cells[j]['dat'], cells[j]['row'], cells[j]['col']);
                    $("#ulam ul").append('<li id="'+cells[j]['dat']+'" class="'+cells[j]['pri']+'">'+cells[j]['dat']+'</li>');
                } 
                $('#tools').show();              
            }
 
            function isPrime(num) {
                var prime = num != 1; // Everything but 1 can be prime
                for (var i = 2; i < num; i++) {
                    if (num % i == 0) {
                        prime = false;
                        break;
                    };
                };
                return prime;
            };            
 
        </script>
 
            <h2>Ulam Spiral Generator</h2>
            <p><a href="http://www.tangledwebsites.com/homework-assignment-a-ulam-spiral-generator/">Read the blog post "Homework assignment, a Ulam Spiral Generator" (Redux) by Daniel Saint James</a></p>
 
            <form>
                <div> 
                    <p>Use the slider to select the number of rows/columns you would like. 
                    <span id="output">10</span>&nbsp;<input id="cols" type="range" min="10" max="40" value="10" onchange="submit_num();" />
                    <span id="tools" style="display: none;">
                        <input id="showprimes" type="button" value="Toggle Primes" /> <input id="animatespiral" type="button" value="Animate Spiral" />
                    </span>
                </div>
 
                <div style="clear: both;">
 
                <div id="ulam">
                    <ul>
 
                    </ul>
                </div>
 
            </form>
    </body>
</html>

See the DEMO

Requires WebKit Browser, i.e. Chrome, Firefox, Safari

Running startup scripts on CentOS Linux (Fedora or RHEL)

Every distro has different ways of handling start up scripts. I maintain a pretty busy server and occasionally it needs to be restarted. There are a couple of Java services that need to run as well as a chat server that needs to be started.

CentOS (and all other distros built around the Red Hat model) use a really simple method. There is a script /etc/rc.local, that you can place all of your startup items in.

Here is a simple example

#!/bin/sh
#
# All the other init scripts will run first. rc.local runs after them
 
sh /var/www/virtual/webapps/RealChatNew/realchat start

That’s really is all there is to it. There’s your handy little tip for the day!

Finding Time

I actually have a growing list of things that I’ve been meaning to post about, and I keep getting pulled off on side projects. Also, I’m between projects right now and I’ve decided to get a full time job.

Contracting is fun for the first decade or so, but after a while, you want some stability. My resolution is to try to post at least once a week.

Part Two of the Ulam spiral is almost done so I just need to get off my duff and post it.

Homework assignment, a Ulam Spiral Generator – Part One

See the DEMO

Requires WebKit Browser, i.e. Chrome, Firefox, Safari

Recently, while at a job interview, the interviewer posed an interesting problem. Could I, in any language, write code to generate a counter clockwise spiral of numbers from the center and radiating out to the edge.

Ulam_spiral_howto_all_numbers.svg

In my limited time, I worked through the problem on a dry erase marker board, and figured out the basics. I got nowhere near the answer though. “Obviously,” he remarked, “I’m not expecting you to have the answer in the limited time we have today.” He was mostly trying to see my process for working through a problem.

Mind you, whenever you give me an interesting problem, my brain tends to get stuck in analysis mode. That evening, I used my GoogleFu to discover that this particular pattern was known as a Ulam Spiral.

The Ulam Spiral, or Prime Spiral, Prime Number Spiral or Ulam Cloth, was devised by Stanislaw Ulam in 1963, while bored during a lecture (the mind reels at what it takes to make a mathematician bored), who started doodling on a piece of paper.

What makes the Ulam Spiral interesting is the odd way that prime numbers reveal themselves on the grid over time, often apparently organizing themselves into diagonal lines (they still haven’t found an explanation for this). For more on the Ulam Sprial, check out the Wikipedia entry.

I found a number of places where the math was laid out, I even found a number of programs designed to teach the concept to math classes, but no where, did I find anyone who had written a solution, let alone one in HTML.

Challenge accepted!

I had an interesting weekend, without a lot of time to focus on the issue, but finally Sunday evening, I was able to sit down and play with it.

Here is the code in it’s entirety. I’ll break it down later:

<html>
    <head>
        <title>Ulam Spiral Generator with jQuery</title>
        <meta name="description" content="Ulam Spiral Generator written with jQuery. The Ulam Spiral is also know as the Ulam Cloth.">
        <!-- I like to use cdn's whenever possible -->
        <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js"></script>
 
        <style>
            * {
                font-family: Arial;
            }
            td { width: 50px; height: 50px; text-align: center; 
                border-style:solid;
                border-width:1px;
                border-radius: 5px;
            }
            #spiral {
                width: 100%;
                margin: auto;
            }
            .prime {
                color: #fff;
                background-color: #666;
            }
 
        </style>
    </head>
 
    <body>
        <script>
            $(document).ready(function() {
                // Apply the class numericOnly to any html input to force it to only accept numbers
                // this class function isn't currently used on this page, but I find it handy to keep around
                $("input.numericOnly").bind('keypress', function (e) {
                    return ( e.which != 13 && e.which != 8 && e.which != 0 && (e.which < 48 || e.which > 57) && e.which != 46) ? false : true;
			    });
			    // we assign the class "true" to the table when we find prime numbers, we just toggle the class the reveal the prime numbers
                $("#showprimes").on('click', function () {
                    $(".true").toggleClass("prime");
			    });			    
			    // we'll update the div when the slider is updated so the user has feedback
                $('#cols').change(function() {
                    var x = $(this).val();
                    $('#output').text(x);
                });                 
 
            });
 
            function submit_num() {
                // remove any td from previous 
                $("#ulam td").remove();
                $("#ulam tr").remove();
 
                var cells = [];
                var x = 0,
                y = 0,
                delta = [0, -1],
                cols = $("#cols").val();
                maxsize = cols*cols;
                for (i = 1; i <= maxsize; i++) {
                    cells[i] = {};
                    // I know I didn't have to add maxsize to the value, I just don't naturally visualize negative numbers
                    cells[i]['row'] = y + maxsize;
                    cells[i]['col'] = x + maxsize;
                    cells[i]['dat'] = i;
                    prime = isPrime(i);
                    cells[i]['pri'] = prime;   
                    //console.debug('POINT ', i, y + maxsize, x + maxsize);
 
                    if (x === y || (x < 0 && x === -y) || (x > 0 && x === 1-y)){
                        // change direction
                        delta = [-delta[1], delta[0]]            
                    }
 
                    x += delta[0];
                    y += delta[1];        
                }
                // Now we sort the array based on row, then column, this actually stumped me for a bit, I'm not used 
                // to sorting arrays on two fields, but fortunately, as usual, stackoverflow to the rescue
                cells.sort( function(a,b){ 
                    if (a.row != b.row) {
                        return b.row - a.row;
                    }
                    return a.col - b.col 
                });
 
                var trackrows = cells[1]['row'];
 
                for (j = 0; j < maxsize; j++) {
                    if (trackrows != cells[j]['row']) {
                        // we need to close previous rows and start a new one if the row number changes
                        // I'm thinking about rewriting this using ul, li
                	    trackrows = cells[j]['row'];
                    	$("#ulam").append('</tr><tr>');
                    }
                    console.log('POINT ', cells[j]['dat'], cells[j]['row'], cells[j]['col']);
                    $("#ulam").append('<td class="'+cells[j]['pri']+'">'+cells[j]['dat']+'</td>');
                }                
            }
 
            function isPrime(num) {
                var prime = num != 1; // Everything but 1 can be prime
                for (var i = 2; i < num; i++) {
                    if (num % i == 0) {
                        prime = false;
                        break;
                    };
                };
                return prime;
            };            
 
        </script>
 
        <div id="spiral">
            <h2>Ulam Spiral Generator</h2>
            <p><a href="http://www.tangledwebsites.com/homework-assignment-a-ulam-spiral-generator/">Read the blog post</a></p>
            <table>
                <tbody id="ulam">
 
                </tbody>
            </table>
 
            <form onsubmit="submit_num();">
                <p>Use the slider to select the of rows/columsns you would like.<br />
 
                <input id="cols" type="range" min="9" max="20" value="9" />
                &nbsp;&nbsp;<input type="button" value="Go" onclick="submit_num();" /><br />         
 
                <span id="output">9</span><br />
 
                <input id="showprimes" type="button" value="Toggle Primes" />
            </form>
 
        </div>
 
    </body>
</html>

So this is complex, but far simpler than I had originally thought. Let’s walk through the pieces.

The very first thing I do, is include the Google CDN for jQuery.

<html>
    <head>
        <!-- I like to use cdn's whenever possible -->
        <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js"></script>

Then I define some really basic styles:

        <style>
            * {
                font-family: Arial;
            }
            td { width: 50px; height: 50px; text-align: center; 
                border-style:solid;
                border-width:1px;
                border-radius: 5px;
            }
            #spiral {
                width: 100%;
                margin: auto;
            }
            .prime {
                color: #fff;
                background-color: #666;
            }
 
        </style>
    </head>

The next little bits are just two jQuery functions that we will use later.

        <script>
            $(document).ready(function() {
                // we assign the class "true" to the table when we find prime numbers, we just toggle the class the reveal the prime numbers
                $("#showprimes").on('click', function () {
                    $(".true").toggleClass("prime");
                });			    
		// we'll update the div when the slider is updated so the user has feedback
                $('#cols').change(function() {
                    var x = $(this).val();
                    $('#output').text(x);
                });               	    
            });

I’ll explain the core function in Part Two of this posting.

See the DEMO

Requires WebKit Browser, i.e. Chrome, Firefox, Safari

Blast From the Past

corsair

Back in the late 90′s, I still spent a lot of time with my various hobbies. One of these was 3D modeling and animation. I used to spend hours designing textures, shapes, polygon deformations, and then ten times as much time waiting on models to render.

I was all right, not great, but good for a amateur.

Then in Nine-teen Ninety-Eight, I got a call from a company in Hollywood (who’s name I won’t drop) based on my third submitted portfolio in as many years.

Anxiously, I flew out to California, interviewed for about six hours, at the end of which, they offered me a position.

At about half what I was currently making as a systems engineer.

Thus ended my dream to be a computer animator in Hollywood.