Category Archives: Uncategorized

Bayesian calculations using MySQL 5.0.1+ Views (ratings)

I’ve been incredibly lax about posting, but I’m going to try to write a new article every week. I have more than enough material, what I’m short on is time. That being said:

One of the sites I manage allows users to cast ratings for other members. It’s always been sort of a kludge but it worked. However, it was far from perfect. I recently had the time to improve it substantially by using Bayesian calculations and a feature of MySQL 5.0.1+ called a view.

For all intents and purposes, a View is a Stored Procedure (for those familiar with Oracle, Sybase or Microsoft SQL).

Bayesian calculations are pretty simple but they can be very CPU intensive. Take a simple example. Say two users have all ratings of 10. User A has six votes and User B has 7 votes. Basically, we want to see which user is more popular over all.

What we have is essentially a table of all votes and we need to track the average number of votes multiply by the average rating. We then add the individuals total votes and multiply that by their rating. We then divide that by the average number of votes plus the number of votes this individual has. The equation looks like this:

((avg_num_votes * avg_rating) + (this_num_votes * this_rating)) / (avg_num_votes + this_num_votes)

I created the view like this: (this assumes we have an existing table in the dB to track ratings, where the columns is “user_account”, the person the vote was cast for. “user” which is the person who cast the vote, and “rating” which is the rating value left by “user” for “user_account”)

CREATE 
    ALGORITHM = UNDEFINED 
    DEFINER = `jon`@`%` 
    SQL SECURITY DEFINER
VIEW `ratings_bayesian` AS
    SELECT 
        `ratings`.`user_account` AS `user_account`,
        ((SELECT 
                COUNT(`ratings`.`user_account`)
            FROM
                `ratings`) / (SELECT 
                COUNT(DISTINCT `ratings`.`user_account`)
            FROM
                `ratings`)) AS `avg_num_votes`,
        (SELECT 
                avg(`ratings`.`rating`)
            FROM
                `ratings`) AS `avg_rating`,
        COUNT(`ratings`.`user_account`) AS `this_num_votes`,
        avg(`ratings`.`rating`) AS `this_rating`
    FROM
        `ratings`
    GROUP BY `ratings`.`user_account`

You call views in MySQL just like you would an ordinary table. The difference is that all of the heavy lifting is handle as a background process.

Now the Display Query is simply

SELECT User_account, ((avg_num_votes * avg_rating) + (this_num_votes * this_rating)) / (avg_num_votes + this_num_votes)
as real_rating FROM `urnawm`.`ratings_bayesian` ORDER BY real_rating DESC

We now have a weighted ratings system that processes in milliseconds and provides an incredibly accurate real time snap shot of your ratings system.

For more information on Bayesian methods and theory, check out http://en.wikipedia.org/wiki/Bayesian.

Nano Syntax Highlighting (using git on Centos)

Having used linux for years, I’ve run the gamut of command line text editors. VI, VIM, EMACS, Pico and of course Nano. My get down and dirty editor, need to edit it fast editor of the moment is Nano.

Have you ever wished that Nano supported Syntax Highlighting? Well it does. All you need are the files. Thanks to git, creating a local repository of syntax highlighting definitions is pretty straight forward.

Just clone the repository of your choice: git clone git://github.com/serialhex/nano-highlight.git ~/.nano and then edit your .nanorc to include the highlighting definitions that you want.


## CSS
include "~/.nano/css.nanorc"

## HTML
include “~/.nano/html.nanorc”

That’s really it. Pretty easy eh?

Sublime Text 2, the best Text Editor you aren’t using.

images

If you’ve been living under a rock, then you haven’t heard about the Sublime Text Editor. You are probably using something like EMACS, TextWrangler or (gasp) Notepad++.

If you are, then you need to stop everything you are doing and run over to the Sublime Text 2 website and install this gem. It exists for pretty much every operating system out there, and out of the box, it’s feature set alone is worth the $80 price tag.

My recommendation is to hit up YouTube and look for Sublime tutorial videos. You will be amazed. Not only that, but you will find mundane repetitive tasks no longer take up a bulk of your editing time.

imgres

Probably the biggest selling point for me was multi-line editing. The ability to edit every single instance of a variable name for instance. A close second was auto complete followed by column select.

That being said, in addition to the already full featured list of capabilities, Sublime Text 2 allows for third party extensions. Need Lint? FTP or SFTP to a remote site? How about a robust Dif utility? All this and more are at your fingertips. Especially after you have installed package control.

I’m not going to waste eny more of your valuable time yammering on. Go, check it out. Thank me later.

Finding Time

I actually have a growing list of things that I’ve been meaning to post about, and I keep getting pulled off on side projects. Also, I’m between projects right now and I’ve decided to get a full time job.

Contracting is fun for the first decade or so, but after a while, you want some stability. My resolution is to try to post at least once a week.

Part Two of the Ulam spiral is almost done so I just need to get off my duff and post it.

Offshore Recruiters

Back in 2005, the Information Technology industry in the United States began offshoring software development at a rapid pace. Suddenly there was a booming IT industry in India, but a floundering one in North America.

It was a grand experiment… that failed.

As project lead times spiraled into months, Quality Assurance became almost non-existent.

It’s turns out that all the money companies thought they would save wasn’t worth the low quality of products they were getting. You really do get what you paid for.

One interesting side effect is that I’ve started getting contacted by recruiters in India. Mind you, they always claim that they are in New Jersey, Chicago or California, but the telephone connections are flaky and the accents often incomprehensible.

I have nothing against companies trying to make money, but I find it ironic that they are trying to profit from the IT jobs they are losing. It’s pragmatic if nothing else.

I do however tell them all the same thing. Thank you for your interest, but I only work with companies that maintain a physical presence in Columbus Ohio (where I live).

The Queer Man’s Guide for Straight Girls (QMGFSG)

QMGFSG_landscape

Hey gang. I’ve been told so many times that I should write a book. What many of you don’t know is that I actually am. It’s really mostly a part time hobby. I don’t really expect it to go anywhere, but who knows. Here then is the preface and an excerpt from Chapter One:

The Queer Man’s Guide for Straight Women
By Daniel Saint James

Preface

Oh god. Not “another” guide written by some self involved twit with a laptop. Let’s face it. Every swinging dick on the planet has advice for you. Opinions are like assholes. Everyone has one and they all stink (unless you shell out sixty bucks for a bleaching kit, but more on that later).

Why would you want to read “this” particular guide. What makes “this” one special? Well I’m glad you asked (just play along here). “Queer Eye” meted out wonderful advice to hapless straight men. “What Not To Wear” attempts the same trick with straight women with mixed results (WNTW is technically not a gay show, but when one of your advisors is a gay man, that tends to be a given). There are dozens of guides, television shows and websites purporting to be “the” solution to all of your problems.

I’m not here to do that. This book is about obtaining the perspective you need to figure things out on your own. Not about giving you the answers to things. Give a man a fish and you feed him for a day. Teach a man to fish, and you’ve added a new competitor to the already over crowded fishing industry.

Sure, you will find plenty of, “Maybe you should try this” and lots of “Never, never under any circumstances do this” but think of it like Algebra 1 class, where you’re given the basics that you need to solve larger more complex problems. The only difference is that I’m not a fat man in polyester pants and a short sleeve button down shirt with pit stains on it.

Who should read this book? Well, personally, I think everyone should, but that’s because I have bills to pay. Specifically, this book is written for single heterosexual women between the ages of 25 and 45. I think parts of it might be appropriate for anyone. Granted, there may be parts that AREN’T appropriate for ANYONE.

But there. I’ve outlined my target audience. My editor is no doubt having a seizure right now. Let’s get on with it.

How am I qualified to give advice to love lorn spinsters? I’m a 42 year old bisexual father of four. I have a gay daughter, a bisexual daughter, a straight son and an infant daughter of un-determined sexuality (at least until she’s old enough to figure out what she is). I’ve dated men, I’ve dated women. I have a ton of advice, but it’s not about what you should do. Basically, it’s a list of things you shouldn’t do. Why am I an expert? Because I’ve made a lot of mistakes. Learn from my mistakes. It may be that the purpose of my life, is merely to serve as a warning to others.

I haven’t given up. While love is a fairy tale based on neurochemistry, there have been precedents. I do think it’s possible. Beyond a shadow of a doubt, I HAVE figured out what not to do.

Call me a “Hopeful Romatic”, but I know that the right person just might be out there. The real question for anyone is: “Am I ready to find them?”

Think of this guide less as a “guide” and more of an open letter from your queer best friend that you never have to buy drinks for (unless you really want to).

Chapter One
Get Over Yourself

“You would worry a lot less about what people were saying about you, if you realized how little they were” – Stacey G

No one will ever care more about you than you do. That’s the truth. You’ll “hear” lots of people say otherwise, but when push comes to shove, we always look out for number one. Your mother “might” care almost as much for you as you do, but I doubt it. Consider this possibility, especially if you have siblings, a step father, or a family cat.

You may occasionally meet men who seem to have unnatural relationships with their mothers. When you meet them, run. They do. There is nothing more unnatural then a thirty-four year old man who still puts his mother first, unless of course, she’s the Queen of England.

Just because we are our own built in number one fans, doesn’t mean that anyone else will think that. Ever. This is a very real possibility that you need to come to grips with before we can go any further in this book. When someone tells you that you are the most important thing in the world to them, you need to ask yourself what else they are lying to you about.

I’m not trying to be cynical here, and I’m not trying to suck the joy out of any relationship you might have. I’m just trying to get you to admit the truth to yourself.

The person you pretend to be when you are around others is only half as interesting as the person you actually are. Don’t be afraid to let those around you see you for who you actually are. You might be surprised by the number of people that suddenly list themselves as your friend.

The truth is, we can usually only guess at what the people around want us to be anyway. Generally, they want us to be silent so that they can talk about themselves. The easiest way to be the most interesting person at a party, is to ask everyone else about themselves.

About the only time people actually take notice of us, is when we do something in incredibly poor taste. Then they LOVE to talk about us. Lets not worry about this particular situation right now, but we will address that later in the book.

Right now, we need to make you realize that no matter how fabulous you think you are, the people around you only get about ten percent of that off of you. That’s all right. Would you really want everyone to worship you and hang on your every word? Well, maybe you would, but for the sake of argument right now, we’ll say that you don’t.

Perspective can be a bitch if it happens too fast. Cosmologists and Mathematicians have gone insane trying to fathom the un-fathomable. We aren’t dealing with anything even close to that scale. We are going to keep it simple.

Fire is hot. Water is wet. In the grand scheme of things, you just aren’t that important. A freckle on a wart on a homeless dog.

Before I take this analogy too far, be mindful of one fact. It’s really all a matter of perspective. I’m sure even Nicole Kidman has days where she feels like no one loves her.

/end

The Evolution of PHP and the rise of Composer and Git

This could be a really long post, but I’m going to keep it pretty short. PHP has been around in various forms since 1994. That’s almost 20 years, which is ancient in the tech world.

It’s gone through many changes over the years but it’s truly beginning to mature. Like most evolutionary changes, it took a couple of things to accelerate the process.

There are a number of content versioning systems. Over the years, I’ve used most of them on various projects hear and there. Git has been around for a while too, but it was always relegated to the status of the “geek cvs” with it’s arcane command line interface, and often difficult to grasp methodologies.

I’m not sure when the community began to rally around git or why. Perhaps the community at large finally became savvy enough to comprehend it, or git itself had enough useful tools and utilities mature so that the entry point or ease of accessibility became more attainable.

For whatever reason, git has become the gold standard. Developers who aren’t using it are missing out on an effective tool to track code changes and a vibrant community of knowledgable people exchanging ideas.

Git made the next major step possible. A lovely dependency manager called Composer.

PHP has had dependency managers in the past. Everyone has wrestled with PEAR. It has still been a struggle to keep software up to date and organized. Spaghetti code has been rampant and often left in place simply because the only alternative was to re-develop an entire project from scratch.