Strength in Numbers Blog Tue, 16 Jun 2015 19:18:41 +0000 en-US hourly 1 Beware of Lucene DAO when constructing queries Wed, 03 Jun 2015 04:51:14 +0000 When using Lucene in Java or Scala, you may be tempted to skip the QueryParser and use the “DAO” (for lack of a better term) to construct queries using the classes provided. It is generally a best practice to use DAOs and such abstractions when available over raw query compilation for a variety of reasons, foremost being security (implicit injection protection) and query syntax integrity.

However, you may experience perplexing, incorrect result sets with your Lucene query if the following circumstances are true:

  • Your index is written with an analyzer other than the default StandardAnalyzer (e.g. EnglishAnalyzer or any of the plethora of others).
  • Your query is a boolean query with number of OR (aka SHOULD) clauses where ≥ 2.
  • Your query requires a minimum m number of boolean clauses should match where m ≥ n.

Ordinary query, incorrect results

Here is a simple example of a query that exhibits the latter two circumstances above as built entirely with the DAO (code examples henceforth using Scala for brevity):

// Boolean query with OR clauses
val q = new BooleanQuery
q.add(new TermQuery(new Term("articleTitle", "thanks")), BooleanClause.Occur.SHOULD)
q.add(new TermQuery(new Term("articleTitle", "obama")), BooleanClause.Occur.SHOULD)
q.add(new TermQuery(new Term("articleTitle", "barack")), BooleanClause.Occur.SHOULD)

// Match at least 2 of the clauses

This query in plain English means “find documents that contain at least 2 of the terms thanksobama, and barack.”

Now imagine an index of documents as follows written with EnglishAnalyzer (or some other non-StandardAnalyzer):

doc  articleTitle
1    president jails congress; thanks, obama!
2    obama thanks al qaeda for joining in the fight against isis
3    obama to produce presidential library consisting entirely of ebooks

Running the query on the above documents should yield 2 hits—documents #1 and #2. However, you will receive 0 results in Lucene 3.5.0 (and possibly other versions; did not check).

Unfortunately for me, I was stuck with Lucene 3.5.0 in this particular codebase. Luckily I found a way to sidestep the bug by avoiding the DAO for at least part of the query construction.

Same query, but without DAO (and working now!)

val analyzer = new EnglishAnalyzer // or the same analyzer used to write the index
val qp = new QueryParser(Version.LUCENE_35, "articleTitle", analyzer)
val q = new BooleanQuery
q.add(qp.parse("thanks"), BooleanClause.Occur.SHOULD)
q.add(qp.parse("obama"), BooleanClause.Occur.SHOULD)
q.add(qp.parse("barack"), BooleanClause.Occur.SHOULD)

// Match at least 2 of the terms

Surprise, surprise, this works! Documents #1 and #2 from before will match as expected.

Note on protecting against query injection

If you must use

  as in the case above, you should also make it a habit to use 
 (a static method) on the string you pass to the parse method (e.g.
myQueryParser.parse(QueryParser.escape("potentially dangerous user input"))
 ). The reasons are beyond the scope of this post; just Google “query injection” and pick one of the endless writings on that.

]]> 0
Religious Hate Crime in the USA 1996–2013 Data Viz Fri, 06 Mar 2015 10:19:18 +0000 I stumbled upon the FBI Uniform Crime Reporting data on Hate Crime and was compelled to visualize some numbers considering how comprehensive the data are over a few decades. The data I chose to visualize concern religious hate crime for the years 1996 to 2013. Interestingly, though not surprisingly I suppose, the year of the September 11, 2001 attacks shows the largest deviation from the trend. That’s all for now!

The total number of religiously motivated crime victims has not changed appreciably over this time period and thus
is not worth visualizing. However, you can hover over points in the chart above to see the raw number of victims
at each point.

Religious affiliation is based on values interpolated between survey years conducted in 1990, 2001, and 2008.

]]> 0
Fixing autocomplete (autofill) on AngularJS form submit Sun, 12 Jan 2014 09:11:32 +0000 Many browsers support autocomplete but do not trigger “change” events on inputs when the user initiates autocomplete. This is a problem for many libraries and code in the wild that rely on performing some action (e.g. input validation) when input data change.

With respect to AngularJS forms, the problem becomes obvious if you are using AngularJS form validation. If you autofill an empty form then press Submit, AngularJS will think the inputs are still empty:

AngularJS unaware of autofilled inputs

AngularJS unaware of autofilled inputs

Here is a simple login form subject to this issue:

<form ng-submit="submitLoginForm()">
    <input type="email" ng-model="email" ng-required />
    <input type="password" ng-model="password" ng-required />
    <button type="submit">Log In</button>

Unfortunately the underlying issue of not having an appropriate event related to autofill must be addressed by browser vendors. However in the meantime, we can use a custom directive to ensure AngularJS knows about form changes made by autofill at time of form submit.

Patch directive

In CoffeeScript

myApp.directive 'formAutofillFix', ->
  (scope, elem, attrs) ->
    # Fixes Chrome bug:!topic/angular/6NlucSskQjY
    elem.prop 'method', 'POST'

    # Fix autofill issues where Angular doesn't know about autofilled inputs
    if attrs.ngSubmit
      setTimeout ->
        elem.unbind('submit').submit (e) ->
          elem.find('input, textarea, select').trigger('input').trigger('change').trigger 'keydown'
          scope.$apply attrs.ngSubmit
      , 0

In JavaScript

myApp.directive('formAutofillFix', function() {
  return function(scope, elem, attrs) {
    // Fixes Chrome bug:!topic/angular/6NlucSskQjY
    elem.prop('method', 'POST');

    // Fix autofill issues where Angular doesn't know about autofilled inputs
    if(attrs.ngSubmit) {
      setTimeout(function() {
        elem.unbind('submit').submit(function(e) {
          elem.find('input, textarea, select').trigger('input').trigger('change').trigger('keydown');
      }, 0);

Directive in use

The directive is simple to use; just apply it to your form and away you go:

<form ng-submit="submitLoginForm()" form-autofill-fix>
    <input type="email" ng-model="email" ng-required />
    <input type="password" ng-model="password" ng-required />
    <button type="submit">Log In</button>

]]> 14
How to use rich objects and typed objects with Bootstrap Typeahead Fri, 21 Jun 2013 14:44:38 +0000 The JS components of Twitter Bootstrap are somewhat lacking in functionality when compared to well established plugins that do the same things (autocompletes, tooltips, etc.). However, I still use them on Bootstrap projects just because it is nice to have homogeneous interfaces to things—and, of course, it is easy then to jump on other Bootstrap projects and be in familiar territory right off the bat.

One of the continuing problems I face is with Bootstrap Typeahead, in particular its lack of native support for non-String items. I constantly come across this; rare is it that one can identify an item in a list by that item’s user-friendly string representation, i.e. more often than not you need to be working with a fully typed thing, not just a string.

Other autocompletes and typeaheads support rich objects out of the box but Typeahead needs a little massaging—fortunately not much, and not anything requiring changes to the core. Here is a sample showing a Typeahead that uses a rich UsState class (a natural extension to the Twitter demo using string state names):

Bootstrap Typeahead only gets tripped up with rich object sources for the updater method namely. The key component here is ensuring your class has a toString() method that serializes instances correctly, as well as a fromString() static method to deserialize. For my purposes I include the popular json2.js for JSON.stringify() and JSON.parse():

 * @return {String} Serialized state.
 * @see fromString()
UsState.prototype.toString = function() {
    return JSON.stringify(this);

 * @static
 * @param {String} serializedState
 * @return {UsState}
 * @see toString()
UsState.fromString = function(serializedState) {
    return $.extend(new UsState(), JSON.parse(serializedState));

Then, in your Typeahead updater:

updater: function(state) {
            state = UsState.fromString(state);
            $stateMessage.html('<strong>' + + '</strong> has ' + state.numElectoralVotes + ' electoral votes.');

Note that Typeahead passes the stringified object, which we deserialize, and expects a string return to stuff into the text input.

And because I love CoffeeScript so much, here is the same demo in CS:

]]> 0
PHP error_reporting E_STRICT at runtime Fri, 05 Apr 2013 08:06:12 +0000 Often times PHP applications configure error_reporting at runtime, perhaps because it appears to be more bootstrappable:

error_reporting(E_ALL & ~E_NOTICE);

// ...or...

ini_set('error_reporting', E_ALL & ~E_NOTICE);

In dev environments, it is often desirable to log errors of level E_STRICT to be warned about practices that are “deprecated or not future proof.” Unfortunately, enabling E_STRICT at runtime using one of the methods above will never work—giving you a false sense of security with respect to strict-level errors! This is because PHP performs strict checks at compile time before your runtime error_reporting configs are set. In other words, this will not work:

// Neither of these:
error_reporting(E_ALL | E_STRICT);
ini_set('error_reporting', E_ALL | E_STRICT);

To get E_STRICT errors, you must take care to set the value of error_reporting before compile time, such as in php.ini, or httpd.conf or .htaccess if you are running Apache:

Setting error_reporting in php.ini:

error_reporting = E_ALL | E_STRICT

Setting error_reporting in httpd.conf or .htaccess:

php_value error_reporting 32767 # Equivalent int value of E_ALL | E_STRICT
                                # for PHP 5.3.3 (later versions may vary)

The manual does mention this topic, though it may be easily overlooked.

Last but not least, always remember to turn display_errors off in production!

display_errors Off in php.ini

display_errors = Off

display_errors Off in httpd.conf or .htaccess

php_flag display_errors Off

]]> 0
Decoupling JS and HTML with AngularJS Tue, 19 Mar 2013 07:48:19 +0000 The primary reason for the influx of interest in JavaScript MVC frameworks is improving separation of concerns between HTML and JS. I’m going to demonstrate why decoupling of HTML and JS is so important and how AngularJS helps solve the problem.

Why decoupling is so important

Traditional DOM selection, traversal, and manipulation is the root of tightly coupled HTML and JS—and the bane of a front-end developer’s existence. It’s not that it is difficult to write or understand, but rather that it is exceedingly difficult to maintain. Even the most seemingly innocuous JS containing node names, CSS classes, or event bindings causes a tightly coupled architecture to quickly emerge. Consider a simple jQuery app that lets the user arbitrarily add items to a list:

I took it a step further than many jQuery apps in the wild by introducing a rudimentary template for the list item rather than compositing the item entirely in JS. However, the app still naturally suffers from some serious coupling. Here are a few realistic scenarios the app might face in the future:

  1. Multiple vehicle entry methods needed.
    • Will need to change element selection by ID to selection by class.
    • Will need to adjust DOM selection in JS.
    • What is conceptually a change only to HTML clearly will also require changes to JS.
  2. List item template changes.
    • Will need to change template DOM selection and manipulation in JS.
    • Same as above; conceptually HTML-only, actually not.
  3. Multiple displays of garage needed.
    • Similar problems as #1 and #2.

The code changes required to implement these features are significant and involve disparate contexts and thus are costly and prone to error. Here is a simple diagram to visualize the coupling:

Separation of concerns

Surely there must be a better way. Consider how the diagram above might change if we introduce MVC-inspired separation of concerns:

Enter AngularJS

Here’s the same app using basic AngularJS instead of jQuery. (If you’re keen on UX, you’ll notice I left out a few usability features from the previous example. We’ll add those back momentarily using more advanced AngularJS techniques.)

The DOM selection, traversal, and manipulation disappear! What you see is effectively the diagram of the three circles—the vehicle data is now abstract and the DOM and JS coupling is virtually eliminated. The three “what ifs” mentioned before which are conceptually HTML-only changes now actually are HTML-only changes since the underlying data model doesn’t change (and the JS is only referencing the abstract data model).

If I add only a little bit of HTML on the view layer, I get cool new features that operate on the same data set—no changes to JS required (conceptual view-only change now a reality):

<p ng-show="!vehicles.length">
    Your garage looks a little empty.
    Do you want <a ng-click="addVehicle({name: 'Ferarri', type: Vehicle.TYPE_CAR})">a Ferrari</a>?
  <p ng-show="vehicles.length">You have {{vehicles.length}}
    vehicle<span ng-show="vehicles.length != 1">s</span> in your garage.</p>
  <form ng-submit="addVehicle({name: addCarName, type: Vehicle.TYPE_CAR}); addCarName = ''">
      <label for="addCarName">Add a car:</label>
      <input type="text" ng-model="addCarName" placeholder="Car name" />
      <button type="submit">OK</button>
  <form ng-submit="addVehicle({name: addMotoName, type: Vehicle.TYPE_MOTORCYCLE}); addMotoName = ''">
      <label for="addMotoName">Add a motorcycle:</label>
      <input type="text" ng-model="addMotoName" placeholder="Motorcycle name" />
      <button type="submit">OK</button>

Usability sugar as promised

So you may have noticed a few UX niceties in the jQuery version of the app that we lost later on:

  • Vehicle name input focused after changing vehicle type.
  • Vehicle name input focused after clicking submit button with empty name and getting “name required” error.
  • Vehicle name input focused after clicking submit button and successfully adding a vehicle.

Doing this in AngularJS is to some extent another topic—still along the lines of separation of concerns, but not exactly MVC—very specific to the problem domain of the DOM. The way AngularJS solved this particular problem is clever, unique among JS frameworks, and extremely robust; the solution is in what are called directives, “a way to teach HTML new tricks.” What I am doing here is essentially inventing a couple new HTML element attributes that will abstract out the concept that is focusing an input on various cues:

Note the new directives

defined in the JS along with associated usage in HTML as
, which elegantly encapsulates the behavior and still manages to abstract away DOM selection, etc.

Comments or questions are always welcome!

]]> 0
HTML5 Canvas Gradient Creator Tue, 05 Mar 2013 08:54:06 +0000 There are plenty of CSS3 gradient creators out there, but much to my surprise there was not a single gradient creator for HTML5
<canvas />
to be found. Thus I present to you the only one of its kind, the HTML5 Canvas Gradient Creator.


  • Visual point-and-click, drag-and-slide creation of gradients for HTML5/JS canvas
  • 1-n color stops; as many as you need
  • 0-255 bit alpha support
  • Linear and radial gradients
  • Linear gradient rotation
  • Radial gradient inner and outer circle position
  • Elliptical radial gradients
  • Saving of custom presets (browser local storage)
  • Valid, commented HTML5 and JS code output
  • Responsive design

Minimum compatibility

This is minimum compatibility for usage of this tool, as well as for gradient code generated by this tool. Earlier versions of Chrome, Safari, or Opera might be supported.

  • Chrome: WinXP/OS X SL Chrome 14
  • Firefox: WinXP FF4 (FF3 for gradient code generated by tool), OS X SL FF5
  • IE: Win7-8 IE9*
  • Safari: WinXP/OS X SL Saf 4
  • Opera: WinXP Opera 10, OS X SL Opera 11.1

* Inclusion of FlashCanvas or excanvas might enable support for gradients in earlier versions of IE.

Languages and libraries

Check out the source on Github.

]]> 2
Solving AngularJS empty select option caused by jQuery UI effect Thu, 17 Jan 2013 08:42:34 +0000 If you’re anything like me, while working with AngularJS you will soon want to integrate your “standard library” of jQuery and UI plugins into AngularJS directives. Recently on the job, I wrote an AngularJS directive to apply a jQuery UI effect to an element—I wanted to briefly shake a small form if an input was invalid (such as you often see in native app GUIs). The directive looked something like this:

 * Shakes the element using jQuery UI shake if expression is truthy.
 * Usage:
 *   <ANY shake="{expr}" [after-shake="{expr}"]>...</ANY>
MyApp.directive('shake', function() {
  return function(scope, elem, attrs) {
    scope.$watch(attrs.shake, function(shake, wasShaking) {
      if(shake && !wasShaking) {
        // The jQuery UI shake part
        elem.effect('shake', function() {
          // Digest afterShake if it was given
          attrs.afterShake !== undefined && scope.$apply(function() {

Much to my surprise, during the shake effect a

<select />
 in the form went from having a non-empty option selected to having an empty option selected, and there was not an empty option in the select to begin with.

You may already be familiar with a select in AngularJS “mysteriously” gaining an empty option. That’s not mysterious at all as it turns out; it is reasonable behavior when the model associated with the select is not given a valid value. Here is a quick sample to demonstrate this problem and solution:

This was clearly not the problem I experienced. I had initialized the model correctly, and the empty selected option only appeared after the jQuery UI effect had finished. Admittedly, I still don’t know what jQuery UI is doing to cause this. However, without delving into the core of jQuery UI, I do at least have a solution to the problem for now; the answer lies in using ng-options instead of

<option />

Here is a sample of the problem and solution side by side. Switch to the Result tab, then click the “Shake” button to see one select lose its value and the other retain its value as desired.

Cookie to the first person who can say what jQuery UI is doing to trigger the problem when using

<option />
elements vs. ng-options!

]]> 5
Old Fashioned Metronome in AngularJS, HTML5, and CSS3 Tue, 01 Jan 2013 06:17:35 +0000 This went from a fun Saturday project inspired by a suggestion from my brother to a weekend-long, no-holds-barred tapping into some of the more powerful browsers’ HTML5 and CSS3 feature sets within the context of AngularJS. Check out the full screen app here.

Old Fashioned Metronome App


  • Responsive
  • Can update the tempo while metronome is on (try tabbing focus to the bob,
    then using arrow keys to move it while it is swinging)
  • HTML5 Audio API
  • CSS animation
  • Fancy but not very practical source code view buttons to the right.
    (Just View Source or see the fiddle, it’s much easier lol.)

Libraries, frameworks, etc.

Compatibility as of 12/31/2012

  • Chrome 23.0 all features OK
  • Firefox 17.0 all features OK
  • Safari 5.1.7 all features except audio OK
  • Opera 12.12 all features OK
  • IE 10 nothing appears (due to use of background-size + background-position?)
  • Chrome, Safari all features OK

Pshhh… Yea, right! Firefox and Opera on my Android are close but are missing key features.


Used Ceaser CSS Easing Tool to create the custom
easing function for the wand “return to center” animation.

]]> 5
Make AngularJS $http service behave like jQuery.ajax() Fri, 21 Dec 2012 07:26:57 +0000 There is much confusion among newcomers to AngularJS as to why the $http service shorthand functions ($, etc.) don’t appear to be swappable with the jQuery equivalents (, etc.) even though the respective manuals imply identical usage. That is, if your jQuery code looked like this before:

(function($) {'/endpoint', { foo: 'bar' }).success(function(response) {
    // ...

You may find that the following doesn’t exactly work for you with AngularJS out of the box:

var MainCtrl = function($scope, $http) {
  $'/endpoint', { foo: 'bar' }).success(function(response) {
    // ...

The problem you may encounter is that your server does not appear to receive the { foo: 'bar' } params from the AngularJS request.

The difference is in how jQuery and AngularJS serialize and transmit the data. Fundamentally, the problem lies with your server language of choice being unable to understand AngularJS’s transmission natively—that’s a darn shame because AngularJS is certainly not doing anything wrong. By default, jQuery transmits data using Content-Type: x-www-form-urlencoded and the familiar foo=bar&baz=moe serialization. AngularJS, however, transmits data using Content-Type: application/json and { "foo": "bar", "baz": "moe" } JSON serialization, which unfortunately some Web server languages—notably PHP—do not unserialize natively.

Thankfully, the thoughtful AngularJS developers provided hooks into the $http service to let us impose x-www-form-urlencoded on all our transmissions. There are many solutions people have offered thus far on forums and StackOverflow, but they are not ideal because they require you to modify either your server code or your desired usage pattern of $http. Thus, I present to you the best possible solution, which requires you to change neither server nor client code but rather make some minor adjustments to $http‘s behavior in the config of your app’s AngularJS module:

// Your app's root module...
angular.module('MyModule', [], function($httpProvider) {
  // Use x-www-form-urlencoded Content-Type
  $['Content-Type'] = 'application/x-www-form-urlencoded;charset=utf-8';

   * The workhorse; converts an object to x-www-form-urlencoded serialization.
   * @param {Object} obj
   * @return {String}
  var param = function(obj) {
    var query = '', name, value, fullSubName, subName, subValue, innerObj, i;
    for(name in obj) {
      value = obj[name];
      if(value instanceof Array) {
        for(i=0; i<value.length; ++i) {
          subValue = value[i];
          fullSubName = name + '[' + i + ']';
          innerObj = {};
          innerObj[fullSubName] = subValue;
          query += param(innerObj) + '&';
      else if(value instanceof Object) {
        for(subName in value) {
          subValue = value[subName];
          fullSubName = name + '[' + subName + ']';
          innerObj = {};
          innerObj[fullSubName] = subValue;
          query += param(innerObj) + '&';
      else if(value !== undefined && value !== null)
        query += encodeURIComponent(name) + '=' + encodeURIComponent(value) + '&';
    return query.length ? query.substr(0, query.length - 1) : query;

  // Override $http service's default transformRequest
  $httpProvider.defaults.transformRequest = [function(data) {
    return angular.isObject(data) && String(data) !== '[object File]' ? param(data) : data;

Do not use jQuery.param() in place of the above homegrown param() function; it will cause havoc when you try to use AngularJS $resource because jQuery.param() will fire every method on the $resource class passed to it! (This is a feature of jQuery whereby function members of the object to parametrize are called and the return values are used as the parametrized values, but for our typical use case in AngularJS it is detrimental since we typically pass “real” object instances with methods, etc.)

Now you can go forward with using $ and other such methods as you would expect with existing server code that expects x-www-form-urlencoded data. Here are a few sample frames of the final result for your day-to-day, end-to-end code (i.e. what you hoped and dreamed for):

The HTML template


Response: {{response}}

The client code (AngularJS)

var MainCtrl = function($scope, $http) {
  $scope.loading = true;
  $'/endpoint', { foo: 'bar' }).success(function(response) {
    $scope.response = response;
    $scope.loading = false;

The server code (PHP)

header('Content-Type: application/json');

// Ta-da, using $_POST as normal; PHP is able to
// unserialize the AngularJS request no problem
echo json_encode($_POST);

Other notes

So you may wonder now, is it possible for PHP to read the JSON request from stock AngularJS? Why, of course, by reading the input to PHP and JSON decoding it:

$params = json_decode(file_get_contents('php://input'));

Obviously the downside to this is that the code is a little less intuitive (we’re used to $_POST, after all), and if your server-side handlers are already written to rely on $_POST, you will now have to change server code. If you have a good framework in place, you can probably effect a global change such that your input reader will transparently detect JSON requests, but I digress.

Thank you for reading. I hope you enjoyed my first POST. (Get it?!)

EDIT March 12, 2014: Moved the param() function def up a level to help interpreter only create one such instance of that function.
]]> 102