PHP array_insert

I had the need the other day to create an ordered collection object. I wanted to be able to add elements to the end of the collection as well as at any point in the middle of the collection. Since I was using PHP as the language to create this object, an array was the obvious construct to use as the internal implementation for this collection. Adding elements to the end of an array is trivial, but inserting elements into the middle of the array turned out to be a bit more difficult.

My first thought was that PHP has all kinds of array functions so there must be an array_insert function. This assumption turned out to be wrong. My second thought is that I would have to cut the array into two pieces, insert the new element to the end of the first piece and then glue to two arrays back together. In fact, I coded this before reading the documentation for the array_splice function more closely.

The array_splice function has two optional parameters which are very helpful in creating an array_insert function. The first optional parameter is the length option. This option specifies how many elements to remove from the original array starting at the specified offset. The second optional parameter is the replacement option. This parameter specified what to replace the removed elements with. So, in other words, you can tell array_splice to remove 6 elements from an array starting at offset 2 and to replace the 6 elements with what ever is in the replacement array.

This is perfect for mimicking an array_insert function. If you specify to remove 0 elements and pass in the new element as an array of 1 item, it gets inserted at the specified index. Now for the example:

function array_insert($array, $index, $new_element)
{
    return array_splice($array, $index, 0, array($new_element));
}

There you have it, array_insert nice and clean and optimized in c.

PHPUnit Custom Extension - Class ... could not be found in ...

(For the impatient, include your class early, that is the solution.)

So I have been using PHPUnit for quite some time now and find that it is a great library for testing. I also find that testing really helps in development. I also find that PHPUnit can be infuriating to use at times. For instance, I wanted to create a new base class that extendeds PHPUnit_Framework_TestCase so I can add some custom logging or something like that to all of my tests. So I simply create,

class lib_MyBaseUnitTest extends PHPUnit_Framework_TestCase {
    ....
}

Then I create my test cases,

class tests_MyFirstTest extends lib_MyBaseUnitTest {
    ....
}

Now whenever I run

phpunit tests/MyFirstTest.php

 I get the error:

Class lib_MyBaseUnitTest could not be found in tests/MyFirstTest.php

So clearly from the error the base class in not being loaded...? Actually, the problem is that the base class is being loaded by the autoloader when phpunit tries to load the test and the base class gets in the way. The phpunit code that causes the problem is located in StandardTestSuiteLoader.php.

PHPUnit_Util_Class::collectStart();
PHPUnit_Util_Fileloader::checkAndLoad($suiteClassFile, $syntaxCheck);
$loadedClasses = PHPUnit_Util_Class::collectEnd();

The collectStart call gathers the list of loaded classes and stores them in memory. The checkAndLoad call then includes each of the files listed in your suite, or the file specified on the commandline, thus loading more classes into memory. The collectEnd call now gathers the new list of loaded classes and diffs the result with the list collected in the collectStart call. Each of the resulting classes should be a test case that needs to be run. The problem is that the autoloader loaded both the specified test case and its base class. When PHPUnit tests to see if it can run the base class as a test, it can't and therefore throws and exception, "Class ... can not be found in ...".

So the solution in the end that I have taken is to place require_once calls in my bootstrap file for all custom base classes. This way the class will be in the initial list of classes created by the collectStart call. It will not be in the $loadedClasses variable and therefore will not cause an exception to be thrown.

PHP Errors and Exceptions

I learned something about php error handling the other day. I was digging through the docs and discovered how to catch errors, warnings, and such and then re-throw them as exceptions. This is great because I already implemented a default exception handling and logging system to catch all exceptions. Now I can use the same routines for PHP errors. The keys to making this work are the set_error_handler function and the ErrorException class. The set_error_handler has been around for a long time, but I have never really used it. This is a failing on my part. The ErrorException class seems to have been made especially for this type of operation. The ErrorException documentation shows exactly how to use these two things together for object oriented goodness in PHP.

<?php
  function exception_error_handler($errno, $errstr, $errfile, $errline ) {
    throw new ErrorException($errstr, 0, $errno, $errfile, $errline);
  }
  set_error_handler("exception_error_handler");
?>

The function name passed to set_error_handler will be called if there is ever a PHP error. The function set up to handle errors takes the error information, constructs an ErrorException and then throws it. Errors never go directly out to the user or logs, they are always wrapped in an exception. Doing this give you greater control over error handling.

In the user comments on the ErrorException page troelskn gives a suggestion for improvement to the basic example above. He suggests the following code.

<?php
function exception_error_handler($severity, $message, $filename, $lineno) {
  if (error_reporting() == 0) {
    return;
  }
  if (error_reporting() & $severity) {
    throw new ErrorException($message, 0, $severity, $filename, $lineno);
  }
}
set_error_handler('exception_error_handler');
?>

This code allows the error reporting value set in php.ini to effect the behavior of the exception_error_handler. Only errors of a high enough severity will be passed on as exceptions. Using his code, if you have warnings suppressed, they will not raise an exception. They will just not be reported. I hope this removes one more of the little PHP quarks for you.

Many to Many Join with Propel

In my symfony application I have many images. Each of those images has lots of tags. Each tag might be associated with many different images. Yes, the classic many to many relationship. Now, using propel I would like to get all of my images, with their associated tags. Seems simple enough... except that "there is no magical support for many-to-many relationships in Propel." Propel is not very good at many to many relationships.

The propel documentation gives an example of how to retrieve objects for this type of situation. There are two major problems with the example. First, it only works if every image has a tag because the joins are inner joins instead of left joins. Second, it runs 1+n querys against the database which is terrible if you are retrieving a lot of images at once. For a lot of cases the example may be sufficient, but it did not work in my case.

The first step to making this work the way I needed was to construct a query using left joins retrieving all the needed data. I did this by creating a new method inside my ImagePeer class. I chose this place because my real goal is to get a list of images, I just want to do it efficiently.

public static function doSelectWithTags(Criteria $c, $con = null){
  $c = clone $c;
  $c->setDbName(self::DATABASE_NAME);
  
  $c->addJoin(ImagePeer::ID, ImageTagPeer::IMAGE_ID, Criteria::LEFT_JOIN);
  $c->addJoin(ImageTagPeer::TAG_ID, TagPeer::ID, Criteria::LEFT_JOIN);
  
  ImagePeer::addSelectColumns($c);  
  ImageTagPeer::addSelectColumns($c);
  TagPeer::addSelectColumns($c);
  

  $first_image_tag_col = (ImagePeer::NUM_COLUMNS - ImagePeer::NUM_LAZY_LOAD_COLUMNS) + 1;
  $first_tag_col = $first_image_tag_col + (ImageTagPeer::NUM_COLUMNS - ImageTagPeer::NUM_LAZY_LOAD_COLUMNS);
  
  $rs = BasePeer::doSelect($c, $con);

The next step was to hydrate all of the objects. This is fairly complex, but much of the code is taken straight from other methods in the ImageTagPeer class.

$images = array();
  $tags = array();
  
  while($rs->next()) {
    $image_id = $rs->getInt(1);

    if(array_key_exists($image_id, $images)){
      $image = $images[$image_id];
    }else{
      $omClass = ImagePeer::getOMClass();
      $cls = Propel::import($omClass);
      $image = new $cls();
      $image->hydrate($rs);
      $images[$image_id] = $image;
      $image->initImageTags();   
    }

    $tag_id = $rs->getInt($first_tag_col);
    if($tag_id > 0){
      $omClass = ImageTagPeer::getOMClass();
      $cls = Propel::import($omClass);
      $image_tag = new $cls();
      $image_tag->hydrate($rs, $first_image_tag_col);
    
      if(array_key_exists($tag_id, $tags)){
        $tag = $tags[$tag_id];
      }else{
        $omClass = TagPeer::getOMClass();
        $cls = Propel::import($omClass);
        $tag = new $cls();
        $tag->hydrate($rs, $first_tag_col);
        $tags[$tag_id] = $tag;
        $tag->initImageTags();
      }
      $image->addImageTag($image_tag);
      $tag->addImageTag($image_tag);
    }
  }
  return array_values($images);  
}

The last thing that I had to figure out was how to work around propel's query caching system. I never found a very elegant way to solve this problem, but I did find a way. Even though I set up all the object connections between images and tags, propel doesn't understand how it was done and will try to rebuild the connections. When the call getImageTags is made the empty criteria passed in does not match the left join criteria that was used to find the ImageTag objects. Therefore, propel refetches the ImageTags to make sure it has all the right ones. This is usually the desired functionality, but there is no nice way to get around it. In my Image class I added the method getImageTags to override the parent functionality. It can be passed a flag that says "just use what ever you have."

public function getImageTags($criteria = null, $con = null, $check_cleanliness = true){
  if($check_cleanliness){
    return parent::getImageTags($criteria, $con);
  }else{
    return $this->collImageTags ? $this->collImageTags : array();
  }
}

Not the most pretty code, but very functional. I can now get a list of images with all of the associated tags in a single query across three database tables.

for, find, and whitespace

I recently wrote a short bash script to create thumbnails of all of my images and place them into a single directory. Basically I needed a bunch of images that I could do some testing with. It seemed like a simple enough task, but then I ran into the dreaded bash whitespace problem. This seems to be an issue for a lot of people but it still took a long time to find a solution that would work in my case.

Here is an illustration of the problem I had.

for FILE in `find`
do
  echo $FILE
done

In this example if a file or directory has the name "My File", the for loop will iterate once for the word "My" and once for the word "File". By default bash parses the string and splits it on tabs, spaces, and newlines. You can fix this a few different ways. The best way I determined was to change what bash considers to be whitespace. You do that by changing the value of IFS.

export IFS=$'\n'
for FILE in `find`
do
  echo $FILE
done

After figuring that out the rest of the script was pretty easy. The complete script is below for anyone who might like to do the same thing. This was a quick hack to work for my situation and you will certainly need to adjust things for what you need.

export IFS=$'\n'
for PATH in `find /Users/username/Pictures/ -type f -name '*.jpg' \
        -o -name '*.gif' -o -name '*.jpeg' \
        -o -name '*.tif' -o -name '*.png'`
do
 FILE=${PATH##*/}
 /usr/local/bin/convert -size 180x180 $PATH -thumbnail 90x90 images/$FILE
done

Testing Propel Models in Symfony 1.1

I have been struggling for many hours trying to figure out how write unit tests to test my propel models in symfony. The documentation discussing how to test propel models has not been updated for symfony 1.1. There is one key thing that must be done to initialize the propel classes so that you can test your models. The key is that you must create a sfDatabaseManager object. The following is a complete test.

include(dirname(__FILE__).'/../bootstrap/unit.php');
require_once(dirname(__FILE__).'/../../config/ProjectConfiguration.class.php');
$configuration = new ProjectConfiguration(realpath($_test_dir.'/..'));

$databaseManager = new sfDatabaseManager(
$configuration->getApplicationConfiguration('frontend', 'dev', true)
);

$t = new lime_test(1, new lime_output_color());
$model = MyModelPeer::retrieveByPK(1);
$t->is($model->getName(), 'Test', 'Name retrieved correctly');

All the credit for this information goes to pentium133 for his post on the symfony forum that contained the solution.

Omniture Web Analytics and Blogger

I moved recently. Both my blog and my person. One of the side effects of moving to a new home was that my little Pentium II webserver in my basement had to retire. I no longer have my own IP address. So I had to find a new place to host my blog. Blogger or Blogspot worked out well for me because I could keep my own domain name.

One thing I have struggled with, switching from my own install of webpress to blogger, is trying to get Omniture's javascript code to work with my blog. I tried fruitlessly to get the code mixed into a template. I could never get it to render correctly. Maybe someone with more Blogger template experience could get it to work, but I could not. But I finally figured out an easy way to go about it. You can add a HTML/JavaScript page element to the layout of the blog.

Blogger has lots of different types of page elements you can add, such as polls, slideshows, or a blog archive. They also have a page element labeled HTML/JavaScript. This page element allows you to add any third-party JavaScript functionality to your blog. I can't believe it took me this long to find it.

Now knowing where I could place JavaScript code it was pretty easy to deploy. The one catch was that I had to make a small change to the Omniture code. The Omniture code comes in two parts. There is the base code and the configuration code. Generally the configuration code includes the base code. I hade no where to save a javascript file for the configuration code to include. Therefore I removed the link to include the code and just pasted the code above the configuration code. This required that I add <script type='text/javascript'></script> tags around the code, but other than that it was a fairly painless process.

Yes, this means that I am now collecting statistics about you as you read this. It is fun to see where people are coming from and what pages they look at. It looks like Google collects stats about everyone using blogger already too. Thanks for visiting. Come back soon.

2007 Backpacking highlight video

I created a short highlight movie of some of the backpacking trips I went on this summer with our boy scout troop. I created the movie with iPhoto and iMovie. It was surprisingly easy. After finishing I remembered one of the apple ads. “Better Results” is the personification of how easy it is to create a very nice movie. There is sample music, useful clip transitions, easy music adjustments, clip effects, and many more useful tools. The tools are available, but more than that, they are easy to use. It took only about an hour to build the movie, and upload it to YouTube. It was fun to waste the hour creating the movie.

paul

Closed in the Last 7 Days

I use trac for most of my project management purposes. One issue I had for quite some time was that I did not have a report showing me what tickets were closed in the last x amount of time. After studying the stock reports and digging through the database I found what I needed. The following query generates a report showing a list of closed tickets over the last 7 days (actually 604800 seconds).

SELECT id AS ticket, summary, component, version,
 priority, t.type AS type, owner, resolution,
 changetime AS _changetime, description AS _description,
 reporter AS _reporter
FROM ticket t
LEFT JOIN enum p ON p.name = t.priority AND p.type = 'priority'
WHERE status = 'closed'
  AND _changetime > (strftime('%s','now')-604800)
  ORDER BY changetime DESC

I hope this report will help someone else. Enjoy!

Timeout a ruby system call

I was rather disappointed that I could not set a timeout on a system call. This is especially a problem because of the very poor threading model that ruby currently has. But, at least there is a way to get the job done. The following is a method I wrote to make a system call and then kill it if the timeout is reached.

require 'timeout'
class System
def self.system_with_timeout(timeout, *args)
 if ( (pid = fork) == nil )
   #child process
   @@logger.debug(args.join(' '))
   exec(*args)
 else
   success = false
   #parent process
   begin
     #TODO if the process fails return false
     success = Timeout::timeout(timeout){ Process.waitpid(pid) }
   rescue Timeout::Error
     @@logger.error "***** Timeout error"
     Process.kill("HUP", pid)
     Process.detach(pid)
   end
   success
 end
end
end

ActionMalier and GMail

This post has been in my queue to write about for over a month now. I just wanted to give thanks to Stephen Chu for posting about how to use ActionMailer with gmail. At FamilyLearn we use Google Apps for all of our collaboration services. It has been great to hook our application into the same system. This helps so I do not have to maintain a mail server. Here is the link to his excellent article.

http://www.stephenchu.com/2006/06/how-to-use-gmail-smtp-server-to-send.html

RMagic and Fedora Core 6

So it turns out that installing the rmagic gem requires you to install the Microsoft True Type fonts. This seems like a very odd requirement to me. Maybe there is a flag that says install the gem without the fonts, but I could not find it. Fedora Core 6 does not come with an rpm to install the corefonts. You need to visit corefonts.sourceforge.net. Daniel Resare has put together all of the resources that you need to build your own rpm. You may want to visit Mauriat Miranda’s site for lots of good fc6 information including corefont stuff.

The rpm I built installed the fonts in the wrong place. I had to move them from the /usr/share/fonts/msttcorefonts directory to the /usr/share/fonts/default/TrueType directory. After this, ‘gem install rmagick’ worked like a charm.

Integration Testing Rails

After reading Jamis’s excellent blog about integration testing a ruby on rails app, I decided to take the plunge and write my first integration test. I have never cared much for testing, but with the wonderful testing architecture set up in rails, I have started to actually enjoy it. I now write many tests before I finish coding anything else. I have found it to be very useful and I suspect it has saved me a lot of debugging time. I was kind of forced into this recently because there was no way to test my most recent code with a browser for our iMemoryBook product. The only way I could actually see if what I was writing worked at all was to write a series of functional tests. Today, though, I found I needed to test how two controllers worked together. Hence my need for learning about integration tests.

Duane had written one integration test already so I had a good example to go by, but there was one thing that I did not know how to do. This is why I am writing now, to share what I learned.

Setting the Accept header

Our site is running on edge rails and we are using the new REST interface. The site also uses a lot of Ajax to make requests that only accept XML as a response. Setting the accept header turned out to be very easy…Just add the header information to the third parameter of the request, which is a hash of HTTP headers.

put "/bookshelf/1/sections/5",
   { :lp => 0,
     :pvW => 300,
     "section[title]" => "New title",
     "section[body]" => "new very short body" },
   { "Accept" => "text/xml" }

In this example I invoke the sections_controller’s update method because of the put call and I will receive my response in XML because I set the Accept header to “text/xml”. Now that I understand this, I can write integration tests for the rest of the application.

Killing System Processes in Ruby

So I have recently had the need to create system process and kill them at times. I have never done this in ruby and it turned out to be fairly easy. In fact, like many things in ruby it was much easier than I was trying to make it. The real problem I was having was trying to figure out what the pid of the system process was. Process.fork returned the pid of the child ruby process that was forked, not the system call that was made inside the fork block, or so I thought. It turns out if you fork and then exec, the pid returned from the fork is the pid of the exec’ed process. This is of course how it should work, I just wanted it to be more tricky than that for some reason. Here is the little class I wrote to test it out.

class ExternalProcesses
def initialize
@pids = []
end

def run_command(cmd)
pid = Process.fork do
  exec(cmd)
end
Process.detach(pid)
@pids << pid =" @pids.pop" e =" ExternalProcesses.new">

During the ’sleep 10′ you can verify that this works as expected by opening another terminal and looking at your systems process list.

Backup Plan

So I have recently setup a backup plan for our mysql database. I am pretty happy with how it came out. I use Amazon S3 as the final storage device for all of the backups. A fairly simple ruby script (below) runs mysqldump and then pushes the result out to S3.

I wanted a weeks worth of daily backups, a months worth of weekly backups, and monthly backups for all of eternity. After figuring out how to use the ruby date object the rest was pretty easy. The ruby cookbook held the only decent documentation that I could find on how to actually use the date object. Thanks for the excellent documentation!

I also used some information from a great post on the Mission Data Blog. The current code I had opened a file read it into memory and then transferred that to S3. The mentioned blog post explained how to change Net::HTTP to stream a file.

Here is the interesting part of the code I wrote:

# create AWS connection
conn = S3::AWSAuthConnection.new(AWS_ACCESS_KEY_ID,
                              AWS_SECRET_ACCESS_KEY,
                              USE_SSL)

today = Date.today
system("mysqldump --opt dbname |
       bzip2 -c > /var/backups/dbname-#{today.to_s}.sql.bz2")

#put out today's backup
putfile conn, :bucket => "db-backup",
           :key => "dbname-#{today.to_s}.sql.bz2",
           :file => "/var/backups/dbname-#{today.to_s}.sql.bz2"

#remove old copies
if today.day % 7 == 1
delete conn, :bucket => "db-backup",
            :key => "dbname-#{(today < "db-backup",               :key => "dbname-#{(today - 7).to_s}.sql.bz2"
end

#remove old local copies
FileUtils.rm "/var/backups/dbname-#{(today - 7).to_s}.sql.bz2",
           :force => true

If anyone has any interesting ideas on how to implement a better backup plan I would be very interested in hearing some more ideas.

What the world needs now...

No, not love sweet love, but another blog! After fighting destiny for so many years, the oocha blog is officially started! I will put my first real post up soon. First it is off to the docs to figure out how this crazy contraption works.

paul