RealityForge.org

Posted by Peter Donald Fri, 02 Dec 2005 04:45:00 GMT

Most of my personal applications require a small database that enforces foreign key constraints and has a reasonably standard SQL interface. Mysql 4.1 + Mysql Front have served me well but recently I have been getting the dreaded Lost connection to MySQL server during query error. I believe it is a problem with the drivers I am using but I am still digging to find the cause of the error.

I have contemplated the switch to Postgres as my database of choice; It feels like a much more mature offering and has far more liberal license . I would not miss out on having a nice graphical front end as there is a number available (PGnJ , pgAdmin , EMS SQL Manager for PostgreSQL).

After the initial period of cognitive I expect I will be happy with Postgres so maybe this mysql driver error is the kick I need to finally make the move.

Posted by Peter Donald Wed, 30 Nov 2005 01:12:00 GMT

Over the last few weeks I have been looking at what is required to deploy rails applications in an environment with a peak load of about 30 requests a second but with an average rate of 1 request per second.

I am running Apache and initially I was hesitant to try fastscgi due to the horror stories that I read about on the rails mailing list. I initially tried mod_ruby but it was not speedy and it consumed large chunks of memory. I assume this was due to to the ruby interpreter being added to each process.

Next I migrated to fastcgi which was not anywhere near as painful as I had heard. I followed the directions in Apache tuning for Rails and FastCGI and I had it up and running in no time. Very occasionally I am seeing fastcgi processes that are left alive after a web server reload. Apparently Lighttpd does not have this problem but I need to stick with Apache2 because of applications running on the server.

I want to try an approach that I read about on the rails mailing list that involves proxying requests to Lighttpd from Apache2 using configuration like the following. This would then allow me to run a separate Lighttpd on to handle the rail requests.

<VirtualHost *:80>
    ServerName napts.realityforge.org
    ProxyPass / http://napts.realityforge.org:8080/
    ProxyPassReverse / http://napts.realityforge.org:8080/
</VirtualHost>

I also noticed an easy way to host multiple rails applications on one hostname. Previously I had created a new hostname for each rails application ala napts.realityforge.org, iplan.realityforge.org, etc.

<VirtualHost *:80>
    ServerName www.realityforge.org
    ProxyPass /napts/ http://www.realityforge.org:8080/
    ProxyPassReverse /napts/ http://www.realityforge.org:8080/
</VirtualHost>

And then in my rails application I need to add the following to my environment.rb

ActionController::AbstractRequest.relative_url_root = "/napts"

Pretty neat! and it clears up many of my remaining issues with rails.

Posted by Peter Donald Wed, 23 Nov 2005 00:07:00 GMT

I have always considered data modeling as the one of the most critical aspects of software development. A well designed data model can outlive the specific software product it was designed for and provide a valuable asset to the organization. In the ideal scenario the data model will evolve and adapt with the organization as requirements change.

However, as a software developer I have rarely worked in an environment where I needed a deep understanding of any particular vendors database implementation. Recently I have been developing database-centric software on a number of different platforms. It may be that I am missing something but I have yet to find a decent database centric scripting language.

Consider the following problem that I was tackling a month ago. We have a central database server and application server to access the database. Database writes MUST occur through the application server to maintain data integrity due to limitations at our software layer. We also support distributed servers that can periodically synchronize with the central server. The synchronization process requires heavy use of buisness logic to detect and resolve conflicts in synchronized data. Some data may also come from other external systems such as personal or payroll. This data needs to be cleaned and converted into our schema before being synchronized with the central server.

I was tasked with automating the synchronization from an external system to our central server. I also needed to have a test run through the synchronization prior to the real run to stop the process if synchronization would fail. This involved the following steps

1. Import, convert and clean the data from external database into INCOMING database
2. Backup INCOMING database
3. Backup CENTRAL database
4. Restore CENTRAL database into TEST database
5. Run synchronization between INCOMING and TEST. This involves;
   1. Startup TEST application
   2. Start INCOMING application and initiate synchronization with TEST
   3. Shutdow TEST application
6. If synchronization in previous step was successful then synchronize between INCOMING and CENTRAL servers. This involves;
   1. Start INCOMING server and initiate synchronization with CENTRAL
7. Backup CENTRAL database

At each step along the way we need to log information about progress into another database as the process can take several hours. If an error occurs we need to inform appropriate party.

I ended up implementing this as a stored procedure in Microsoft SQL Server. This is not without it’s problems. For starters it is tied to a specific vendors database server (and possibly a specific database server version). Secondly there is a large number of ugly code hacks. To execute external processes in SQL Server you need to create a job with the command then start the job. Then I poll a system table every 10 seconds until the job has completed using GOTO’s .

If that was not bad enough, I needed to come up with a mechanism to log progress messages to a different database. My problem was that if an error occured during a number of the steps a transaction roll back was issued which reverted all the log messages. The only way I could find to get around this was to open another connection to SQL Server using the SQL-DMO COM object. The COM object only used to write log entries and as it was a different connection it would not be rolled back when the main transaction rolled back. ugly!!

• an xml document defining transformation rules
• auxilliary SQL scripts to support non-standard rules
• a look-up-table in another database

• another xml document defining consistancy rules
• custom java code to support non-standard rules

It is not a pretty sight.

Admittedly if the system was to be rewritten from scratch the whole process would probably be a lot cleaner. But even then, I was skeptical that there was a nice way to implement this. The software would need to be able to define a domain model with rules that contain both imperative/procedural (ie java or some other imperative language) and declarative elements (ie sql and some data constraint language).

Previously I had thought that the best path to tackle this problem was to use some sort of Domain Specific Language to define the declarative aspects of the data model and then define the procedural elements using a language like Java. I have used this approach with success before. I defined the static model characteristerics and data constraints in an XML document and then used Velocity to generate the java code that was enhanced with procedural elements.

Recently I have been playing with Ruby on Rails and I have been re-evaluating my position. Rails has the ActiveRecord library that allows you to define model classes (using the Active Record pattern as described by Martin Fowler). These model classes can define validations that offer a psuedo-constraint language for the data. It also offers support for defining associations, and aggregations between different active record elements and is generally a nice and easy toolkit to use to access relational data. If you need to escape to SQL for performance or conceptual reasons then that is possible with few hassles.

Even more recently I discovered migrations in rails that make it possible to incrementally modify your database schema as your application evolves. You can add or remove columns, tables, indexes etc all the while preserving and migrating data as per application requirements. To upgrade to the latest schema you need only run the “migrate” rake task and be done with it.

This makes rails or more specifically ActiveRecord a very strong contender for my toolkit of choice to script database It would make it possible to avoid vendor specific stored procedures or SQL, to a certain degree and make it much easier to develop software to manipulate schemas and data.

The only negative is that it is in ruby and I have a greater understanding of the java language. Then again maybe ruby does not require the breadth of understanding java does – ir is much simpler to just get stuff done.

Maybe ruby is the next java.

Posted by Peter Donald Sat, 12 Nov 2005 13:49:00 GMT

I did not realize that my blog was subscribed over at JavaBlogs.com so when I wrote my recent entry on AAA in rails I was surprised to get a comment like this one and despite the tone he did have a point. I should not have it subscribed at java blogs. The version of Typo I am using did not seem to support category based rss feeds. So I did a quick google for answers and found that others had added support this and it had even landed on the issue tracker . So I had a quick browse and decided it was easier to write myself and added a route and a quick hack to app/controllers/xml_controller.rb from

 def rss
    @articles = Article.find(:all, :conditions => 'published=1', 
 :order => 'created_at DESC', 
                             :limit => config[:limit_rss_display])
  end

to

 def rss
    joins = nil
    conditions = 'published=1'
    if params[:category]
    conditions = ['published = 1 AND categories.name = ? AND 
                    articles_categories.article_id = articles.id AND 
                     articles_categories.category_id = categories.id', 
                    params[:category] ]
      joins = ', categories, articles_categories'
    end
    @articles = Article.find(:all, 
                             :select => 'articles.*',
                             :conditions => conditions, 
                             :joins => joins, 
               :order => 'created_at DESC', 
                             :limit => config[:limit_rss_display])
  end

And it took me less than 15 minutes which includes time to go and make a coffee. Now that is why I have come to like rails. I may have been able to do the same in another framework but chances are I would have had to guess the table names, file locations, foreign keys etc. With rails I don’t really have to think if the app follows the conventions.

Posted by Peter Donald Fri, 11 Nov 2005 20:23:00 GMT

It is interesting to watch how the “login” discussion on the Rails Mailing List evolves. There is an abundance of choice for developers who want to integrate login functionality into a rails application. A sample selection;

• LoginGenerator
• SaltedHashLoginGenerator
• Auth Generator
• Model Security
• LoginEngine
• ActiveRBAC
• LoginGeneratorACLSystem
• AccessControlList
• Extensible Authorization for Rails

My current project requires “login” functionality so I thought I would have a look at some of these projects to see what they offered and to see how easy it would be to integrate one of the existing software products. To my dismay I found that all of them seemed to try and do too much of one thing or not enough of another. Now I think I am begining to understand why rails does not consider these components worthy of core or not evil but distracting .

All of the systems seem to mix different concerns in the one system. What I would like to see is the ability to separate out authentication from authorization. In the software I am currently developing users do not register for an account but must be explicitly added and passwords must not be stored in clear text. Next month I will be helping to develop a system where each user authentication attempt is delegated to an external system, if the user authenticates then a user account is created if it does not exist. If all goes well I will be developing another system about March next year that uses HTTP authentication.

However in each of these applications there will need to be a very similar concept of authorization. Each application will have a fixed set of parameterizable permissions, roles/groups that are granted these permissions and linkage between users and roles/groups. All of the solutions that I looked at seem to lump too much functionality together. If only they had separated out the pieces into smaller reusable chunks then I would be jumping at the chance to reuse them. As it stands it looks like I am going to end up reinventing the wheel as it is just less work.