W3C Mozilla DOM Connector was released today

I have announced the upcoming release of the W3C Mozilla DOM Connector in one if my previous posts, and now it has finally arrived. You can view it at

http://svn.rubyrailways.com/W3CConnector/

or check it out with svn:

svn co http://svn.rubyrailways.com/W3CConnector/

For a description about the connector, please refer to my previous post. If you would like to try it out, here is how:

#this code snippet gives you a DOM document of the currently loaded page:
  nsIWebBrowser brow = getWebBrowser();
  nsIWebNavigation nav =
      (nsIWebNavigation)
      brow.queryInterface(nsIWebNavigation.NS_IWEBNAVIGATION_IID);
  nsIDOMDocument doc = (nsIDOMDocument) nav.getDocument();
  Document mozDoc = (Document)
org.mozilla.dom.NodeFactory.getNodeInstance(doc);

From now on, you can use all the existing java/dom libraries such as an XPath2 engine like saxon, xalan, whatever you want working on mozilla documents.
This means tremendous power compared to (in their category outstanding, but still limited) tools like RubyfulSoup or Mechanize, stemming from the power of XPath to query XML documents.
A simple example – dumping DOM of the html document to stdout:

public static void writeDOM(Node n)
      throws IOException
  {
      try {
          StreamResult sr = new StreamResult(System.out);
          TransformerFactory trf = TransformerFactory.newInstance();
          Transformer tr = trf.newTransformer();
          tr.setOutputProperty(OutputKeys.ENCODING, "UTF-8");
          tr.transform(new DOMSource(n), sr);
      }
      catch (TransformerException e) {
          throw new IOException();
      }
   }

Cool, isn’t it?
At the moment, I am discussing different integration issues with the Mozilla guys, since the connector should be the part of Mozilla and the Eclipse editor in the future.

Ruby on Rails Migrations Reloaded

In my previous post on migrations i wrote that “…they are not covered in any of the basic books on RoR”. Well, this statement does not hold anymore, since Agile Web Development with Rails, 2nd ed. is already creating the models with migrations.

While the last part of the post (why are migrations so cool) is still up-to-date, they way of creating migrations is different from 1.1 on, so i have decided to review the topic and add some new points, too.

Migrations are now created automatically with the model

In my previous post, i have been creating the migration manually with the command

ruby script/generate migration ProductMigration

However, as of Rails 1.1, you don’t have to do this anymore. When you generate the model (let’s stick with the Product model as an example) the migration is automatically generated:

ruby script/generate model Product
...
... #some lines omited
...
create db/migrate/001_create_products.rb

Now you can edit the file db/migrate/001_create_products.rb to contain something like this:

class ProductMigration < ActiveRecord::Migration
def self.up
create_table :products do |table|
table.column :title,       :string
table.column :description, :text
table.column :image_url,   :string end
(rest of the file omited)

Then run

rake db:migrate

To update the database. That's even easier than in the previous versions of rails!

Valid column data types and possible options

Valid columns are:

integer, float, datetime, date, timestamp, time, text, string, binary and boolean.

Valid column options:

  • limit ( :limit => “50” )
  • default (:default => “blah” )
  • null (:null => false implies NOT NULL)

string is the equivalent of varchar(255), so if you would like to have a string column (called title) of length 100 instead of 255, with default value 'Some title' and to forbid NULL value, you have to type

table.column :title,
:string,
:limit   => 100,
:default => "Some title",
:null    => false

Generating test data

I am quite sure you know the situation when you want to test something quickly and you waste precious time to generate some test data, which you trash after the testing just to find yourself in the same situation later?

Well, migrations can help you to prevent headaches because of this, too. Here is how:

ruby script/generate migration create_test_data
Create db/migrate/002_create_test_data.rb

You can create test data inside the migration file like this:

class ProductMigration < ActiveRecord::Migration
def self.up
Product.create(:title => 'My cool book about the meaning of life',
:description => '42',
:image_url => /images/cool_book42.png)

You can now commit this migration to the RCS you are using, and modify/add more test data later.

The other advantage is that your colleagues won't spend time writing dummy test data either: they can just check out this migration and happily use the provided tests.

If this is still not enough for you...

You can write SQL statements inside the migrations. For example:

execute "alter table items
add constraint fk_items_products
foreign key (product_id) references products(id)"

However, use this with care since you have to write native DDL statements, which violates one of the fundamental 'cool factors' of migrations: independence from DB vendors.

Conclusion

The Agile Web Development with Rails, 2nd ed can be considered as the Rails bible and since it is promoting migrations as the definitive way to handle your DB issues, i think migrations will become (in fact the already did for lots of people) the state of the art. After using them for a while and enjoying the power and flexibility they provide without having significant drawbacks, i don't really see why should one not use them in the future.

WWW::Mechanize problem (probably Ubuntu only(?))

I am working on a small screen-scraping utility written in Ruby, and since I have been working previously with RubyfulSoup, I wanted to give WWW::Mechanize a try this time.

So i have installed the WWW::mechanize gem:

sudo gem install mechanize

I wanted to try a ‘Hello world’ application first, to see wheter it works. Here are some official examples (click on ‘Examples’). I Copy&pasted the first one, run and got the following error:

/usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:21:in
`require__': no such file to load -- net/https (LoadError)
from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:21:in `require'
from /usr/lib/ruby/gems/1.8/gems/mechanize-0.4.4/lib/mechanize.rb:15
...
...

After some googling i have found the answer: I had to install libopenssl-ruby, and the error was gone.

I wonder if you have to install this amount of additional packages on other distributions also? From the time I am using Ruby/Rails I did not have other distro than Ubuntu, but back in my Python days I have been running on gentoo and I don’t remember such problems. Ubuntu is really very cool, but it seems you have to know well which packages do you need and install them manually when it comes to coding/development…

Google Trends: Googlefight v2.0 and much more!

Every second blog I came across recently has an entry about google trends, so I am adding my small findings too! 😉

After playing with it for a few hours, I have to say that writing a relevant query is not always as easy as it seems. People are posting Java vs Python vs Ruby comparisons, but they are not always aware that the graph contains (among other things) the comparison of an island, a comedy troupe (Monty Python) and a character set (Ruby Characters), for example. According to wikipedia, all three terms have more than ten possible meanings, and although a tech nerd may know only one for each of them, not all pages out there are (fortunately) written by tech guys.

Let’s start with some Rails related stuff:

Well, I wonder who else recently (not even necessarily in the computer industry) got so famous in a matter of days…
It is interesting that there is no data available for “David Heinemeier Hansson” or even “David Hansson”, just for DHH.

The next graph could answer the question whether it is a good idea for a web hosting company today to support Ruby on Rails:


For the idea of the following googleTrendFight thanks for Laszlo on Rails blog.

It’s really thrilling to see that a (once) small open source community can compete with enterprise stuff of such magnitude as JBoss/EJB (ok, this is kind of apples-to-oranges, but nevertheless interesting). If you wonder why did JBoss’ search volume go dramatically up – it’s because RedHat bought the company.

Non-Rails related:
slashdot.com vs digg.com vs reddit.com:

No comment…

The last one about wikipedia, kind of funny:

Why should be this funy? Because the only point in the history (so far) when search volume for wikipedia was declining was because of:

Probably (hopefully?!?!) there is no direct link between these facts, but it is an interesting random coincidence then…

I wonder whether google will improve the quality of this search and/or add possibility to specify advanced queries to prevent mixing in of irrelevant results – at the moment, if I did try to narrow the search, in lot of cases i got back ‘data not available’… Interesting toy, though.

Getting Ruby on Rails up and running on Ubuntu Dapper

I have just installed Ubuntu Dapper Drake Flight 6 on my desktop machine, and because I had had different problems to install Rails from scratch several times (even the recent session was no exception), I have decided to write a step-by-step guide, which assumes a clean, fresh install of Ubuntu ( i.e. at this point you do not even have Ruby on your machine) and leads you through installing Rails and creating a working test application.
Why is this writeup better than any other how-to-install-rails tutorials out there?

  • Because it will tell you to install really just what you need, not 50 packages more
  • It will also show you how to configure the DB and other things to really make Rails work, not just installed

Let’s get started!
Note: Some people asked if this manual is for dapper only. I would say mostly yes, because i have had different problems on breezy (for example i had to compile ruby-mysql driver manually). Its not entirely impossible that it will work with breezy – but then you will have to make sure that the packages are the same version as assumed here (e.g. MySQL > 5 etc.)

Part I: Installation

Prepare the system for the installation

  • Check /etc/apt/sources.list – make sure you have access to the ‘universe’ packages by uncommenting them:
deb http://us.archive.ubuntu.com/ubuntu dapper universe
deb-src http://us.archive.ubuntu.com/ubuntu dapper universe
  • Refresh apt packages to make sure you get the most up-to-date stuff:
sudo apt-get update

Install Ruby related packages

  • Install Ruby essentials: ruby, irb, rdoc, ri
sudo apt-get install ruby rdoc ri
  • Install gems: download, unpack, install
go to http://docs.rubygems.org/
download rubygems-0.8.11.tgz (or the latest version)  tar -xzvf rubygems-0.8.11.tgz
cd rubygems-0.8.11/
sudo ruby setup.rb

MySQL installation and configuration

  • Install MySQL:
sudo apt-get install mysql-server
  • Install ruby MySQL bindings
sudo apt-get install libmysql-ruby

Install Rails

sudo gem install rails --include-dependencies

Part II: Configuration

Setup the DB

  • Add an user, create a test database and grant acces for the user
mysqladmin -u root create test_development
mysql -u root

Into the db shell, write the following commands:

create user 'batman'@'localhost' identified by 'robin';
grant all on test_development.* to 'batman'@'localhost';

Don’t forget to replace the username/password (unless you happen to be Batman of course – in this case i suggest to use a different password since this can be guessed easily by social engineers 😉

Create and test the rails app

  • generate the app files

Lets denote your working directory (the root directory where your future rails project s will reside rails_projects).

cd rails_projects
rails test
  • edit config/database.yml
cd rails_projects/test 
vim config/database.yml
  • It should look like this:
development:
adapter: mysql
database: test_development
username: batman
password: robin
host: localhost
  • generate a dummy model
ruby script/generate model Dummy
  • edit the migration file
vim db/migrate/001_create_dummies.rb
class CreateDummies < ActiveRecord::Migration
def   self.up
  create_table :dummies do |t|
    t.column :foo,    :string
    t.column :bar,    :string
  end
end

def self.down
  drop_table :dummies
end
end
  • run the migration
rake db:migrate
  • generate a simple maintenance app
ruby script/generate scaffold Dummy Admin
  • start the server
ruby script/server

Point your browser to http://localhost:3000/admin to see the result.
If you have any problems, please leave a comment, i will try to help you.

Internet contains huge number of opportunities to earn money online. Simply create a site that you think has the potential to sell hot items using ruby on rails. Register a relevant domain name and purchase a web hosting service through hostgator, one of the better web host out there today. Get a internet connection through one of the wireless internet providers to upload your site. Work on search engine optimization to get a better traffic and also use affiliate marketing program for the same reason. Finally get a free voip phone service to contact customers directly. The pc to phone system is the most effective method of marketing.

W3C Mozilla DOM Connector will be released soon!

I am happy to announce that the much anticipated W3C Connector, after lots of coding, testing, bug fixing and several months of successful usage in a commercial product was proven worthy to be released to the public. If everything goes well, it will hit the streets next week.

OK, but what the heck is the W3C Connector?

The W3C Connector is a Java package which can be used to access the Mozilla DOM tree from Java, while implementing the standard org.w3c.* interfaces. This means you can use it with any standard Java package that is expecting org.w3c.* interfaces ( Xerces, Saxon, Jaxen, … ) to execute effective queries on the Mozilla DOM (XML/XSLT/XPath/XQuery operations for example).

Technically,the W3C Connector is an implementation of the standard org.w3c.* interfaces. The implementing classes are calling Javier Pedemonte’s JavaXPCOM package, which in turn wraps the Mozilla XPCOM in order to gain access to the Mozilla DOM. See the image for an illustration:

This is very nice and all, but why should I care about it?

If you ever wanted to do (or have done) a screen scraping application, where you needed to understand the underlying document to some extent (regular expressions were not sufficient) you should know that there are many pitfalls along the way:

  • First of all, malformed HTML code: Despite the continuous efforts of the W3C and other organizations/individuals to remedy this problem by promoting X(HT)ML and other machine parsable formats, a lots of web pages still have malformed code in them. Based on the level of non-standardness, parsing such a page can be more than a moderate technical problem: in practice there are pages which can not be parsed to produce an usable input.
  • You can not use a standard query language like XPath or XQuery – these languages require a XML input, which you can not ensure because of the previous point, so you are left to roll your own code to process the parsed data.

Of course this is not a big problem for a crafted programmer, mainly if he is equipped with tools like HTMLTidy to address the first point, RubyfulSoup or similar to tackle the second. However, even these (and other) tools and a cool programming language are still just easing up the pain of effective screen scraping, but not offering a generic solution. If you want to scrap a lot and different pages, these problems in practice will cripple your efforts (or at least make it last very long time in practice).

How does the W3C Connector solve this problem?

By solving both points: The Mozilla DOM is a structure reflecting how gecko (the mozilla rendering engine) renders the page, and it always translates to valid XML (no unclosed tags or otherwise malformed code), and because of implementing the org.w3c.* interfaces you can use very robust and effective XPath packages (like Saxon) to query the document for effective HTML extraction.

There are of course a lot of other possible uses – the connector is not a tool itself, but a gateway to the world of W3C compliant XML tools – it is up to you how to leverage the power it gives you.

The Big Brother

The W3C Connector will be released officially as the part of theATF project. The code is under the last review at the moment, it is possible that I will come out with a preview release before the official one.

Round corners and nifty cubes

I am in the process of redesigning rubyrailways.com, so you can see every kind of weird experiments sometimes (I am too lazy to do the whole thing offline, because that would mean to set up Apache, PHP, WordPress, MySQL … etc, and the other reason is: I have too limited time to do it quick).

As you can see, currently I am experimenting with one of the most widespread cliché of today’s webdesign: round corners. There are infinite possibilities to round your corners – as my primary focus is not web design, i am not really an expert on the topic, but i have seen a lot of methods (various ratio of (no) images, JS and CSS). For example, a Firefox friendly quick’n’dirty solution:

Simple, but limited

(no images and JS needed, but has severe cross-browser restrictions – if you are reading this from IE (or probably anything other than Firefox) you know what i mean).

Browsing through the possible solutions, i have chosen Nifty cube. It is an image-less solution, all you need is to add 1 line of Javascript and a few lines of CSS code to make it work. It has a lot of options (this i already the second version, which is a substantial overhaul compared to the first one),

and for me it worked nicely. (Have to work on the actual usage, though – The rounded div’s around the title are too big ATM, but this is not the problem of Nifty cube)

db:session:create problem (?) in the depot app

As I wrote in my previous post, I am currently reading/coding the depot example from the Agile Web Development with Rails book. I had one unclear issue so far – maybe someone can help me to find it out: I did something wrong or this is really a typo in the book?

Putting sessions in the database
According to the book, this should be done with

rake db:session:create

However, after entering this command to the console I got:

(in /home/peter/development/src/railsprojects/depot)
rake aborted!
Don't know how to build task 'db:session:create'

After some playing with rake –help, I have found the –tasks switch:

rake --tasks

and here I have finally found the remedy for my problem: the correct command is

rake db:sessions:create

(note the additional ‘s’ character)
Is this because my rake version is too old/too new or this is a typo in the book?

After this modification everything worked again as intended.

Fresh Meat – First Impressions of Agile Web Development with Rails (2nd ed) and Ruby for Rails

I have bought both books recently. Maybe it is a little bit early to write a review since I am through just a few chapters in both, I have been so impressed that I had to write a (not so) short entry at least ;-)!
I have been using C++/Java/Python for years, and have been reading as much books on every possible aspect of development with these languages as I could get. In my oppinion Ruby and Rails are absolutely superior to all of these languages (and their web frameworks/related things) in this context – The Pickaxe, AWDwR and R4R cover nearly all the things you will ever need to learn the language and Rails – and not just understand the ‘how’, but also the ‘why’, learn best practices, coding and development methodologies from code style to design issues, related technologies and more.

The point is not (only) this, since you can do the same with a few Java books (although not 3, but something like 10), however you will have harder time with Python (There is a Django book on the way, and also the Turbogears guys are publishing one but neither are out yet – but no PickAxe, AFAIK) – however, the point is, based on the experiences i have had with C++/Java/Python books that neither of them are so well written/to the point/effectively explained/well built up than the Ruby ones. Simply put: The Ruby/Rails books are the best technical books i have ever read on programming and (web) development.

AWDwR 2nd ed Agile Web Development with Rails, Second Edition
I begun to develop the depot application with the first edition about a month ago, and since I was new to both Ruby and Rails, I have thought: WOW! I was about halfway through the depot chapter when i have noticed the announcement about the Second Edition a few days ago. I have purchased the pdf version immediately, and I am quite sure this purchase has to be listed in my ‘Best value for money TOP 10’ list (along with R4R ;-).

It was a very refreshing experience to code the depot application from the first edition, but as I got into Rails more and more, I have felt that there are some small gaps here and there. On the mailing list, everybody was talking about migrations (i had no clue that time about them), helpers, AJAX/RJS, REST and other stuff i was not able to find during the depot development, and though the amount of information and level of coolness was overwhelming, I still sensed there is still even more than this.

After redoing the depot application with the second edition, all these things (among others) are finally there! You get all the goodies from the very beginning (migrations, writing your own helpers, AJAX etc) so you do not have to search the Web for the newest features anymore.My overall impression was that the small annoying things are gone, the good things are even better, thus the overall experience of reading the book/following the code is even more delightful! I can’t wait for the next chapters! This upgrade definitely rocks! Well done Dave et al.

Ruby for Rails Ruby techniques for Rails developers
I have gone through just the firs three chapters so far, and though they are supposed to be introductory chapters (entitled How Ruby works, How Rails works and Ruby-informed Rails development) I could not believe how much info I got out of them. I have to say that I am a totally mega-n00b to both Ruby and Rails, but still, I have gone through the PickAxe and half of AWDwR, 1st ed, i am a regular reader of Ruby-talk and the RoR mailing lists, so on the other hand I have some basics, and still these chapters provided me a lot of new insight.

The book (or at least the first three chapters) is extremely well written, easy to grasp yet the breadth of knowledge is really impressive. It really shows the design philosophy, logics, inner working of the things rather than just providing some theory with a few examples, or being a ‘yet another Ruby/Rails book’ in any way. If you would like to find out how the things really work, and why they work that way, rather than just be a developer who can do this and that with Rails, definitely check this book out!

R4R

There was a kind of flamewar on the Ruby on Rails mailing list about the pricing of the AWDwR second edition: One group argued that they should get some discount because they own the first edition, and the other party did not agree with this standpoint. Well, personally I am definitely in the second group – I did not hesitate to buy the PDF for a moment – I can understand (but not support in any way) the arguments of the first group, but I would gladly pay for this book even $100, not $20+, regardless of the editions I own. (And just FYI, I am a full time Java developer and though I would like to get a Ruby (on Rails) job ASAP, due to different constraints RoR is and will be just my hobby for some time)

Just my 2c.

Ruby on Rails Migrations

Although migrations are a very cool feature of Ruby on Rails, they are not covered in any of the basic books on RoR i have encountered so far (Agile Web Development with Rails, Ruby for Rails Programmers).

Update: Check out my recent post: Ruby on Rails Migrations: Reloaded for an update.

Both these books are using an ‘in medias res’ style approach – they guide the reader through the essential features of Rails by building a web app from scratch. The models in the examples are creaed in SQL rather than with migrations. Let’s examine the difference on a simple example, taken from AWDwR. (Further I am assuming that you have generated a rails application, a development database for the application and the DB connection settings (database.yaml) are correct.)

The classic way: SQL DDL

Create the sql file, create.sql:

drop table if exists products;
create table products (
id           int            not null auto_increment,
title        varchar(100)   not null,
description  text           not null,
image_url    varchar(200)   not null,
price        decimal(10,2)  not null,
primary key (id)
);

After this, you can create the table with:

mysql name_of_your_DB < create.sql

You are now ready to generate your model.

Doing the same with migrations

In your rails app directory, issue the following command:

ruby script/generate migration ProductMigration

then open the file db/migrate/001_product_migration.rb and edit it. To achieve the same result as in the SQL example, the file should look like this:

class ProductMigration < ActiveRecord::Migration
def self.up create_table :products do |table|
table.column :title,
:string,
:limit => 100,
:null => false

table.column :description,
:text,
:null => false

table.column :image_url,
:string,
:limit => 200,
:null => false

table.column :price,
:float,
:null => false
end
end

def self.down
drop_table :products
end
end

Run the migration wit the following command:

rake migrate [VERSION=version_number]

And you achieved the same result as with the first method!

That's very nice, but...

Well, if the only purpose of migrations would be solely the possibility to write Ruby code instead of SQL, even this would be enough for me to go for them. However, i have to admit that this alone would be a rather feeble argument. The good news is that it is not! There is much more to migrations than writing Ruby code:

  • Migrations are DB agnostic - The 'write once, use everywhere' principle really works here!
  • You don't have to think about obscure SQL specific things anymore - let Rails handle them for you! (OK there are some really complicated things, but fortunately they are adressed by some great books like Rails Recipes, code snippets like Migrate Plus, and I believe that by the Rails team, too.)
  • You can change the database as much as you want, and the data you have already there is not affected.
  • You get very effective versioning: track changes, concurrent versions, upgrade/downgrade your schemas easily!
  • You can generate DB schemas from migrations.
  • And possibly much much more... I am a newbie too! 😉

In my oppinion, judging based on the Rails mailing list discussions, migrations are accepted more and more as the definitve way of creating, maintaining, versioning your DB models - so everybody considering serious Rails development should give them a look!