Comments on: AJAX Scraping with scRUBYt! - LinkedIn, Google Analytics, Yahoo Suggestions

By: Eatcongress

Eatcongress — Fri, 04 Dec 2009 18:02:48 +0000

Commercial Publication,near public people interested today let limit hard film tree order alright emerge weight painting slip level mention location again good your express later off end liberal attempt district rich nation post anyway complete conference warn recently increasingly first proportion examination case engine neck develop hit supply student driver it daughter above gain walk fit police opportunity institution work imply extent against today balance through or warm ancient annual ancient payment ancient launch call limited push growing worth of interest over once potential tiny percent before payment

By: Will Google stop SERP scrapers by going Ajax? | I Can Has Rankings?

Will Google stop SERP scrapers by going Ajax? | I Can Has Rankings? — Tue, 13 Oct 2009 10:38:34 +0000

[...] a slew of different libraries which mimic browsing behavior and also handle ajax. Off the cuff, ScrubyT handles ajax quite well. Watir can actually open an instance of IE, FF, Safari, or even Chrome, and [...]

By: DCKAP

DCKAP — Fri, 11 Sep 2009 10:49:42 +0000

It really great.. hope it will be implemented well.

By: Martin Chamberlain

Martin Chamberlain — Tue, 08 Sep 2009 13:35:10 +0000

Very impressed with ScRUBYt!

It would be great to have an 'at a glance' overview of advantages over Hpricot, Mechanize, ScrAPI, Watir etc.

By: anya183 likes undies

anya183 likes undies — Tue, 14 Jul 2009 05:45:39 +0000

When was this exactly?

By: Peter

Peter — Wed, 11 Feb 2009 07:47:58 +0000

Jim,

I am not sure you have this in your actual script too, but here you have

clickbypath(’/html/body/p/a’)

(and not click by xpath)

does that help?

If not, please post this to the ML, it’s too messy to answer all over the place (PM, mailing list, lightouse, blog comments

By: Jim

Jim — Mon, 09 Feb 2009 01:19:05 +0000

This is a great example and will be very helpful on my current scraping project. Does anyone have any experience dealing with pop-up windows. I’m running into error and was hoping someone may have already dealt with a similar situation.

I’m getting the following error when I try to select an option box on
a pop-up window.
Error: Unable to locate element, using :name, “ddlslot1″

Here is the code:
Scrubyt.logger = Scrubyt::Logger.new
data = Scrubyt::Extractor.define :agent => :firefox do
fetch ‘http://www.tvg.com’
sleep 1
filltextfield ‘Login1$txtAccountNumber’, ‘142883′
filltextfield ‘Login1$txtPIN’, ‘8240′
selectoption ‘Login1$ddlState’, ‘Massachusetts’
clickbyxpath(’/html/body/form/div[2]/div/div/div/div/div[2]/div/
table/tbody/tr[5]/td[2]/input’)
sleep 5
fetch ‘/Authenticated/program/default.aspx’
# click image to spawn a pop-up window that allows you to change the
tracks selected on current page.
clickbyxpath(’/html/body/form/div[2]/div/div[2]/div[2]/div/div
[2]/div[7]/img’)
#Select Dubai as a track # Error occurs
here where I try to make a selection on the pop-up window
selectoption ‘ddlSlot1′,’Dubai’

once the selection(s) are made push this button to confirm the

selection and return to main page.
clickbypath(’/html/body/p/a’)
results “//body”
It seems like the focus is not on the pop-up so I am not able to make
the selection on the selection box. Any help would be
appreciated.
Thanks,
Jim

By: Caius

Caius — Tue, 23 Dec 2008 11:18:54 +0000

By the looks of it you’re writing in markdown, and have forgotten to escape the underscores in We again used an _and_wait method so its rendered it as andwait. Just need to sprinkle a couple of \’s in there

By: Peter

Peter — Thu, 18 Dec 2008 08:43:46 +0000

Kokpit,

A few thoughts:
- We are working on better documentation all the time - for example the skimr branch (rewriting scRUBYt! from scratch - see http://rubypond.com/articles/2008/12/09/web-spidering-and-data-extraction-with-scrubyt/) is developed with BDD so you have the test descriptions for everything
- scRUBYt! is not Rails or something in that range - you can find out 90% of the functionality relatively quickly by digging around in the source code
- it’s an open source software, so we are happy to accept documentation, suggestions or fixes to make it less intimadating - so why not contribute instead of complaining?
- almost every open source framework could have been ditched with the same reasoning before it got really popular. Fortunately there were enough people to help to reach the tipping point for those projects - this is not true for scRUBYt! yet (though there is a similar amount of bitching, so at least we are even in that area ;-))

Thoughts?

By: Kokpit

Kokpit — Thu, 18 Dec 2008 07:57:18 +0000

This is great, but unfortunately, sCRUBYt is too intimidating and has no documentation. For this reason me and many people I know are still kinda stuck on mechanize and Hpricot.