Monthly Archives: May 2013

Scraping Amazon item offers

In my pet project Dealesque, I am trying to compare all offers on a number of Amazon items, the idea being that it can help decide which offers to use to minimize shipping and total cost. Using Amazon Product Advertising API was the logical first step, but it doesn’t return all the offers for an item. It does however return the “more offers URL” for each item. Hence, the old scrapin’ was due, and none too late!

Plain wget-like action would not suffice, since Amazon is taking care to block unwanted traffic. So, mechanize gem to the rescue! It actually allows you to impersonate a real browser:

agent = Mechanize.new { |agent| agent.user_agent_alias = 'Mac Safari' }

After that, you can navigate the site, click away, read any forms etc.
For scraping, what I actually ended up using was to get the content of the “more offers URL” page and parse it using Nokogiri. Something like:

page = agent.get(more_offers_url)
root = Nokogiri::HTML(page.content.strip)
scrape_content(root)

For the current development stage, this is doing just fine. Unfortunately, for production use it will not suffice. There will probably be some traffic throttling from Amazon and some benchmarking will need to be done to determine the limits. Also, proxying the requests will probably be required too. But, I leave this for some other times.

The result of scraping the offers for picked items:

Dealesque picked items

Advertisements
Tagged , , , ,

Mixin Less & Sass with twitter-bootstrap-rails

When working with a Rails app, if you wan’t to use Twitter Bootstrap as the base of your design, there are plenty of options out there. From manual import to a more than a fair list of existing gems. Probably the best place to start is the “Twitter Bootstrap BasicsRailsCast by Ryan Bates.I decided on twitter-bootstrap-rails gem by Seyhun Akyürek mostly because  it maintains a fresh version of Bootstrap. The usual setup and usage are covered in the mentioned casts and related material, but the gem uses Less instead of Sass. I really wanted to work with Sass, so the question was how to combine the two?

The answer actually ended up being pretty simple. Let’s say the gem has generated a style-sheet Less file named “bootstrap_and_overrides.css.less“. If you want to use the classes and mixins from it, all you have to do is to place the following at the top of your Sass file:

  @import "bootstrap_and_overrides";

Now, all the definitions from Less part of the story are available in your Sass too. An added bonus is that now you can separate your HTML from the style completely and just add Twitter related classes in your Sass style-sheet. For example, here is how you can set the image container div width to “span2” Twitter class.

The Haml source:

.item
  .image_container
    .image
    ....

Sass style-sheet:

@import "bootstrap_and_overrides";
.item {
  .image_container {
    @extend .span2;
  }
}

The resulting html is:

<div class="item">
  <div class="image_container">
    <div class="image">...

The end result is that the “span2” will actually be applied to the image container because the “image_container” css class is acutally extended with “span2” mixin.

A lot like Ruby, go figure!

Haven’t decided whether to keep using this style or not, just playing with it for now, but it is good to know you can actually mix apples & oranges 🙂

Tagged , , , ,