はるかなる熊

久々にRubyでクローラー作成。
クローラー先は内緒で。
素敵にライブラリがあるので一瞬でデータを取ってこれちゃいます。

まずはgemでscrapiをインストール

gem install scrapi

rubyファイルを作成。Scrape用のクラスを作成してみます。

scrape_site.rb

require File.expand_path(File.dirname(__FILE__) + "/../config/environment")
require "rubygems"
require "open-uri"
require "logger"
require "scrapi"
require "jcode"
require "uri"

class ScrapeSite

  SITE_URL = "http://**********"

  def scrape
    ###################################################################
    # サイトURL生成
    ###################################################################
    url = SITE_URL
    url = NKF::nkf('-s -m0', url)
    url = URI.escape(url)
    ###################################################################
    # スクレイピング
    ###################################################################
    html = NKF::nkf('-w -m0', open(url).read)
    places = Scraper.define do
      process "table.tbl td.price", "prices[]" => :text
      process "table.tbl td.detail b", "details[]" => :text
      process "table.tbl td.image a img", "images[]" => "@src"
      process "table.tbl span.title", "titles[]" => :text
      result :prices, :titles, :details, :images
    end
    values = places.scrape(html, :parser_options => {:char_encoding=>'utf8'})
  end
end

st = Scrape.new
st.scrape

Tags: ruby

This entry was posted on Wednesday, April 22nd, 2009 at 11:05 pm and is filed under ruby. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.