Class: CraigScrape

Inherits:
Object
  • Object
show all
Defined in:
lib/libcraigscrape.rb,
lib/geo_listings.rb

Overview

A base class encapsulating the various libcraigscrape objects, and providing most of the craigslist interaction methods. Currently, we’re supporting the old Class methods in a legacy-compatibility mode, but these methods are marked for deprecation. Instead, create an instance of the Craigslist object, and use its Public Instance methods. See the README for easy to follow examples.

Defined Under Namespace

Classes: GeoListings, Listings, Posting, Scraper

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(*args) ⇒ CraigScrape

Takes a variable number of site/path specifiers (strings) as an argument. This list gets flattened and passed to CraigScrape::GeoListings.find_sites . See that method’s rdoc for a complete set of rules on what arguments are allowed here.



35
36
37
 
# File 'lib/libcraigscrape.rb', line 35

def initialize(*args)
  @sites_specs = args.flatten
end

Class Method Details

.scrape_full_post(post_url) ⇒ Object

This method is for legacy compatibility and is not recommended for use by new projects. Instead, consider using CraigScrape::Posting.new

Scrapes a single Post Url, and returns a Posting object representing its contents. Mostly here to preserve backwards-compatibility with the older api, CraigScrape::Listings.new “listing_url” does the same thing



156
157
158
 
# File 'lib/libcraigscrape.rb', line 156

def scrape_full_post(post_url)
  CraigScrape::Posting.new post_url
end

.scrape_listing(listing_url) ⇒ Object

This method is for legacy compatibility and is not recommended for use by new projects. Instead, consider using CraigScrape::Listings.new

Scrapes a single listing url and returns a Listings object representing the contents. Mostly here to preserve backwards-compatibility with the older api, CraigScrape::Listings.new “listing_url” does the same thing



124
125
126
 
# File 'lib/libcraigscrape.rb', line 124

def scrape_listing(listing_url)
  CraigScrape::Listings.new listing_url
end

.scrape_posts(listing_url, count) ⇒ Object

This method is for legacy compatibility and is not recommended for use by new projects. Instead, consider using the CraigScrape::each_post method.

Continually scrapes listings, using the supplied url as a starting point, until ‘count’ summaries have been retrieved or no more ‘next page’ links are avialable to be clicked on. Returns an array of PostSummary objects.



165
166
167
168
 
# File 'lib/libcraigscrape.rb', line 165

def scrape_posts(listing_url, count)
  count_so_far = 0
  self.scrape_until(listing_url) {|post| count_so_far+=1; count < count_so_far }
end

.scrape_posts_since(listing_url, newer_then) ⇒ Object

This method is for legacy compatibility and is not recommended for use by new projects. Instead, consider using the CraigScrape::posts_since method.

Continually scrapes listings, until the date newer_then has been reached, or no more ‘next page’ links are avialable to be clicked on. Returns an array of PostSummary objects. Dates are based on the Month/Day ‘datestamps’ reported in the listing summaries. As such, time-based cutoffs are not supported here. The scrape_until method, utilizing the SummaryPost.full_post method could achieve time-based cutoffs, at the expense of retrieving every post in full during enumerations.

Note: The results will not include post summaries having the newer_then date themselves.



179
180
181
 
# File 'lib/libcraigscrape.rb', line 179

def scrape_posts_since(listing_url, newer_then)
  self.scrape_until(listing_url) {|post| post.post_date <= newer_then}
end

.scrape_until(listing_url, &post_condition) ⇒ Object

This method is for legacy compatibility and is not recommended for use by new projects. Instead, consider using the CraigScrape::each_post method.

Continually scrapes listings, using the supplied url as a starting point, until the supplied block returns true or until there’s no more ‘next page’ links available to click on



133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
 
# File 'lib/libcraigscrape.rb', line 133

def scrape_until(listing_url, &post_condition)
  ret = []

  listings = CraigScrape::Listings.new listing_url
  catch "ScrapeBreak" do
    while listings do
      listings.posts.each do |post|
        throw "ScrapeBreak" if post_condition.call(post)
        ret << post
      end

      listings = listings.next_page
    end
  end

  ret
end

Instance Method Details

#each_listing(*fragments) ⇒ Object

Determines all listings which can be construed by combining the sites specified in the object constructor with the provided url-path fragments.

Passes the first page listing of each of these urls to the provided block.



50
51
52
 
# File 'lib/libcraigscrape.rb', line 50

def each_listing(*fragments)
  listing_urls_for(fragments).each{|url| yield Listings.new(url) }
end

#each_page_in_each_listing(*fragments) ⇒ Object

Determines all listings which can be construed by combining the sites specified in the object constructor with the provided url-path fragments.

Passes each page on every listing for the passed URLs to the provided block.



58
59
60
61
62
63
64
65
 
# File 'lib/libcraigscrape.rb', line 58

def each_page_in_each_listing(*fragments)
  each_listing(*fragments) do |listing|
    while listing
      yield listing
      listing = listing.next_page
    end
  end
end

#each_post(*fragments) ⇒ Object

Determines all listings which can be construed by combining the sites specified in the object constructor with the provided url-path fragments.

Passes all posts from each of these urls to the provided block, in the order they’re parsed (for each listing, newest posts are returned first).



80
81
82
 
# File 'lib/libcraigscrape.rb', line 80

def each_post(*fragments)
  each_page_in_each_listing(*fragments){ |l| l.posts.each{|p| yield p} }
end

#listings(*fragments) ⇒ Object

Determines all listings which can be construed by combining the sites specified in the object constructor with the provided url-path fragments.

Returns the first page listing of each of these urls to the provided block.



71
72
73
 
# File 'lib/libcraigscrape.rb', line 71

def listings(*fragments)
  listing_urls_for(fragments).collect{|url| Listings.new url }
end

#posts(*fragments) ⇒ Object

Determines all listings which can be construed by combining the sites specified in the object constructor with the provided url-path fragments.

Returns all posts from each of these urls, in the order they’re parsed (newest posts first).



89
90
91
92
93
 
# File 'lib/libcraigscrape.rb', line 89

def posts(*fragments)
  ret = []
  each_page_in_each_listing(*fragments){ |l| ret += l.posts }
  ret
end

#posts_since(newer_then, *fragments) ⇒ Object

Determines all listings which can be construed by combining the sites specified in the object constructor with the provided url-path fragments.

Returns all posts from each of these urls, which are newer than the provider ‘newer_then’ date. (Returns ‘newest’ posts first).



100
101
102
103
104
105
106
107
108
109
110
 
# File 'lib/libcraigscrape.rb', line 100

def posts_since(newer_then, *fragments)
  ret = []
  fragments.each do |frag|
    each_post(frag) do |p|
      break if p.post_date <= newer_then
      ret << p
    end
  end

  ret
end

#sites ⇒ Object

Returns which sites are included in any operations performed by this object. This is directly ascertained from the initial constructor’s spec-list



41
42
43
44
 
# File 'lib/libcraigscrape.rb', line 41

def sites
  @sites ||= GeoListings.find_sites @sites_specs
  @sites
end