Ever wanted to get a list of information such as URLs, Articles, tabular data, or whatever else that you know is on one website or across multiple websites, then manipulate it to reuse elsewhere? Stop wondering, because we are about to get down to business!
There’re many ways to scrape / mine data, but I’ve found that the easiest and most efficient way is to use a combination of cURL and XPATH. cURL is neat because it will easily let you use proxies, manipulate browser information, catch errors, etc. XPATH is great because you don’t need to write a bunch of regular expressions or other functions to manage the data – you just manipulate the DOM tree with a single string and you can get the group of elements that you want in an array. Both of these modules are available in nearly every programming language, so your code, if written correctly, can easily be ported from one language to another.
What We’ll Touch On
Before We Begin
Our Goal
I called this post “advanced data scraping” because we are going to be doing more than just getting a small bit of information. We are going to be crawling multiple pages, gathering URLs, and then parsing those URLs and scraping data on those pages. The techniques taught here are used by many programmers for a variety of reasons, and will even let you scrape pages rendered entirely in Javascript (such as Google)!
So, we’ll start with a generic site meeting that criteria, such as EzineArticles.
If you don’t have Firebug yet, you need it. It’s amazing for quickly checking the elements of a page, especially when setting up XPATH queries. You’ll also want to grab Xpath Checker to quickly see if your XPATH queries are selecting the correct elements. Both are FireFox extensions.
Pre Coding
Before we start coding, we want to gather our XPATH queries to make sure we can select the data we want and get a feel for how we’re going to program the scraper.
First, we need to figure out what section of EzineArticles we’re going to scrape. I think Business >> Ethics is a suitable section. We have two options here, only scrape the RSS feed, or scrape the pages directly. For this tutorial, we are scraping the pages directly so that we can get all the articles, not just the 100 or so most recent. It also lets you see how you would scrape sites with pagination (a section broken up into multiple sections of 10, 20, 30, etc.).
Category Section Page
EzineArticles Article Snippits DOM
In order to see how you want to go about selecting sections of the DOM with XPATH, you need to look at the code. Firebug is great for this.
On the section page, the XPATH query to get the information we want is this:
//div[@class='ea-category-list']/ol/li
That query will get an array of every list element within a div that contains the “ea-category-list” class. The list items contain a link to the article, a short description, and the author; so, the only thing we lack now to complete all the article information is the article itself.
If you want a more in depth explanation, the plugin page for Xpath Checker has some, as well as this page about Xpath from W3Schools.
While we are still on the section page, we might as well get the XPATH query that will check if there are still more pages to the section so we can continue grabbing URLs.
//div[@class='ea-category-list']/p[@class='title']/a[text()='Next 30']/@href
This XPATH query is a little different, because we need to check if there is a link for “Next 30″. This type of situation is what makes XPATH such an excellent choice for scraping. Now checking if there are more pages is as simple as just popping in an “if” statement to see if there are elements in an array. In addition, I added “@href” at the end, which will only return the URL of the link to save some coding.
On to the Article Page.
On the article page, the XPATH query to get the information we want is this:
id('body')
This one was simple because EzineArticles puts all their articles into a div with an id.
Writing the Script
Now that we have the XPATH queries, we can begin coding. We’ll want to make this into a class, because you never know if you will want to reuse the script later. We might as well go ahead and take a few additional minutes so that we can use it for “any” section on EzineArticles.
The Constructor
PHP Classes, like most object-oriented programming, allow you to make a constructor that will fire certain actions once the class has been instantiated. This is especially useful for our needs so that we can just have it do stuff once we pass it a URL at instantiation.
class Scraper {
protected $articles = array();
protected $domain;
// Set actions to run when the class is instantiated
function __construct($url){
// Set the maximum execution time of the script to unlimited so that it can grab all the articles if there are a lot of them to scrape
set_time_limit(0);
// Set the root domain of the URL to concatinate with URLs later
$this->domain = explode("/", $url);
$this->domain = 'http://' . $this->domain[2];
// Pass the page URL you want to start scraping and start scraping through the section pages
$this->getArticleUrls($url);
echo count($this->articles) . ' - Done counting articles items, now adding articles.
';
// Loop through the article pages and grab the full article to finish populating the articles array with data
foreach ($this->articles as $item){
$item['article'] = $this->getArticles($item['url']);
}
echo count($this->articles) . ' - Done adding articles.';
// Add function here to start adding items in the article array with articles to a database
}
The above block of code basically creates a constructor that will do three things (minus the print outs to the screen which can be eliminated): take the given URL and get the root domain name (since EzineArticles uses relative URLs instead of the full URL for it’s article pages), call the getArticleURLs function, and call the getArticles function. So, when we instantiate the Scraper class those three things will happen without any more interaction needed.
Get Article URLs Function
// Start Get Article Urls
private function getArticleUrls($url){
// Instantiate next page variable to check at the end
$nextPageUrl = NULL;
// Instantiate cURL to grab the HTML page.
$c = curl_init($url);
curl_setopt($c, CURLOPT_HEADER, false);
curl_setopt($c, CURLOPT_USERAGENT, $this->getUserAgent());
curl_setopt($c, CURLOPT_FAILONERROR, true);
curl_setopt($c, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($c, CURLOPT_AUTOREFERER, true);
curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
curl_setopt($c, CURLOPT_TIMEOUT, 10);
// Add curl_setopt here to grab a proxy from your proxy list so that you don't get 403 errors from your IP being banned by the site
// Grab the data.
$html = curl_exec($c);
This part of the function Sets up cURL. It’s basically being told to not return header information, call the function I set to randomly show a common browser’s user agent, fail if it has an error, manage the referer itself, return the page instead of show it once it goes to the URL, and time out after 10 seconds so we won’t be waiting forever if the site is slow.
I’m not going to get into how to setup and use proxies with cURL, but it’d be wise if you did that since EzineArticles, and many high-traffic websites, will limit the number of requests per second or ban IP addresses if they get a lot of traffic from a single IP. It’s fairly simple to have a text file of proxies and randomly choose one to use with cURL.
The last line executes cURL with the above mentioned options and stores it in a variable named $html.
// Check if the HTML didn't load right, if it didn't - report an error
if (!$html) {
echo "
cURL error number: " .curl_errno($c) . " on URL: " . $url ."
" .
"
cURL error: " . curl_error($c) . "
";
}
// Close connection.
curl_close($c);
// Parse the HTML information and return the results.
$dom = new DOMDocument();
@$dom->loadHtml($html);
$xpath = new DOMXPath($dom);
// Get a list of articles from the section page
$articleList = $xpath->query("//div[@class='ea-category-list']/ol/li");
// Add each article to the Articles array
foreach ($articleList as $item){
$this->articles[] = array(
'url' => $this->domain . $item->getElementsByTagName('a')->item(0)->getAttribute('href'),
'title' => $item->getElementsByTagName('a')->item(0)->nodeValue,
'author' => $item->getElementsByTagName('em')->item(0)->getElementsByTagName('a')->item(0)->nodeValue,
'description' => $item->getElementsByTagName('div')->item(0)->nodeValue,
'article' => ''
);
}
The above block of code will output errors that cURL has while parsing the pages and then close the connection. It’s good practice to close a connection as soon as you don’t need it anymore. It frees up memory and CPU on your server.
Next, a DOM object is instantiated, then the page information we got through cURL is broken up into DOM elements that we can traverse with XPATH, which is what is instantiated next.
The XPATH query we setup earlier is plugged into the code and that little bit of the page is stored in $articleList for us to easily use in a foreach loop to grab the bits of information we need and store it in our $articles array.
In case you hadn’t noticed, I’m storing an associative array for each article. This makes it super easy to grab the specific information I want about the article later in the code and if I decided to store the array information into a database later.
Also, take note about how I’m setting the information for each array element. Each DOM element that I call $item in the foreach loop can be accessed just like you would if you’re familiar with JavaScript. It’s sharing the same DOM model for accessing each element, tag, and attribute – which is another reason that XPATH is so powerful and easy to use.
// Check to see if the Next 30 link is active
$nextPageUrl = $xpath->query("//div[@class='ea-category-list']/p[@class='title']/a[text()='Next 30']/@href");
if ($nextPageUrl){
$nextPageUrl = $nextPageUrl->item(0)->nodeValue;
// If there is a next page, go to it.
if (isset($nextPageUrl) && $nextPageUrl != ""){
$this->getArticleUrls($nextPageUrl);
}
}
}
// End Get Article Urls
This last bit of the function checks to see if there’s a link around “Next 30″. If there is, it will call the getArticleUrls function again. This is called recursion. Recursive programing is very powerful because it allows you to keep using the same code again and again until it’s done. The function is its own loop!
Get Article Function
I’m not going to display this one since it contains code that’s been touched on earlier: setting up and executing cUrl and XPATH.
Final Thoughts
Scraping data is a very powerful knowledge and skill to have. There’s a ton of ways to use scraped data: sell databases, use articles gotten from sites and spin them to use on your Web 2.0 or blog properties, etc. The possibilities are endless and only limited by your imagination.
Keep in mind, not all scraping is the same. Manipulating data to reuse and do stuff with (such as grabbing a list of sports stats from ESPN) is not the same as grabbing an article off EzineArticles or someone’s site and showing it on your own site. The later is copyright infringement, so be smart about what you do with scraped data.
I used EzineArticles as an example, I’m not advocating ripping their articles without you giving credit to the author and linking back to the main article and leaving their links in tact.
Download the Script
You can get the scraping script in a text file. Have fun and go make some money!
data mining, programming, scraping data, web content, web scraping
