Shamiq

Pictures from my Nikon D80, and possibly some from the cell phone if I can't sneak in the DSLR. Larger versions available, just leave a comment.
December 22

Scraper

Writing a website scraper in Ruby for a freelance gig I picked up on Craigslist. Using Nokogiri to make an XML tree out of the documents, then grepping for the stuff I need within the document.

Turns out if the designers of the page used a very structured format, it’s much easier to grab the text I need. This should have been obvious.

So my workflow:

  1. Do the task by hand and note all the URLs involved.

  2. Change URL parameters and see if you can get the web app to respond appropriately.

  3. Build your script to automatically generate the parameters.

  4. Build the modules that can parse the results.

  5. Test to ensure you’re covering all cases with your parser.

  6. Build the modules that’ll extract the data you need.

  7. Test the extractor.

  8. Manipulate the data so that it’s usable.