Data scraping in Ruby on Rails using Nokogiri and Mechanize gem

What is Data scraping?

Website/Data Scraping is a technique to operating large amounts of data from websites whereby the data is extracted and displayed in own sites or it can be stored to a File/Database. Data scraping is basically used where the websites does not provides API. 

We are having the Sample application running on the local



The above local site we will scrapped through the NokoGiri gem.
The steps are required:

1.  Add  the gem "gem 'nokogiri', '~> 1.8', '>= 1.8.1'".
2.  Then run the bundle install command.
3. Add the "require 'nokogiri'",  "require 'open-uri'" line where you will write the code for the scraping. 

staff_data = Nokogiri::HTML(open("http://localhost:4000/staffs"))
staff_data = staff_data.to_yaml

The Entire list of the staffs will be stored in the staff_data  We can process these data and we can stored it.

Mechanize Gem in rails 
The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, and can follow links and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

For the above site using Mechanize gem we can scrapped the data or search the record
The steps are required:

1.  Add  the gem "gem 'mechanize', '~> 2.7', '>= 2.7.5'".
2.  Then run the bundle install command.
3.  Add  require 'mechanize' in the controller.

def mechanize_scrap
  url = 'http://localhost:4000/staffs'
  staff_data = Mechanize.new
  #Get the URL
  staff_data.get(url)
  #Fill the input filed
  staff_data.page.forms[0].field_with(:id => "search").value = "test"
  #Submit the search form   
  staff_data.page.forms[0].submit 
  staff_data = staff_data.to_yaml                            
end

By using the mechanize gem we can select radio button.

staff_data.page.forms[0].radiobutton_with(:id => "First_drop").check

By using the mechanize gem we can see all input fields are in the form.

staff_data.page.forms[0].fields

By using the mechanize gem we can also select the drop down value of the site.

staff_data.page.forms[0].field_with(:id => "country").value = "Single"   

By using the mechanize gem we can find the form content of the site. 

form = staff_data.page.form_with(:id => "search_form")   

By using the mechanize gem we can find the button like these.

button = form.button_with(:value => "Search")

In the Mechanize gem link_with method is available in mechanize and it make simpler to fetch the random record link .

link = staff_data.link_with(text: 'Random article')

In the mechanize gem the click method instructs mechanize to follow the link

page = link.click

By using the mechanize gem we can find the page title of the site.

staff_data.page.title