Web scraping beginner guide

 Every web scraping project should start with a goal in mind. 

The goal of this beginner guide is to collect faculty details from a college website and save data to a spreadsheet using python.

College website: https://www.ifheindia.org/icfaitech/datascience.html

Python code used:

# imports for doing http request, reading text/image date and writing text/image data to spreadsheet

import requests

from bs4 import BeautifulSoup

import xlsxwriter

from io import BytesIO

# HTTP request to staff website

response = requests.get("https://www.ifheindia.org/icfaitech/datascience.html")

# Parse HTML response got from above

soup = BeautifulSoup(response.content, "html.parser")

# Get all div data with class 'profilePic'

profile_pic_divs = soup.findAll("div", {"class": "profilePic"})

# Create spreadsheet 'AI_staff.xlsx'

workbook = xlsxwriter.Workbook('AI_staff.xlsx')

# Add worksheet for above spreadsheet

ws = workbook.add_worksheet()

ws.set_column('A:A', 20)

# Adding a bold format to use to highlight cells.

style = workbook.add_format({'bold': True})

i = 0

# Iterate on all available profile pic div's

for each_div in profile_pic_divs:

# Create image URL

url = ('https://www.ifheindia.org/icfaitech/'+each_div.img['src'])

# Make HTTP request for image URL and reading the steam

data = BytesIO(requests.get(url, stream=True).content)

print(each_div.h4.text)

i+=1

# Writing text and image data to AI_staff.xlsx

ws.write("A"+str(i), each_div.h4.text, style)

ws.insert_image("B"+str(i), url, {"image_data": data})

# Close workbook object

workbook.close()

Explanation:

• Modules and functions used:
  

  Module	Function	Purpose
  Python	requests	to send http request to https://www.ifheindia.org/icfaitech/ and recieve data as response
  bs4	BeautifulSoup	Parse HTML response
  Python	xlsxwriter	Module for writing data or creating XLSX file(Excel) format
  io	BytesIO	BytesIO objects can be used to read and write binary data in this case for reading image data

• logic flow diagram