Job Search Project using Web scraping and Django

job search project using web scraping and Django

Hi Guys..! a few days back I posted one article about optimizing the job search using Python and also I made a video on that in my YouTube channel. Now I decided to give a web interface for that.

I suggest you please check it once before start this, Link is Here

To develop a web application, i am using python Django framework. so let’s get started..!

Scraping Indeed job’s Data using Python

I modified and added a little more things to the previous version of script, here is the script.

# jp.py

import requests
from bs4 import BeautifulSoup

#job = input('Enter job Tilte:')
#location = input('Enter Location:')

#print(job)



#url = 'https://www.indeed.co.in/jobs?q=python+developer&l=Bengaluru&sort=date'

#url ='https://www.indeed.co.in/jobs?q='+job+'&l='+location+'&sort=date'



titles  =[]
links = []
companies = []
summaries  = []

dates =[]


def job_data(url,items):

    res = requests.get(url).content
    soup = BeautifulSoup(res,'html.parser')
    data = soup.find_all('div',class_='jobsearch-SerpJobCard')

    for i in data:

        title = i.find('h2',class_='title')

        if items[0] in title.text:
            company = i.find('span',class_='company')
            link = title.find('a')
            summary  =i.find('div',class_='summary')

            date = i.find('span',class_='date')

            titles.append(title.text.strip())
            links.append('https://www.indeed.co.in'+link['href'])
            companies.append(company.text.strip())
            dates.append(date.text.strip())
            summaries.append(summary.text.strip())

            # print('\nJob Title:',title.text)
            # #print('posted:',date.text)
            # print('Company Name:',company.text)
            # print('Job Summary:',summary.text)
            # print('posted:',date.text)
            # print('https://www.indeed.co.in'+link['href'])
            # print(10*'*****')

    return titles,companies,summaries,dates,links

#job_data(url)

copy this python code and create new python file and named as “jp.py” . you can rename it, but make sure when you import into the Django project views.

Job Search Project using Web scraping and Django

OK, let’s create a new Django project and configure templates and in that create a new template called index.html.

Copy the below HTML code and paste it in your file.



<!doctype html>
<html lang="en">
  <head>
    <!-- Required meta tags -->
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

    <!-- Bootstrap CSS -->
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.5.0/css/bootstrap.min.css" integrity="sha384-9aIt2nRpC12Uk9gS9baDl411NQApFmC26EwAOH8WgZl5MYYxFfc+NcPb1dKGj7Sk" crossorigin="anonymous">

    <title>Job portal</title>
  </head>
  <body>


<div class="container mt-10 p-5 text-justify">
  <nav>
    
    <button class="btn btn-danger" type="button"><a class="text-white" href="{% url 'home'%}">Home</a></button>
    
  </nav>
<h1 class="text-center">Job Search Here..!</h1>

<form action="" method="POST">{% csrf_token %}
    <!-- Search form -->
<div class="active-pink-3 active-pink-4 mb-4 col-xs-4">
    <input name='title' class="form-control col-xs-4" type="text" placeholder="Enter job Title" aria-label="Search" required>
  </div>
  <!-- Search form -->
  <div class="active-purple-3 active-purple-4 mb-4">
    <input name ='location' class="form-control" type="text" placeholder="Enter Location" aria-label="Search" required>
  </div>
  <button class='btn btn-success' type="submit">Search</button>
</form>

<h1 class='text-center'>Job Results Here</h1>

<h1> {{message}}</h1>

<hr>
    {% for title,comapny,summary,date,link in data %}
      
    
    <div class="card">
        <div class="card-header">
         New
        </div>
        <div class="card-body">
          <h5 class="card-title">Job Title: {{title}}</h5>
          <b class="card-title">Company: {{comapny}}</b>
          <p class="card-text">job Description: {{summary}}</p>
          <p class="card-text">Posted:{{date}}</p>
          <a href="{{link}}" class="btn btn-info">Apply</a>
        </div>
      </div>
      {% endfor %}

      <footer>
        <div class="row">
          <p class='p-3'>&copy; Jp @2020</p>
          <p class='p-3'>Visit us: <a href="https://www.devpyjp.com" target="_blank">DevPyJp</a> </p>
          <p class='p-3'>Watch us: <a href="https://youtube.com/c/jpnan/" target="_blank">Jp NAN</a> </p>
        </div>
      </footer>

    </div>
    <!-- Optional JavaScript -->
    <!-- jQuery first, then Popper.js, then Bootstrap JS -->
    <script src="https://code.jquery.com/jquery-3.5.1.slim.min.js" integrity="sha384-DfXdz2htPH0lsSSs5nCTpuj/zy4C+OGpamoFVy38MVBnE+IbbVYUew+OrCXaRkfj" crossorigin="anonymous"></script>
    <script src="https://cdn.jsdelivr.net/npm/popper.js@1.16.0/dist/umd/popper.min.js" integrity="sha384-Q6E9RHvbIyZFJoft+2mJbHaEWldlvI9IOYy5n3zV9zzTtmI3UksdQRVvoxMfooAo" crossorigin="anonymous"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.5.0/js/bootstrap.min.js" integrity="sha384-OgVRvuATP1z7JjHLkuOU7Xw704+h835Lr+6QL9UvYjZE3Ipu6Tp75j7Bh/kR0JKI" crossorigin="anonymous"></script>
  </body>
</html>


Note: This is a web scraping project so we don’t need to create models and apps, so I created a project level views.py

from django.shortcuts import render 
import requests
from bs4 import BeautifulSoup

from .jp import job_data


def jobs(request):


    if request.method == 'POST':
        job = request.POST['title']
        location = request.POST['location']

    
        job = job.strip()
        j = job.title().strip()
        items = j.split(' ')
        location = location.strip()
        job = job.replace(' ', '+')

        url ='https://www.indeed.co.in/jobs?q='+job+'&l='+location+'&sort=date'

        x,y,z,a,b = job_data(url,items)
       

        data = zip(x,y,z,a,b)

        if len(x)>0:

            context ={
                'data':data,
                
            }
        else:
            context ={
                'message':'No Jobs Found..!',
                
            }


        return render(request,'index.html',context)
    
    return render(request,'index.html')

And now configure this view in your urls.py, if you made any rename to the files, please make sure to change them whenever you imported them.

from django.contrib import admin
from django.urls import path
from .views import jobs


urlpatterns = [
    path('admin/', admin.site.urls),
    path('',jobs,name='home'),
]

We don’t need to made migrations for this because we never create any models.

lets execute your project using the below command.

python manage.py runserver

I hope it will helps you to make a web scraping project with Django. In this way, we can create web application for job search.

Git repository to this project : Code

If you like this Job Search Project using Web scraping and Django tutorial please share with others and if you have any doubts please comment below.

1 thought on “Job Search Project using Web scraping and Django”

Leave a Reply