This is the "backend" scraping script for the Home Finder project.
If you want to also deploy the frontend website, read it here.
To deploy this project, you will need to do the following steps:
- Clone the project using the
Use this template
button on GitHub and make sure to change the project to private, otherwise you may leak user data. - Update the
template.html
to use your own website URL- or if you're skipping the frontend, you can simply remove the link in the footer of the email.
- Get the Google Spreadsheet from the previously created form and make sure to publish the Google Spreadsheet to the web and make it available to anyone with the link too.
- or if you're skipping the frontend any Google Spreadsheet will do, but the spreadsheet needs to have the same header fields as the Google Forms.
- Get the Google Spreadsheet
ID
andGrid ID
, for example, for this spreadsheet, theID
is1jRMEvfI6OsWUwnaHgH5UwnoikZA0a3s8wPnCortNJ_A
and theGrid ID
is0
. - Add 5 new repo's secret:
EMAIL_USER
: The Gmail email that will be used to send emails.EMAIL_PASSWORD
: The password of the Gmail account. It's highly recommended that you create an App Password for that.GOOGLE_SPREADSHEET_ID
: The Google SpreadsheetID
.GOOGLE_SPREADSHEET_GID
: The Google SpreadsheetGrid ID
.USER_AGENT
: The user-agent to be used by the scraper script. You can use your own user-agent if you want.
- Update the cron job frequency to match your needs.
- Push your changes to the
main
branch and the script will automatically run accordingly with the cron job settings.
- v0.2.0:
- Add check for valid email and valid URL
- Add bugs to fix later
- v0.1.0:
- First version