/go-web-crawler

An experiment of crawling web pages with Go and asking Chat GPT for a summary of each page after collecting some initial data, and then merging the results with Google Analytics data.

Primary LanguageGo

Go Web Crawler

This project attempts to crawl a list of provided domains, merge the title, h1 and h2 values with a provided analytics export from Google Analytics, along with a generated summarization of each page from Open AI's text-davinci-003 completion model.