The ai-web-scrapping project is an advanced web scraping application integrated with artificial intelligence capabilities. The primary purpose of this app is to intelligently collect data about a specified piece of information from various online sources. The application utilizes AI algorithms to enhance the efficiency and accuracy of the web scraping process.
First, you must install the dependencies with npm i
.
Now, you have to create your .env
for this, copy the .env.example
in your .env
and fill in the envs.
OPENAI_API_KEY="YOUR OPENAI_API_KEY"
OPENAI_API_URL="YOUR OPENAI_API_URL"
SERP_API_KEY="YOUR SERP_API_KEY" Access the https://serpapi.com/ to get your SERP_API_KEY
DATABASE_URL="YOUR DATABASE_URL"
DB_USER="YOUR DB_USER"
DB_PASSWORD="YOUR DB_PASSWORD"
ACCESS_TOKEN_SECRET="YOUR ACCESS_TOKEN_SECRET"
Now, you must run docker compose -d
to upload the database container.
Then you have to run npx prisma generate
and npx prisma Migrate deploy
.
Now, you can run npm run dev
to run the development server and make requests.
-
AI Integration: Leverage the power of artificial intelligence to intelligently extract relevant data.
-
Web Scraping: Collect information from websites based on user-defined parameters.
-
Security: Implement security measures such as encryption and authentication to ensure data integrity.
Follow these steps to set up and run the project locally:
-
Installation:
npm install
-
Development Server:
npm run dev
-
Production Build:
npm run build npm run start:prod
lint: Run ESLint to ensure code quality.
dev: Start the development server using Nodemon for automatic restarts.
start: Start the server using ts-node for easy TypeScript execution.
start:prod: Build the project and run the production server.
build: Compile TypeScript code into JavaScript.
@prisma/client: 5.7.0
bcrypt: 5.1.1
cors: 2.8.5
dotenv: 16.3.1
express: 4.18.2
helmet: 7.1.0
jsonwebtoken: 9.0.2
langchain: 0.0.202
prisma: 5.7.0
serpapi: 2.0.0
zod: 3.22.4
@types/bcrypt: 5.0.2
@types/cors: 2.8.17
@types/express: 4.17.21
@types/jsonwebtoken: 9.0.5
@types/node: 20.10.3
@typescript-eslint/eslint-plugin: 6.13.2
@typescript-eslint/parser: 6.13.2
eslint: 8.55.0
eslint-config-prettier: 9.1.0
eslint-plugin-prettier: 5.0.1
nodemon: 3.0.2
prettier: 3.1.0
ts-node: 10.9.1
typescript: 5.3.2
License
Claudio Cassimiro
-
Endpoint:
- POST
/api/topic
- POST
-
Description: Create a new topic with web scraping and AI integration.
-
Endpoint:
- POST
/api/getTopics
- POST
-
Description: Returns all topics and related data.
-
Middleware:
authenticateToken
: Ensures the request is authenticated.
-
Controller:
topicController.topic
: Handles the creation of a new topic.
-
Endpoint:
- GET
/api/:id
- GET
-
Description: Retrieve information about a single user.
-
Middleware:
authenticateToken
: Ensures the request is authenticated.
-
Controller:
getSingleUserController
: Retrieves information about a user.
-
Endpoint:
- POST
/api/register
- POST
-
Description: Register a new user.
-
Controller:
createUserController
: Handles user registration.
-
Endpoint:
- POST
/api/login
- POST
-
Description: Authenticate a user's login.
-
Controller:
authPasswordController
: Handles user authentication.
-
Endpoint:
- PUT
/api/:id
- PUT
-
Description: Update the password of a user.
-
Middleware:
authenticateToken
: Ensures the request is authenticated.
-
Controller:
updateUserPasswordController
: Handles the update of a user's password.
-
Endpoint:
- DELETE
/api/:id
- DELETE
-
Description: Delete a user account.
-
Middleware:
authenticateToken
: Ensures the request is authenticated.
-
Controller:
deleteUserController
: Handles the deletion of a user account.