/xiaohongshu

简单的抓小红书笔记，仅用于学习。

Primary LanguageHTMLMIT LicenseMIT

🕷️xiaohongshu

Warning

The primary purpose of this repository is to learn. It is important to note that web crawling may be considered illegal, and therefore, it is crucial to refrain from exerting any pressure or engaging in unauthorized activities on websites.

Introduction

tanpenggood/xiaohongshu is a crawling application designed to extract data from xiaohongshu page.

crawling data range: only parsed data in window.__INITIAL_STATE__ of xiaohongshu page.

Development Environment

  • windows 11
  • jdk 1.8
  • maven 3.6.0

Useage

Use UI

Run com.itplh.xhs.XhsCrawlabUI

See:

home.png

Use API

  • crawl notes

    reference test class: com.itplh.xhs.XhsCrawlabTest

    xiaohongshu/src/test/java/com/itplh/xhs/XhsCrawlabTest.java

    Lines 9 to 21 in 075cdda

    @Test
    public void crawlHome() {
    // url带参数时，请求头可不携带cookie
    String url = "https://www.xiaohongshu.com/user/profile/64a91898000000001001e673?xhsshare=CopyLink&appuid=62064cd3000000001000acd1&apptime=1690553952";
    UserInfo userInfo1 = XhsCrawlab.getInstance().crawlHome(url);
    Assert.assertNotNull(userInfo1);
    Assert.assertNotNull(userInfo1.getRedId());
    // url不带参数时，请求头需要携带cookie
    String url2 = "https://www.xiaohongshu.com/user/profile/64a91898000000001001e673";
    UserInfo userInfo2 = XhsCrawlab.getInstance().crawlHome(url2);
    Assert.assertNotNull(userInfo2);
    Assert.assertNotNull(userInfo2.getRedId());
    }

  • save note to excel

    com.itplh.xhs.excel.ExcelGenerator.writeNotes2Excel(UserInfo userInfo)

Project Structure

xiaohongshu
├── src/main
│   ├── java/com.itplh.xhs       
│   │   ├── constant
│   │   ├── domain
│   │   ├── excel            # generate excel, use easyexcel
│   │   ├── parse            # parse json data (parse window.__INITIAL_STATE__)
│   │   ├── ui               # build ui, use javafx    
│   │   ├── util               
│   │   ├── XhsCrawlab       # core api   
│   │   └── XhsCrawlabUI     # ui
│   └── resources
│       ├── desktop          # response data of desktop access xiaohongshu
│       ├── mobile           # response data of mobile access xiaohongshu
│       └── logback.xml      # log config
├── src/test/java            # unit test
├── pom.xml
└── README.md

Technology Stack

Build

mvn clean package -Dmaven.test.skip=true

Download exe

https://github.com/tanpenggood/xiaohongshu/releases