/scraper-project

Primary LanguageJavaScriptMIT LicenseMIT

💎 BelieveInBunny

As I encode anime and share them, I wanted to share OneDrive folder links to users. So I indexed my OneDrive to make the experience better (and to not receive DMCAs on my actual drive).

Issues:

#Users could easily access folders of other encodes or the complete indexed OneDrive.

#I choose .mkv as container for my videos, which wasn't supported.

#It supports H264 in-browser streaming but not for H265.

🎁My mods:

  1. If the user visits the Public directory, they won't be able to access all the indexed files. (here, Team Drive site, you can't see the files/folders indexed, but Test folder, here you can)
  2. Force-added support for .mkv files to detect as streamable file format.

🐱‍👤Failures:

  1. Can't add support for H265 in-browser streaming as majority of the browsers don't support it (except, Safari).
  2. Couldn't find an HTML5 player with multiple audio and/or subs support, so, stuck with default.
  3. Idk anything about JS and was therefore unable to contribute much.

Hosted on Cloudflare Workers Deploy AnimeSub

onedrive-cf-index

🏵 Yet Another OneDrive Index. Powered by Cloudflare Workers. Inspired and originated greatly from heymind/OneDrive-Index-Cloudflare-Worker.

Table of contents

Demo

Live demo: 📁 AnimeSub's OneDrive Index.

Scenario Screenshot
Home
Folder

Features

Improvements

New features

  • New design: spencer.css.
  • File icon rendered according to file type. Emoji as folder icon when available (if the first character of the folder name is an emoji).
  • Use Font Awesome icons instead of material design icons (For better design consistency).
  • Use github-markdown-css for README.md rendering → Demo.
  • Add breadcrumbs for better directory navigation.
  • Support file previewing:
    • Images: .png, .jpg, .gif → Demo.
    • Plain text: .txt → Demo.
    • Markdown: .md, .mdown, .markdown → Demo.
    • Code: .js, .py, .c, .json... → Demo.
    • PDF: Lazy loading, loading progress and built-in PDF viewer → Demo.
    • Music / Audio: .mp3, .aac, .wav, .oga → Demo.
    • Videos: .mp4, .flv, .webm, .m3u8 → Demo.
    • ...
  • Code syntax highlight in GitHub style. (With PrismJS.)
  • Image preview supports Medium style zoom effect.
  • Token cached and refreshed with Cloudflare Workers KV storage. (We got rid of external Firebase dependencies!)
  • Route lazy loading with the help of Turbolinks®.
  • Supports OneDrive 21Vianet.（由世纪互联运营的 OneDrive。）
  • ...

Under the hood

  • CSS animations all the way.
  • Package source code with wrangler and webpack.
  • Convert all CDN assets to load with jsDelivr.
  • Almost all scripts are loaded with webpack! (Other than some libraries for rendering file previews.)
  • ...

All other features

See: New features | OneDrive-Index-Cloudflare-Worker.

Deployment

Online token generation tool taken from the generous: https://heymind.github.io/tools/microsoft-graph-api-auth.

Generating OneDrive API Tokens

  1. Create a new blade app here Microsoft Azure App registrations (OneDrive normal version) or Microsoft Azure.cn App registrations (OneDrive 世纪互联版本) with:
    1. Supported account types set to Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox). OneDrive 世纪互联用户设置为：任何组织目录（任何 Azure AD 目录 - 多租户）中的帐户.
    2. Redirect URI (optional) set to "Web: https://heymind.github.io/tools/microsoft-graph-api-auth".
  2. Get your Application (client) ID - client_id at Overview panel.
  3. Open Certificates & secrets panel and create a new secret called client_secret.
  4. Add permissions offline_access, Files.Read, Files.Read.All at API permissions.
  5. Get your refresh_token using https://heymind.github.io/tools/microsoft-graph-api-auth.
  6. Create a dedicated folder for your public files inside OneDrive, for instance: /Public. Please don't share your root folder directly!

If you can't fetch the access_token and/or refresh_token on step 5, please resolve to the solution suggested in the pinned issue #13.

After all this hassle, you should have successfully acquired the following tokens and secrets:

  • refresh_token
  • client_id
  • client_secret
  • redirect_uri: Defaults to https://heymind.github.io/tools/microsoft-graph-api-auth.
  • base: Defaults to /Public.

Preparations

Fork the repository. Install dependencies.

# Install cloudflare workers official packing and publishing tool
yarn global add @cloudflare/wrangler

# Install dependencies with yarn
yarn install

# Login to Cloudflare with wrangler
wrangler config

# Verify wrangler status with this command
wrangler whoami

Create a DRAFT worker at Cloudflare Workers with a cool name. Get your own Cloudflare account_id and zone_id: Docs - Account ID And Zone ID.

Create Cloudflare Workers KV bucket named BUCKET:

# Create KV bucket
wrangler kv:namespace create "BUCKET"

# ... or, create KV bucket with preview functions enabled
wrangler kv:namespace create "BUCKET" --preview

Modify wrangler.toml:

  • name: The draft worker's name, your worker will be published at <name>.<worker_subdomain>.workers.dev.
  • account_id: Your Cloudflare Account ID.
  • zone_id: Your Cloudflare Zone ID.
  • kv_namespaces: Your Cloudflare KV namespace, you should substitute the id and preview_id values accordingly. If you don't need preview functions, you can remove the preview_id field.

Modify src/config/default.js:

  • client_id: Your client_id from above.
  • base: Your base path from above.

For Chinese 21Vianet OneDrive users. OneDrive 世纪互联用户：将 useOneDriveCN 设置（修改）为 true。

Add secrets to Cloudflare Workers environment variables with wrangler:

# Add your refresh_token and client_secret to Cloudflare
wrangler secret put REFRESH_TOKEN
# ... enter your refresh_token from above here

wrangler secret put CLIENT_SECRET
# ... enter your client_secret from above here

Building and deployment

You can preview the worker with wrangler:

wrangler preview

After making sure everything is ok, you can publish your worker with:

wrangler publish

You can also create a GitHub Actions for auto publishing your worker on push. See main.yml.

Customizations

  • You can (AND SHOULD) change the intro on the default landing page here: src/folderView.js. Write HTML directly.
  • You can (AND ALSO SHOULD) change the header of the site here: src/render/htmlWrapper.js.
  • Your custom styles are loaded from themes/spencer.css, change that according to your customizations. You may also need to change the commit HASH at src/render/htmlWrapper.js.
  • You can also customize Markdown CSS styles, PrismJS code highlight color schemes, etc.

🏵 onedrive-cf-index ©Spencer Woo. Released under the MIT License.

Authored and maintained by Spencer Woo.

@Portfolio · @Blog · @GitHub