defenseunicorns/leapfrogai

bug(api): Sometimes PDF contents gets mangled on extraction

CollectiveUnicorn opened this issue · 0 comments

CollectiveUnicorn commented

Steps to reproduce

  1. Download a PDF like the one attached here: aesopsfables00aeso.pdf
  2. Upload the PDF to using the API
  3. Create a new vector store or update an existing one using the PDF
  4. Wait for the vector indexing to complete
  5. Observe in the DB that the content is missing spaces between words.

Expected result

  • The content piece of the vector_content has spaces so that when given to the LLM the text is intelligible.

Actual Result

  • The content piece of the vector_content does not have spaces, so the LLM gets a glob of text.

Visual Proof (screenshots, videos, text, etc)

  • Screenshot 2024-09-25 at 2 47 03 PM
  • Screenshot 2024-09-25 at 2 47 37 PM

Additional Context

  • This does not happen for all PDFs only some of them.