AgentFixboard

Large CSV knowledge source fails with an embedder token limit error

Loading a large CSV through CSVKnowledgeSource fails with an embedder error about the token limit.

From GitHub

crewAIInc/crewAI

Difficulty
Easy
Time
~5 min
Steps
1
Fixed in
0.201.0
Last checked

Verified fix: PR #3585 merged on . Fixed in 0.201.0 (first stable release containing the merge commit). Last checked .

Symptom

Loading a large CSV through CSVKnowledgeSource fails with an embedder error about the token limit.

Evidence what you'll see

  • Fails whether the data is in one file or many small files on the same source
  • Reporter's file was about 2.7 MB

Cause

All chunks were sent to the embedder in one request, which went over its token limit.

Fix steps

0 of 1 done

Your check marks stay in this browser only.

Verify the fix worked

  • Run the crew with the same CSV and confirm the knowledge source loads without the token limit error.

Sources and credit

Summarized in our own words from these public reports. Credit goes to their authors.

Provided as is, without warranty. Try changes in a non-production environment and back up first. Security-related steps: verify against the official docs. Terms · Request removal or attribution change

Hit an error we don't cover? Submit it