# Firecrawl: turning a customer's website and public documents into Markdown for agents and RAG

> A help centre built as a SPA, a pile of PDFs and nobody knows where the sitemap is. Firecrawl handles the fetching, but you still have to set the scope with pathname regexes, cap the cost with limit and keep the data fresh with maxAge.

Original: https://fdetimes.net/en/tools/firecrawl-customer-docs-markdown-rag/

The customer's help centre is rendered in JavaScript, so curl gets back an empty HTML shell, while the product documentation is scattered across PDF and DOCX files.

Firecrawl was built for exactly this step. The company calls its product a "context API" for AI agents: give it a live web page and get back clean Markdown or structured data.

For an FDE, this shortens the path from "the customer sent a link" to "the agent can answer its first question" considerably. Using it well, though, means understanding three things: crawl scope, cost and data freshness.

## What does it take off your hands?

According to Firecrawl's documentation, the tool handles the hardest parts of web access: proxies, anti-bot measures, JavaScript rendering and dynamic content. Output comes as Markdown, structured JSON, screenshots and several other formats, all ready for an LLM to read. You do not have to stand up a headless browser just to read a customer's FAQ page.

The `/scrape` endpoint works on individual URLs. It detects PDF, DOCX and other document types from the URL and converts them to Markdown. So when a customer points you to a user guide published as a public PDF, you use the same call as you would for an HTML page.

Crawl works on a whole site. You send a root URL, Firecrawl uses the sitemap and links to discover pages, then scrapes every subpage it can reach.

## Example: crawling only the customer's documentation

Imagine the customer is a software company. Its site has `/blog`, `/careers`, `/docs/api` and `/docs/guides`, but the customer-support agent only needs the two `/docs` sections. If you crawl from the home page with no limits, Firecrawl will go up to its default of 10,000 pages, and each page costs one credit.

You narrow the scope with `includePaths` and `excludePaths`. This is where the most common mistake happens: both parameters are regexes matched only against the URL's **pathname**, not the full URL and not query parameters. A pattern written as `https://customer.com/docs/.*` will never match; the correct form is `^/docs/.*`.

The request body for a trial crawl, shown here for illustration, needs only three parameters:

```json
{
  "url": "https://customer.com",
  "includePaths": ["^/docs/.*"],
  "limit": 50
}
```

Because the regex sees only the pathname, you cannot use includePaths to filter on something like `?version=2`. If the customer versions its documentation through the query string, study the URL structure before running anything. The safe sequence is: scrape a few pages, crawl with a small `limit` as above, count the pages returned, and only then raise the limit.

**Key point:** Before crawling a customer's site, know how many pages you need and why.

## Stale data is a silent trap

By default, Firecrawl returns results from cache if a recent copy exists, with a 2-day cache window. Version 2 puts the emphasis on smart caching, which Firecrawl says makes scraping up to 10 times faster. For most documentation this helps: it is faster and means fewer repeat requests to the customer's site.

But think about a pricing page or a changelog. Because the cache can hold a copy for up to 2 days, if the customer updated prices this morning, your agent may still answer with the old ones. For URLs like these, set `maxAge=0` to bypass the cache:

```json
{
  "url": "https://customer.com/pricing",
  "maxAge": 0
}
```

Write this convention down in the customer handover documents: which URLs are read from cache and which always fetch a fresh copy.

## Who may crawl, and who may self-host?

Crawl respects robots.txt by default. Only the Enterprise plan can enable `ignoreRobotsTxt`. Even where that feature is available, permission to crawl a customer's site should be settled in a written agreement, not in a config file; on third-party sites, robots.txt is a line not to cross.

On licensing, the GitHub repository describes Firecrawl as an open-source API. The core uses AGPL-3.0, while the SDKs and some UI components use MIT. If the customer wants to self-host inside its own infrastructure, have the customer's legal team review the AGPL first, and you should be the one to raise that question.

Firecrawl names Zapier, Shopify and Replit among its customers, but what will convince your customer is a Markdown trial run on their own site.

## What to learn first

Start with `/scrape` on a difficult page: a SPA and a PDF. The point is to see what the Markdown output looks like before it goes into chunking. Next, learn to write pathname regexes and set `limit`, since that is where cost is decided, and only then move on to a caching strategy for each group of URLs.

When you put this experience on your CV, do not just write "used Firecrawl". Describe how you built a document-ingestion pipeline for RAG: how you scoped the crawl, how you estimated cost, what rules you used to refresh the data. That tells whoever reads your CV that you think carefully about customer data, rather than just listing tool names.

Firecrawl gets you Markdown quickly. Deciding which pages to fetch, and when to fetch them again, is still your job.

**Try this week:**

- Pick a public documentation page built as a SPA and a public PDF. Run /scrape on both and compare the Markdown you get with what you see in the browser.
- Write an includePaths regex for a single documentation section and run a trial crawl with a small limit. Check whether the number of pages returned matches your estimate.
- List the URLs whose content changes most often (pricing pages, changelogs) and decide which ones must be scraped with maxAge=0.

## Sources

- [Firecrawl Docs – Introduction](https://docs.firecrawl.dev/introduction)

- [Firecrawl Docs – Scrape](https://docs.firecrawl.dev/features/scrape)

- [Firecrawl Docs – Crawl](https://docs.firecrawl.dev/features/crawl)

- [firecrawl/firecrawl – GitHub](https://github.com/firecrawl/firecrawl)

- [Firecrawl v2 Series A announcement](https://www.firecrawl.dev/blog/firecrawl-v2-series-a-announcement)