Skip to content

Conversation

@nearestnabors
Copy link
Contributor

@nearestnabors nearestnabors commented Jan 17, 2026

Summary

  • Add <link rel="alternate" type="text/markdown"> to all page headers, pointing to the .md version of each page
  • Improve the MDX-to-markdown compilation so .md URLs return clean, readable markdown instead of raw MDX
  • Add fallback content for component-only pages (like the landing page) that shows the title, description, and link to the full interactive page

This enables LLM crawlers and training pipelines to discover and consume the markdown versions of our documentation, similar to what Vercel does with their docs.

Test plan

  • Visit any page and check the HTML source for <link rel="alternate" type="text/markdown" href="...">
  • Visit https://docs.arcade.dev/en/get-started/quickstarts/call-tool-agent.md - should return clean markdown with code blocks preserved
  • Visit https://docs.arcade.dev/en/home.md - should return fallback content with title/description and link to full page

🤖 Generated with Claude Code


Note

Introduces a clean markdown surface for each docs page and links to it from HTML.

  • Add <link rel="alternate" type="text/markdown"> in app/layout.tsx for all non-root pages, pointing to .../<path>.md
  • Implement MDX→Markdown compilation in app/api/markdown/[[...slug]]/route.ts:
    • Preserves frontmatter; strips imports/exports and JSX (extracts text from JSX children)
    • Protects code blocks, normalizes indentation, removes excessive blank lines
    • Provides fallback markdown (title/description/link) for component-only pages
    • Sets response headers to Content-Type: text/markdown and returns 404 for missing pages

Written by Cursor Bugbot for commit b0682e8. This will update automatically on new commits. Configure here.

nearestnabors and others added 3 commits January 17, 2026 01:04
- Add <link rel="alternate" type="text/markdown"> to page headers pointing to .md version
- Improve MDX-to-markdown compilation to produce clean markdown output
- Preserve code blocks and frontmatter while stripping JSX components

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Pages that only contain React components (like the landing page) now
return a helpful markdown response with the title, description, and
a link to the full interactive page.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@vercel
Copy link

vercel bot commented Jan 17, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Review Updated (UTC)
docs Ready Ready Preview, Comment Jan 20, 2026 7:53pm

Request Review

@evantahler
Copy link
Contributor

Screenshot 2026-01-16 at 5 18 00 PM On https://docs-git-feature-add-md-alternate-links-arcade-ai.vercel.app/en/get-started/quickstarts/call-tool-agent.md, there's something weird with the insertion of new codeblocks. The "```" I think needs to be at the start of the line

cursor[bot]

This comment was marked as outdated.

- Add dedent function to normalize indentation when extracting content from JSX components
- Add normalizeIndentation function to clean up stray whitespace while preserving meaningful markdown indentation (nested lists, blockquotes)
- Move list detection regex patterns to module top level for performance
- Ensures code block markers (```) start at column 0

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
cursor[bot]

This comment was marked as outdated.

cursor[bot]

This comment was marked as outdated.

The previous regex patterns `["']?([^"'\n]+)["']?` would truncate text
at the first apostrophe (e.g., "Arcade's" became "Arcade").

This fix:
- Uses separate patterns for double-quoted, single-quoted, and unquoted values
- Requires closing quotes to be at end of line to prevent apostrophes from
  being misinterpreted as closing delimiters
- Adds stripSurroundingQuotes helper for fallback cases

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
cursor[bot]

This comment was marked as outdated.

When x-pathname header is not set, pathname defaults to "/" which would
produce an invalid alternate link "https://docs.arcade.dev/.md".
Only render the alternate link when we have a real page path.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@nearestnabors
Copy link
Contributor Author

@evantahler Got a minute to have a second look?

@evantahler
Copy link
Contributor

I like the <link rel="alternate" type="text/markdown"> idea a lot that this PR introducdes, but I'm curious as to why the MDX needed to 'rendered' to MD? Were the react components harming LLM legibility? There are still HTML elements in the pages (e.g. https://docs-git-feature-add-md-alternate-links-arcade-ai.vercel.app/en/get-started/setup/api-keys.md). Are HTML fragments OK?

If the goal is to keep html fragments, buy 0-out react, I'd suggest an alternative approach:

  • make a new MDX renderer in app/api/markdown/[[...slug]]/route.ts
  • pass a wildcard react component that returns "</>"
  • rather than using regular expressions, render the mdx to md "for real", preserving anything that wasn't react.

@nearestnabors
Copy link
Contributor Author

@evantahler Dammit!

So when agents parse markdown, HTML and MDX getting mixed in there make it hard on them. IIIRC from the friend who did this, the entire thing needs rendering down to markdown.

What approach do you recommend in light of this?

@nearestnabors
Copy link
Contributor Author

Actually, let me just try parsing the HTML back into Markdown. We have some complex MDX.

nearestnabors and others added 2 commits January 20, 2026 17:51
- Add scripts/generate-markdown.ts to pre-render MDX to markdown
- Update proxy.ts to serve static .md files from public/
- Delete API route in favor of static file serving
- Add link rewriting to add /en/ prefix and .md extension
- Add markdown-friendly component implementations
- Fix localhost URL in gmail integration page

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
cursor[bot]

This comment was marked as outdated.

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants