[docs] Add `llms.txt` and fix `robots.txt` 404s to improve LLM accessibility and SEO

[emerson-proenca]

Question you want answered

How do I make my agents better at Tauri?

Where did you look for an answer?

Currently, the Tauri documentation does not provide llms.txt files for LLM crawlers, which could help LLMs locate and interpret documentation more accurately. Also, robots.txt returns a 404 error on both tauri.app and v2.tauri.app.

Proposed Changes

1. Add llms.txt to documentation pages. Example: https://tauri.app/reference/acl/capability/llms.txt
2. Fix robots.txt 404 errors. Example: https://tauri.app/robots.txt and https://v2.tauri.app/robots.txt

This will probably require changes in the docs, I'm not sure how they are created.

Why?

• Improves LLMs code: LLMs can scrape clean text versions instead of HTML, leading to more precise answers and code which is aligned with Tauri’s docs.
• Future: As LLMs increasingly rely on web scraping for knowledge, having llms.txt ensures Tauri’s documentation is interpreted correctly and up to date!
• SEO: Having robots.txt increases SEO, and having llms.txt redirects bot traffic to robots pages, which can be useful.

Implementation

The HTML change is simple, just add "llms.txt" link next to the "Edit page" button. Example:

<div class="meta sl-flex astro-ill2nn3f">
   <a href="https://github.com/tauri-apps/tauri-docs/edit/v2/src/content/docs/reference/acl/capability.md" class="sl-flex print:hidden astro-jpwltabt">
      <svg aria-hidden="true" class="astro-jpwltabt astro-sqj4yx7f" width="16" height="16" viewBox="0 0 24 24" fill="currentColor" style="--sl-icon-size: 1.2em;">
         <path d="M22 7.24a1 1 0 0 0-.29-.71l-4.24-4.24a1 1 0 0 0-1.1-.22 1 1 0 0 0-.32.22l-2.83 2.83L2.29 16.05a1 1 0 0 0-.29.71V21a1 1 0 0 0 1 1h4.24a1 1 0 0 0 .76-.29l10.87-10.93L21.71 8c.1-.1.17-.2.22-.33a1 1 0 0 0 0-.24v-.14l.07-.05ZM6.83 20H4v-2.83l9.93-9.93 2.83 2.83L6.83 20ZM18.17 8.66l-2.83-2.83 1.42-1.41 2.82 2.82-1.41 1.42Z"></path>
      </svg>
      Edit page
   </a>
   <a href="https://tauri.app/reference/acl/capability/llms.txt" class="sl-flex print:hidden astro-jpwltabt">
      <svg aria-hidden="true" class="astro-jpwltabt astro-sqj4yx7f" width="16" height="16" viewBox="0 0 24 24" fill="currentColor" style="--sl-icon-size: 1.2em;">
         <path d="M12 6c1.45-1.1 3.55-1.5 5.5-1.5 1.45 0 2.99.22 4.27.79.73.33 1.22 1.04 1.22 1.85v11.28c0 1.31-1.22 2.27-2.48 1.94-.98-.25-2.02-.36-3.02-.36-1.56 0-3.22.25-4.56.92-.59.3-1.27.3-1.87 0-1.34-.66-3-.92-4.56-.92-1 0-2.04.11-3.02.36-1.26.32-2.48-.64-2.48-1.94V7.14c0-.81.49-1.52 1.22-1.85 1.29-.57 2.83-.79 4.28-.79 1.95 0 4.05.4 5.5 1.5Zm7.8 12.21c.62.11 1.2-.35 1.2-.98V7.76c0-.47-.34-.88-.8-.98-.87-.19-1.78-.28-2.7-.28-1.7 0-4.15.65-5.5 1.5v11.51c1.35-.85 3.8-1.5 5.5-1.5.77 0 1.55.06 2.3.2Z"></path>
      </svg>
      llms.txt
   </a>
</div>

Here's how it would look like:

References

I’m not sure how to implement and maintain this, but I’m happy to help. Here are some other projects using llms.txt:

• https://svelte.dev/docs/svelte/overview/llms.txt/
• https://fastht.ml/docs/tutorials/by_example.html.md/
• https://docs.langchain.com/use-these-docs/
• https://llmstxt.org/

Page URL

https://v2.tauri.app/reference/acl/capability/llms.txt

Additional context

No response

Are you willing to work on this yourself?

• I want to work on this myself

[FabianLars]

There's https://v2.tauri.app/llms.txt / https://v2.tauri.app/llms-full.txt already. I'm really not a fan of per-site llms.txt files, that's so overloaded.

Will keep this open for robots.txt though. Do you perhaps have some examples for that? All sites that are similar to ours that i just tried (electron, vite, etc) all 404 as well so idk really know what kind of robots.txt one would expect from a site like ours.

[emerson-proenca]

I tried llms.txt on many pages but I didn't try at root lol. My problem is solved, because I can use llms-full.txt to train the LLMs, perhaps it's possible to close #3095? Also it'd be a good idea to show that Tauri has a llms-full.txt, somewhere in the website or in GitHub.

About the robots.txt it's mostly for Tauri to allow or disallow crawlers and scrapers to access the website. At first I didn't find llms.txt, so my idea was to daily scrape all the docs. Also having a robots.txt improves SEO not that it'll make a difference, it might go from 99 to 100.