## Overview

Sitemaps serve as a directory of the pages and content on your website, and creating these files is one of the most basic SEO steps to take. A sitemap is essentially a roadmap to the pages on your website. It can be in either HTML or XML format, with the latter used in the `next-sitemap` package that I'll discuss here.

`robots.txt` communicates which pages your site permits bots to crawl. While this considered a terms of use agreement, it *does not* restrict actors from accessing your site. This file is not sufficient for ensuring operational security. Additional measures must be implemented to prevent unauthorized access to URL paths and files. The `robots.txt` file is a tool to assist search engines in crawling your site.

Generating sitemaps and a `robots.txt` file is incredibly simple with [next-sitemap](https://github.com/iamvishnusankar/next-sitemap). Prioritizing efficiency, I used the most basic configuration for this package, which met my needs. As the blog grows, I may revisit some of the configuration options. I just needed these files, and I did not want to manually create them. It's also great that the `postbuild` script ensures that the files are updated whenever the application is built for production. This means that anytime I add a new post and rebuild, the sitemap will be updated with the new URL.

## Using `next-sitemap`

By default, the build process generates an index sitemap along with other sitemaps containing website URLs. To generate a `robots.txt` file, you must add `generateRobotsTxt: true` as an option in the configuration file. The project's README contains information on splitting sitemap files when you have a large number of URLs (over 7000). That information can be found [in this section](https://github.com/iamvishnusankar/next-sitemap#index-sitemaps-optional).

1. Install the package using your package manager of choice:

```sh
yarn add next-sitemap
```

1. Next, you'll need to create a configuration file. This should go in the root directory of your Next.js project. The file must be called `next-sitemap.config.js`.

```js
// next-sitemap.config.js

/** @type {import('next-sitemap').IConfig} */
module.exports = {
    siteUrl: process.env.SITE_URL || 'https://paul-schick.com',
    generateRobotsTxt: true,
}
```

3. I added a `postbuild` script to my `package.json` to generate these files when the application is built for production.

```json
// package.json scripts
{
  "scripts": {
    "dev": "next",
    "build": "next build",
    "postbuild": "next-sitemap"
  }
}
```

The command run by node is simply `next-sitemap`. I'll be covering more `postbuild` stuff in a later post while I document my Next.js blog strategy. I'm using static files rather than a database (for now), so I want to make use of some command line builds to reduce indexing time and generate static files.

If you're looking for a deployment guide to go with this, see [Deploying a Next.js App with Docker and Nginx on GCP](/posts/deploying-nextjs-with-docker-nginx-on-gcp/) for a full walkthrough of containerization, reverse proxy setup, and SSL.

And if you're building your Next.js site with an AI coding agent, [How to Write the Perfect CLAUDE.md File](/posts/how-to-write-claude-md/) covers giving the agent durable project context - including conventions like the sitemap and build scripts set up here.