How I Built This Site

The SSG promise-land, finally

Have you ever used a static site generator?

I have been largely dissapointed in the SSGs that I have tried. Back in the day, most popular ones like Hyde and Jekyll had simple and rigid data models, locking you into “pages” and “posts”. Modern SSGs have more flexibility but require you to jump through the tedious hoops of their APIs. NextJS, Gatsby, they all have this issue.

However, I have finally found the SSG promise land.

More Than a Blog

I have always wanted a personal website that was more than a simple blog. I wanted a place where I could also showcase my work. Where I could put lists of my favorite movies and games and programming languages. Where I could document my progress on projects and feature demos. I wanted to be able to write series of articles, such that each article knew which article came before and after. I wanted to be able to associate posts with projects so they’d show up as news items on each project page. But I also wanted a central page that would aggregate all these different kinds of posts in a single index, sortable, and filterable by tag.

Essentially, I wanted to be able to take whatever data I cared to create, transform it in arbitrary ways, and then use it to drive the generation of the site. You know, without having to ingest it all first into GraphQL or whatever.

Rolling Your Own SSG

Rolling your own static site generator is basically a meme, so I must not have been the only one unsatisfied. I have made two serious attempts at creating my own SSG, both centered around this idea of “programmable content pipelines”.

You define some content types, say where the data was coming from, how it was transformed, then finally, how it was rendered out.

How data was sourced and transformed should be arbitrary and up to the user. Markdown content could be enriched with frontmatter metadata. CSS could be minified. Content could be aggregated into tags. Different content items could be made to reference each other. The point was it ought to be open-ended and programmable. You could produce whatever resulting dataset was appropriate for generating the site you had in mind.

My first attempt was called Blot and was implemented in Python. You’d define your pipelines as simple data types composing reusable transformation functions:

content_types = dict(
    posts={
        'loader': BasicLoader('content/posts', extensions=['md']),
        'processors': [
            YAMLMetadata(),
            # add metadata derived from source path
            PathMetadata(),
            # create a slug from the `title` property
            Slugifier('title'),
            # humanize the `date` property
            Humanizer('date'),
            # aggregate assets across their `tags` property
            Tags(),
        ],
        'finalizers': [
            JinjaRenderer(),
            MarkdownParser(),
            AutoSummary(),
        ],
    },
    sass={
        'loader': SassLoader('theme2/scss/site.scss',
                             compress=False,
                             paths=['/normalize/sass',
                                    '/typey/stylesheets',
                                    '/font-awesome/scss']),
        'finalizers': [CssMinify()],
    },
    # and so on..

Transformation functions could modify content metadata, or even create new whole content types, like in the case of tag aggregation. Then you’d specify how the content types and entries were to be rendered out.

My old Blot-based site was pretty great. Everything was covered by these pipelines, from the ol’ blog post, to optimized images, minified JS/CSS, to JSON lists of my favorite movies and so on.

The Age of React

Eventually however the age of React arrived and Jinja templates suddenly felt very limiting. I was using React everywhere else and I wanted to be able to enjoy the nice ecosystem React was growing.

My second SSG was called Flapper and was built ontop of Gatsby. Gatsby is a nice SSG, but for smaller use-cases, the GraphQL API is quite tedious to use. Using the Gatsby plugin system I was able to completely replace its data API with my own. It again emphasized simple content pipelines:

const pipelines = {
    'page': [
        SourceFilesRecursively('pages/', ['.tsx']),
        SetUrl('{{relativePath}}'),
        RenderTemplate('{{path}}'),
    ],
    'post': [
        SourceMatchedFiles('posts/{{category_name}}/{{name}}.mdx'),
        SetUrl('posts/{{name}}'),
        BindMdx('{{path}}'),
        RenderTemplate('src/templates/MDXPost.tsx')
    ],
}

I think Blot and Flapper are pretty cool. They’re not just html generators. They really are programmable data pipeline tools that can be used for anything. With them I was able to build the kind of site I was after, with nice approachable APIs.

That said, I was never looking to be in the SSG business, I just wanted to build my site.

The Promise Land

I have found the ideal SSG in Astro.

There are a lot of neat things about Astro but there are two features that are key:

  • No nonsense data API
  • Multistage components
  • UI framework agnostic

Content Collections

Astro has a simple data API where you define collections by name and type by defining a Zod schema:

const posts = defineCollection({
  type: "content",
  schema: z.object({
    title: z.string(),
    publishedAt: z.date().optional(),
    description: z.string(),
    tags: z.array(z.string()).optional(),
  }),
})

const movies = defineCollection({
  type: 'data',
  schema: z.object({
    Title: z.string(),
    Year: z.string(),
    Director: z.string(),
    Actors: z.string(),
    Plot: z.string(),
  })
})

Collections of type content are things like markdown-with-frontmatter, and collections of type data are things like JSON files. Once defined, your data becomes easily accessible in your Astro multi-stage components.

Multistage Components

Astro takes the typical approach of having a folder where the internal files and directories represent the routes of your site. However, it offers it’s own .astro file format which has two parts separated by three dashes:

// pages/projects/index.astro
---
import { getCollection } from "astro:content";
import Layout from "@/layouts/Layout.astro";
import ProjectIndex from "@/qwik/ProjectIndex";

const projects = await getCollection('projects')
const activeProjects = projects.map(p => p.data.active)
---

<Layout title="All projects">
    <ProjectIndex projects={activeProjects} />
</Layout>

At the top, you have build-time logic where you can get at your data and transform it however necessary. At the bottom, you have JSX where you can use that data to drive rendering. It’s that simple! If you find you’re doing the same kinds of transformations in multiple pages, then just refactor it into a helper function.

UI Agnosticism

A neat thing about Astro is that while its page layouts start rooted with .astro code, you are free to utilize the components of most major UI frameworks. For example, with @astrojs/react you can straight up import and use React components. In the example above, <ProjectIndex /> is a Qwik component. You can even utilize multiple UI frameworks if desired.

Here’s a ThreeJS based component I built with Qwik:

There are some caveats however:

  • Framework components can’t access the Astro data layer. You need to gather the data on the Astro side then pass it into framework components as props.
  • You can’t use Astro or other framework components within a given framework’s components. So your React components can’t compose Astro components, or Vue components, etc.

These make total technical sense however, and don’t represent any significant hangup.

This site uses Astro exclusively for getting data ready, which is then passed into some second-level Qwik component. It’s all Qwik from there on.

What’s Qwik?

Qwik is a React-like UI framework that eschews component hydration, introducing a new approach called “resumability”.

Imagine a React component, with hook use, event handlers, etc. Imagine all the closures involved were made async and stored as tiny indpendent JS chunks on the server. Instead of downloading all the code for the site at once, Qwik loads the JS closure-by-closure, as they’re needed! For initial top-level closures, Qwik serializes them as attributes on the HTML. This results in essentially instantaneous page loads.

Fireship has a great video on it: https://www.youtube.com/watch?v=x2eF3YLiNhY

Conclusion

Astro’s solution to SSG is one I believe will stand for a long time. It takes care of data ingestion itself, but then hands off rendering to the framework of your choice. Even if better UI frameworks come along to replace React and Qwik, that presents no problem besides a refactor of your view code.

All Astro has to do is keep improving their content management. And they’re doing just that, with a second version of their content API in the works.

Currently content collections must exist in src/content/{collection-name}/. The new version will offer a more flexible glob-based selection API allowing each collection to gather its content from arbitrary locations. This will allow for the colocation of related files. For example, all the differently typed content related to a given project could live under content/projects/{project-name}/. Once that’s in, I will legitimately have no more asks when it comes to what I want from an SSG.