In this blog, we will walk through an approach for setting up a basic Typescript-based monorepo. We will use projen to manage package config, and supporting tools such as nx, pnpm and github actions. A basic level of understanding is assumed for these tools.
Table of Contents
Why Monorepo?
A monorepo is an approach to source control management where a single repository comprises many cohesive projects or packages. Whilst this has its own setup caveats, some benefits include:
- Easier code sharing and dependency management as all code and assets for an application are co-located so local dependencies between packages can be maintained without the need for explicit publishing and versioning.
- Allows atomic commits that involve changes to several packages if needed (or for those changes to be rolled back), enabling easier refactoring.
- Encourages collaboration and visibility where changes can be more easily tracked and understood in one place across the entire application.
Tools Used
Projen
Projen is a tool for managing project configuration through code. It can be used to quickly bootstrap and maintain the setup of various project types through pre-baked and customisable configurations.
Projen provides many common project templates that can be used for React apps, AWS CDK apps, multi language libraries (jsii) etc. These templates can be customised and shared to standardise the setup of projects and associated tooling across an organisation.
Pnpm
Pnpm is a package manager for node similar to npm and yarn but is known for being more strict and faster due to the way it efficiently stores node_modules on disk. It also has built-in support for monorepos.
Nx
Nx is a flexible build system that can be used to execute targets against packages in a monorepo. It understands both package and target dependencies so it knows what to run against which package in the correct order. It utilises caching to optimise execution time and avoid running the same target twice (provided the inputs haven’t changed).
In a CI environment (e.g. github actions), Nx can detect and run targets against the affected packages in a commit.
Walk-through
Overview
The walkthrough will setup a monorepo for a backend comprising:
shared-lib
(which exports a greeting function)service-a (cdk app)
(which has a lambda that uses the greeting function to return a specific message)service-b (cdk-app)
(which has a lambda that uses the greeting function to return a specific message)
Firstly, using projen we’ll bootstrap the three packages in our repo and integrate pnpm. Then we’ll add nx and explore how we can use this to execute targets in each package. Finally we’ll create a basic ci/cd pipeline for building and deploying the app via github actions workflow.
Initialise repository
Bootstrap with Projen
Given the monorepo will comprise of three packages, we can use projen templates and the subproject feature to perform bootstrap of the root as well as the individual packages.
1. Initialise the root projen project.
projen@latest new typescript \ |
2. The projenrc.ts at the root represents the entry point for configuring projen templates. Replace the contents of projenrc.ts with the following:
import { javascript, typescript, awscdk } from ‘projen’; |
3. Add the following under projenrc/pnpm.ts
import path from ‘path’; // and defines the monorepo packages based on the subprojects.
|
4. Add the following under projenrc/vscode.ts (if you use vscode)
import path from ‘path’;
|
5. Run pnpm projen at the root. This will synthesise all files and scaffold the three subprojects pre-configured with eslint, jest, typescript etc.
6. Update projenrc.ts at the root by adding a npm dependency to shared-lib from service-a and service-b. Because this is a workspace dependency, a symlink will be created to shared-lib in each consuming project’s node_modules directory.
new awscdk.AwsCdkTypeScriptApp({ |
7. Run pnpm projen at the root again to re-synth. At this point we now have a basic setup of our monorepo and can start adding business logic to our subprojects / packages.
Add business logic
shared-lib
1. Define a shared function that returns a greeting. Please note, though this walkthrough doesn’t implement any tests, Jest is configured out of the box and test can be added by adding *.test.ts files. These tests will automatically run as part of the build script.
// shared-lib/src/index.ts |
2. Build the shared-lib package
$ cd packages/shared-lib |
service-a
1. Define a lambda in the service-a cdk subproject that uses the function from shared-lib and returns a message.
Please ensure you update YOUR_AWS_ACCOUNT and YOUR_AWS_REGION
// service-a/src/main.ts |
// service-a/src/handler.ts |
2. Build the service-a package. Please ensure you have already run cdk bootstrap in the desired AWS account and logged in, prior to deploying.
cd packages/service-a |
service-b
1. Define a lambda in the service-b cdk subproject that uses the shared function and returns a message.
Please ensure you update YOUR_AWS_ACCOUNT and YOUR_AWS_REGION.
// service-b/src/main.ts |
// service-b/src/handler.ts |
Build and deploy the service-b package. Please ensure you have already run cdk bootstrap in the desired AWS account and logged in, prior to deploying.
$ cd packages/service-b |
Integrate Nx
Nx has first class support for pnpm and uses this to auto discover packages and their dependencies in the monorepo. Whilst it might appear like pnpm and nx do similar things with respect to task execution, nx is much more powerful and utilises a cache to optimise execution. Each package will need to define targets which represent tasks that can be executed against the package. These targets can either be defined in the script section of the package.json file or project.json file for the subproject. Nx combines these sources when searching for targets to execute.
Nx can be configured at the root level for project defaults via nx.json, and also individually at the subproject / package level (via the package.json or project.json). The scope of this walkthrough only configures nx at the root level using the default runner.
We need to update projenrc.ts so that we add dependencies to nx libraries and generate a nx.json file at the root level.
1. Add the following under projenrc/nx.ts
import { Component, JsonFile, typescript } from ‘projen’;
// that have not had their inputs changed (eg. no changes to source files). // github action workflows and need to restore the nx cache for // subsequent job to fetch artifacts |
2. Update projenrc.ts and add the nx component to the root project.
import { Nx } from ‘./projenrc/nx’; |
3. Run pnpm projen at the root to re-synth files.
4. Now that nx has been integrated, let’s run some commands. At the root, run:
pnpm nx run-many –target build
This will run the build target against all projects. Note how the shared-lib package is built before service-a and service-b due to the dependency chain.
5. Try running the command again and notice this time the cache was utilised.
Add github actions workflow
Now that we have the monorepo setup and can run targets against packages using nx, we can create a github actions workflow so that packages can be built and deployed automatically when there is a commit on the main branch.
For this walkthrough, we’ll create a simple workflow at the monorepo root that defines two jobs: build and deploy. We’ll use separate jobs to demonstrate how nx cache works across jobs. For each job we’ll use nx to determine the affected packages for a commit by examining commits since the last successful build on main. It will then execute build or deploy targets (depending on the job), against the affected packages – respecting their dependency chain. Note, the workflow doesn’t need to know specifics about how the targets are implemented as it is up to each individual package to implement (or not) the target.
For example, the following shows how the workflow would run and order of target execution if there are only changes to service-a (given it’s dependency on shared-lib).
1. Generate the workflow using projen.
Important: In order for the workflow to perform deployments to AWS, it needs to assume the correct AWS iam role before deployment. The following uses aws-actions/configure-aws-credentials action and assumes you have configured Open ID Connect in AWS to allow github workflows to access your AWS resources. Please update YOUR_RUNNER, YOUR_ASSUME_ROLE_ARN, YOUR_REGION accordingly.
If you have a different auth strategy, you will need to tweak the deploy steps and ensure you authenticate with AWS before running the deploy target.
Add the following under projenrc/workflow.ts
import { Component, github, typescript } from ‘projen’; // before running any subsequent commands. This allows outputs // from any previous target executions to become available and // avoids re-running previously cached targets unnecessarily. // end of the job so that subsequent jobs get the updated cache. |
2. Update projenrc.ts and add the workflow component to the root project.
import { Workflow } from ‘./projenrc/workflow’; |
3. Run pnpm projen at the root to re-synth files.
Commit and execute workflow
1. From the root, commit and push all changes to your monorepo.
$ git add . |
2. In the github actions interface, notice how nx detects changes in all three packages and executes the build targets in the correct dependency order.
After the build job succeeds, the deploy job is executed. Note how the nx cache is restored from the previous job so build artefacts are available to the deploy job.
When determining the deploy targets, nx only detects service-a and service-b as shared-lib does not define this target.
Given the deploy target depends on the build target (for artefacts), nx attempts to run the build target first (including those of dependent packages), but finds results and artefacts in its cache so skips these steps.
Nx then successfully runs the deploy target against service-a and service-b.
Executing the lambda from each service yields the correct result.
Power of nx affected
Our previous commit was an initial commit and resulted in all packages being built and deployed. However if we only commit changes to service-a, then we can observe how the workflow will only build and deploy service-a (and dependencies).
1. Update the greeting of service-a in packages/service-a/src/handler.ts
import { sayHello } from ‘@my-app/shared-lib’; |
2. Commit the changes.
$ git add . |
3. In github actions, observe the workflow that was triggered and note only service-a and its dependency (shared-lib) are built and deployed.
Executing the service-a lambda returns the updated greeting with the exclamation marks added to the end.
Summary
This walkthrough discussed one approach to setting up a monorepo using projen to bootstrap and maintain configuration for packages. Tools such as pnpm and nx were then integrated to manage package dependencies and easily execute targets against multiple packages. Finally a github workflow was added to provide a basic ci/cd pipeline that triggers when a commit is pushed that only builds and deploys changed packages since the last successful build.
The full source can be found here.