Why Monorepo?

A monorepo is an approach to source control management where a single repository comprises many cohesive projects or packages. Whilst this has its own setup caveats, some benefits include:

  • Easier code sharing and dependency management as all code and assets for an application are co-located so local dependencies between packages can be maintained without the need for explicit publishing and versioning.
  • Allows atomic commits that involve changes to several packages if needed (or for those changes to be rolled back), enabling easier refactoring.
  • Encourages collaboration and visibility where changes can be more easily tracked and understood in one place across the entire application.

Tools Used

Projen

Projen is a tool for managing project configuration through code. It can be used to quickly bootstrap and maintain the setup of various project types through pre-baked and customisable configurations.

Projen provides many common project templates that can be used for React apps, AWS CDK apps, multi language libraries (jsii) etc. These templates can be customised and shared to standardise the setup of projects and associated tooling across an organisation.

Pnpm

Pnpm is a package manager for node similar to npm and yarn but is known for being more strict and faster due to the way it efficiently stores node_modules on disk. It also has built-in support for monorepos.

Nx

Nx is a flexible build system that can be used to execute targets against packages in a monorepo. It understands both package and target dependencies so it knows what to run against which package in the correct order. It utilises caching to optimise execution time and avoid running the same target twice (provided the inputs haven’t changed).

In a CI environment (e.g. github actions), Nx can detect and run targets against the affected packages in a commit.

Walk-through

Overview

The walkthrough will setup a monorepo for a backend comprising:

  • shared-lib
    (which exports a greeting function)

  • service-a (cdk app)
    (which has a lambda that uses the greeting function to return a specific message)

  • service-b (cdk-app)
    (which has a lambda that uses the greeting function to return a specific message)

Firstly, using projen we’ll bootstrap the three packages in our repo and integrate pnpm. Then we’ll add nx and explore how we can use this to execute targets in each package. Finally we’ll create a basic ci/cd pipeline for building and deploying the app via github actions workflow.

Initialise repository

The following initialises a new monorepo under the my-app directory using nvm to install the version of node and corepack to install the version of pnpm.

  $ mkdir my-app
  $ cd my-app
  $ git init
  $ echo “18.16.0” > .nvmrc
  $ nvm use
  $ corepack enable
  $ corepack prepare pnpm@8.6.0 –activate

Bootstrap with Projen

Given the monorepo will comprise of three packages, we can use projen templates and the subproject feature to perform bootstrap of the root as well as the individual packages.

1. Initialise the root projen project. 

projen@latest new typescript \
  –name “@my-app/root” \
  –projenrc-ts \
  –packageManager “pnpm” \
  –projenCommand “pnpm dlx projen” \
  –minNodeVersion “18.16.0” \
  –sampleCode false \
  –licensed false

2. The projenrc.ts at the root represents the entry point for configuring projen templates. Replace the contents of projenrc.ts with the following:

import { javascript, typescript, awscdk } from ‘projen’;
import { VscodeSettings } from ‘./projenrc/vscode’;
import { PnpmWorkspace } from ‘./projenrc/pnpm’;

const defaultReleaseBranch = ‘main’;
const cdkVersion = ‘2.61.1’;
const nodeVersion = ‘18.16.0’;
const pnpmVersion = ‘8.6.0’;

// Defines the root project that will contain other subprojects packages
const root = new typescript.TypeScriptProject({
  name: ‘@my-app/root’,
  defaultReleaseBranch,
  packageManager: javascript.NodePackageManager.PNPM,
  projenCommand: ‘pnpm dlx projen’,
  minNodeVersion: nodeVersion,
  projenrcTs: true,
  sampleCode: false,
  licensed: false,

  // Jest and eslint are disabled at the root as they will be
  // configured by each subproject. Using a single jest/eslint
  // config at the root is out of scope for this walkthrough
  eslint: false,
  jest: false,

  // Disable default github actions workflows generated
  // by projen as we will generate our own later (that uses nx)
  depsUpgradeOptions: { workflow: false },
  buildWorkflow: false,
  release: false,
});

// Defines the subproject for shared lib
new typescript.TypeScriptProject({
  parent: root,
  name: ‘@my-app/shared-lib’,
  outdir: ‘./packages/shared-lib’,
  defaultReleaseBranch,
  sampleCode: false,
  licensed: false,
   
  // Use same settings from root project
  packageManager: root.package.packageManager,
  projenCommand: root.projenCommand,
  minNodeVersion: root.minNodeVersion,
})

// Defines the subproject for ‘service-a’
new awscdk.AwsCdkTypeScriptApp({
  parent: root,
  name: ‘@my-app/service-a’,
  outdir: ‘./packages/service-a’,
  cdkVersion,
  defaultReleaseBranch,
  sampleCode: false,
  licensed: false,
  requireApproval: awscdk.ApprovalLevel.NEVER,
   
  // Use same settings from root project
  packageManager: root.package.packageManager,
  projenCommand: root.projenCommand,
  minNodeVersion: root.minNodeVersion,
})

// Defines the subproject for ‘service-b’
new awscdk.AwsCdkTypeScriptApp({
  parent: root,
  name: ‘@my-app/service-b’,
  outdir: ‘./packages/service-b’,
  cdkVersion,
  defaultReleaseBranch,
  sampleCode: false,
  licensed: false,
  requireApproval: awscdk.ApprovalLevel.NEVER,
   
  // Use same settings from root project
  packageManager: root.package.packageManager,
  projenCommand: root.projenCommand,
  minNodeVersion: root.minNodeVersion,
})

root.package.addField(‘packageManager’, `pnpm@${pnpmVersion}`);
root.npmrc.addConfig(‘auto-install-peers’, ‘true’);

new PnpmWorkspace(root);
new VscodeSettings(root);

root.synth(); // Synthesize all projects

3. Add the following under projenrc/pnpm.ts

import path from ‘path’;
import { Component, Project, YamlFile } from ‘projen’;

// Custom projen component that generates pnpm-workspace.yaml

// and defines the monorepo packages based on the subprojects.


export class PnpmWorkspace extends Component {
  constructor(rootProject: Project) {
    super(rootProject);

    new YamlFile(rootProject, ‘pnpm-workspace.yaml’, {
      obj: {
        packages: rootProject.subprojects.map(
          project => path.relative(
            rootProject.outdir, project.outdir
          )
        ),
      },
    });
  }
}

4. Add the following under projenrc/vscode.ts (if you use vscode)

import path from ‘path’;
import { Component, JsonFile, Project } from ‘projen’;

// Custom projen component that generates vscode settings
// and defines workspace directories for eslint. This allows
// the IDE to use the correct eslint config for the subproject.


export class VscodeSettings extends Component {
  constructor(rootProject: Project) {
    super(project);

    new JsonFile(rootProject, ‘.vscode/settings.json’, {
      obj: {
        ‘eslint.workingDirectories’:
          rootProject.subprojects.map(project => ({
            pattern: path.relative(
              rootProject.outdir, project.outdir
            ),
          })),
      },
    });
  }
}

5. Run pnpm projen at the root. This will synthesise all files and scaffold the three subprojects pre-configured with eslint, jest, typescript etc.

6. Update projenrc.ts at the root by adding a npm dependency to shared-lib from service-a and service-b. Because this is a workspace dependency, a symlink will be created to shared-lib in each consuming project’s node_modules directory.

new awscdk.AwsCdkTypeScriptApp({
  name: ‘@my-app/service-a’,
  deps: [‘@my-app/shared-lib@workspace:*’],
  …
});

new awscdk.AwsCdkTypeScriptApp({
  name: ‘@my-app/service-b’,
  deps: [‘@my-app/shared-lib@workspace:*’],
  …
});

7. Run pnpm projen at the root again to re-synth. At this point we now have a basic setup of our monorepo and can start adding business logic to our subprojects / packages.

Add business logic

shared-lib

1. Define a shared function that returns a greeting. Please note, though this walkthrough doesn’t implement any tests, Jest is configured out of the box and test can be added by adding *.test.ts files. These tests will automatically run as part of the build script.

// shared-lib/src/index.ts

export const sayHello = () => {
  return ‘Hello, there!’;
};

2. Build the shared-lib package

$ cd packages/shared-lib
$ pnpm build

service-a

1. Define a lambda in the service-a cdk subproject that uses the function from shared-lib and returns a message.
Please ensure you update
YOUR_AWS_ACCOUNT and YOUR_AWS_REGION

// service-a/src/main.ts

import { App, Stack, StackProps, Tags } from ‘aws-cdk-lib’;
import { NodejsFunction } from ‘aws-cdk-lib/aws-lambda-nodejs’;
import { Construct } from ‘constructs’;

export class ServiceAStack extends Stack {
  constructor(scope:Construct, id:string, props:StackProps) {
    super(scope, id, props);

    const lambda = new NodejsFunction(this, ‘greeting’, {
      entry: ‘src/handler.ts’,
    });
    Tags.of(lambda).add(‘Name’, ‘service-a-greeting’);
  }
}

const app = new App();
new ServiceAStack(app, ‘service-a’, {
  env: {
    account: ‘YOUR_AWS_ACCOUNT’,
    region: ‘YOUR_AWS_REGION’,
  },
});
app.synth();

// service-a/src/handler.ts

import { sayHello } from ‘@my-app/shared-lib’;

export const handler = async () => {
  return `${sayHello()} This is service a`;
};

2. Build the service-a package. Please ensure you have already run cdk bootstrap in the desired AWS account and logged in, prior to deploying.

cd packages/service-a
$ pnpm build

service-b

1. Define a lambda in the service-b cdk subproject that uses the shared function and returns a message.
Please ensure you update
YOUR_AWS_ACCOUNT and YOUR_AWS_REGION.

// service-b/src/main.ts

import { App, Stack, StackProps, Tags } from ‘aws-cdk-lib’;
import { NodejsFunction} from ‘aws-cdk-lib/aws-lambda-nodejs’;
import { Construct } from ‘constructs’;

export class ServiceBStack extends Stack {
  constructor(scope:Construct, id:string, props:StackProps) {
    super(scope, id, props);

    const lambda = new NodejsFunction(this, ‘greeting’, {
      entry: ‘src/handler.ts’,
    });
    Tags.of(lambda).add(‘Name’, ‘service-b-greeting’);
  }
}

const app = new App();
new ServiceBStack(app, ‘service-b’, {
  env: {
    account: ‘YOUR_AWS_ACCOUNT’,
    region: ‘YOUR_AWS_REGION’,
  },
});
app.synth();

// service-b/src/handler.ts

import { sayHello } from ‘@my-app/shared-lib’;

export const handler = async () => {
  return `${sayHello()} This is service b`;
};

Build and deploy the service-b package. Please ensure you have already run cdk bootstrap in the desired AWS account and logged in, prior to deploying.

$ cd packages/service-b
$ pnpm build

Integrate Nx

Nx has first class support for pnpm and uses this to auto discover packages and their dependencies in the monorepo. Whilst it might appear like pnpm and nx do similar things with respect to task execution, nx is much more powerful and utilises a cache to optimise execution. Each package will need to define targets which represent tasks that can be executed against the package. These targets can either be defined in the script section of the package.json file or project.json file for the subproject. Nx combines these sources when searching for targets to execute.

Nx can be configured at the root level for project defaults via nx.json, and also individually at the subproject / package level (via the package.json or project.json). The scope of this walkthrough only configures nx at the root level using the default runner.

We need to update projenrc.ts so that we add dependencies to nx libraries and generate a nx.json file at the root level.

1. Add the following under projenrc/nx.ts

import { Component, JsonFile, typescript } from ‘projen’;

// Custom projen component that configures nx.


export class Nx extends Component {
  constructor(rootProject: typescript.TypeScriptProject) {
    super(rootProject);

    // Add nx library dependencies
    rootProject.addDevDeps(
      ‘nx@^15’,
      ‘@nrwl/devkit@^15’,
      ‘@nrwl/workspace@^15’
    );

    // Add nx.json file
    new JsonFile(rootProject, ‘nx.json’, {
      obj: {
        extends: ‘nx/presets/npm.json’,
        tasksRunnerOptions: {
          default: {
            runner: ‘@nrwl/workspace/tasks-runners/default’,
            options: {

              // By default nx uses a local cache to prevent re-running targets

              // that have not had their inputs changed (eg. no changes to source files).
              // The following specifies what targets are cacheable.
              cacheableOperations: [‘build’]
            },
          },
        },
        targetDefaults: {
          build: {
         
            // Specifies the build target of a project is dependent on the
            // build target of dependant projects (via the caret)
            dependsOn: [‘^build’],
           
            // Inputs tell nx which files can invalidate the cache should they updated.
            // We only want the build target cache to be invalidated if there
            // are changes to source files so the config below ignores output files.
            inputs: [
              ‘!{projectRoot}/test-reports/**/*’,
              ‘!{projectRoot}/coverage/**/*’,
              ‘!{projectRoot}/build/**/*’,
              ‘!{projectRoot}/dist/**/*’,
              ‘!{projectRoot}/lib/**/*’,
              ‘!{projectRoot}/cdk.out/**/*’
            ],
           
            // Outputs tell nx where artifacts can be found for caching purposes.
            // The need for this will become more obvious when we configure

            // github action workflows and need to restore the nx cache for

            // subsequent job to fetch artifacts
            outputs: [
              “{projectRoot}/dist”,
              “{projectRoot}/lib”,
              “{projectRoot}/cdk.out”
            ]
          },
          deploy: { dependsOn: [‘build’] }
        },
       
        // This is used when running ‘nx affected ….’ command to selectively
        // run targets against only those packages that have changed since
        // lastest commit on origin/main
        affected: { defaultBase: ‘origin/main’ },
      },
    });
  }
}

2. Update projenrc.ts and add the nx component to the root project.

import { Nx } from ‘./projenrc/nx’;

new Nx(root); // add nx to root
root.synth();

3. Run pnpm projen at the root to re-synth files.

4. Now that nx has been integrated, let’s run some commands. At the root, run:
pnpm nx run-many –target build

This will run the build target against all projects. Note how the shared-lib package is built before service-a and service-b due to the dependency chain.

5. Try running the command again and notice this time the cache was utilised.

Add github actions workflow

Now that we have the monorepo setup and can run targets against packages using nx, we can create a github actions workflow so that packages can be built and deployed automatically when there is a commit on the main branch.

For this walkthrough, we’ll create a simple workflow at the monorepo root that defines two jobs: build and deploy. We’ll use separate jobs to demonstrate how nx cache works across jobs. For each job we’ll use nx to determine the affected packages for a commit by examining commits since the last successful build on main. It will then execute build or deploy targets (depending on the job), against the affected packages – respecting their dependency chain. Note, the workflow doesn’t need to know specifics about how the targets are implemented as it is up to each individual package to implement (or not) the target.

For example, the following shows how the workflow would run and order of target execution if there are only changes to service-a (given it’s dependency on shared-lib).

1. Generate the workflow using projen.

Important: In order for the workflow to perform deployments to AWS, it needs to assume the correct AWS iam role before deployment. The following uses aws-actions/configure-aws-credentials action and assumes you have configured Open ID Connect in AWS to allow github workflows to access your AWS resources. Please update YOUR_RUNNER, YOUR_ASSUME_ROLE_ARN, YOUR_REGION accordingly.

If you have a different auth strategy, you will need to tweak the deploy steps and ensure you authenticate with AWS before running the deploy target.

Add the following under projenrc/workflow.ts

import { Component, github, typescript } from ‘projen’;

// Custom projen component that configures a github workflow.

export class Workflow extends Component {
  private pnpmVersion: string;

  constructor(rootProject: typescript.TypeScriptProject, options: { pnpmVersion: string }) {
    super(rootProject);
    this.pnpmVersion = options.pnpmVersion;

    const wf = new github.GithubWorkflow(
      rootProject.github!, ‘release’
    );

    const runsOn = [‘YOUR_RUNNER’];

    wf.on({ push: { branches: [‘main’] } });

    wf.addJobs({
      build: {
        name: ‘build’,
        runsOn,
        permissions: {
          contents: github.workflows.JobPermission.WRITE,
          actions: github.workflows.JobPermission.READ,
        },
        steps: [
          …this.bootstrapSteps(),
          {
            name: ‘Run build target’,
            run: ‘pnpm nx affected –target build  –verbose’,
          },
        ],
      },
      deploy: {
        name: ‘deploy’,
        needs: [‘build’],
        runsOn,
        permissions: {
          contents: github.workflows.JobPermission.WRITE,
          actions: github.workflows.JobPermission.READ,
        },
        steps: [
          …this.bootstrapSteps(),
          {
            name: ‘Configure AWS Credentials’,
            id: ‘configure_iam_credentials’,
            uses: ‘aws-actions/configure-aws-credentials@v1’,
            with: {
              ‘role-to-assume’: ‘YOUR_ASSUME_ROLE_ARN’,
              ‘role-duration-seconds’: 3600,
              ‘aws-region’: ‘YOUR_REGION’
            },
          },
          {
            name: ‘Run deploy target’,
            run: ‘pnpm nx affected –target deploy  –verbose’,
          },
        ],
      }
    })
  }

  private bootstrapSteps(): github.workflows.JobStep[] {
    const project = this.project as typescript.TypeScriptProject;
    return [
      {
        name: ‘Checkout’,
        uses: ‘actions/checkout@v3’,
        with: { ‘fetch-depth’: 0 },
      },
      {
        name: ‘Install pnpm’,
        uses: ‘pnpm/action-setup@v2.2.1’,
        with: { version: this.pnpmVersion },
      },
      {
        name: ‘Setup node’,
        uses: ‘actions/setup-node@v3’,
        with: {
          ‘node-version’: project.minNodeVersion,
          cache: ‘pnpm’,
        },
      },
     
      // Ensures the nx cache for the current commit sha is restored

      // before running any subsequent commands. This allows outputs

      // from any previous target executions to become available and

      // avoids re-running previously cached targets unnecessarily.
      // This action also updates the cache with any changes and the

      // end of the job so that subsequent jobs get the updated cache.
      {
        name: ‘Nx cache’,
        uses: ‘actions/cache@v3,
        with: {
          path: node_modules/.cache/nx,
          ‘fail-on-cache-miss: false,
          key: nx-${{ github.repository_id }}-${{ github.sha }},
        },
      },
      {
        name: Install dependencies,
        run: pnpm install,
      },
     
      // This determines the sha of the last successful build on the main branch
      // (known as the base sha) and adds to env vars along with the current (head) sha.
      // The commits between the base and head sha’s is used by subesquent ‘nx affected’

      // commands to determine what packages have changed so targets only run
      // against those packages.
      {
        name: ‘Derive SHAs for nx affected commands’,
        uses: ‘nrwl/nx-set-shas@v2,
        with: { main-branch-name: main },
      },
    ]
  }
}

2. Update projenrc.ts and add the workflow component to the root project.

import { Workflow } from ‘./projenrc/workflow’;

new Workflow(root, { pnpmVersion });
root.synth();

3. Run pnpm projen at the root to re-synth files.

Commit and execute workflow

1. From the root, commit and push all changes to your monorepo.

$ git add .
$ git commit -m‘feat(Monorepo): initial setup’
$ git push origin main

2. In the github actions interface, notice how nx detects changes in all three packages and executes the build targets in the correct dependency order.

After the build job succeeds, the deploy job is executed. Note how the nx cache is restored from the previous job so build artefacts are available to the deploy job.

When determining the deploy targets, nx only detects service-a and service-b as shared-lib does not define this target.

Given the deploy target depends on the build target (for artefacts), nx attempts to run the build target first (including those of dependent packages), but finds results and artefacts in its cache so skips these steps.

Nx then successfully runs the deploy target against service-a and service-b.

Executing the lambda from each service yields the correct result.

Power of nx affected

Our previous commit was an initial commit and resulted in all packages being built and deployed. However if we only commit changes to service-a, then we can observe how the workflow will only build and deploy service-a (and dependencies).

1. Update the greeting of service-a in packages/service-a/src/handler.ts

import { sayHello } from ‘@my-app/shared-lib’;

export const handler = async () => {
  return `${sayHello()} This is service a!!`; // Add !! marks at the end
};

2. Commit the changes.

$ git add .
$ git commit -m‘feat(ServiceA): update greeting’
$ git push origin main

3. In github actions, observe the workflow that was triggered and note only service-a and its dependency (shared-lib) are built and deployed.

Executing the service-a lambda returns the updated greeting with the exclamation marks added to the end.

Summary

This walkthrough discussed one approach to setting up a monorepo using projen to bootstrap and maintain configuration for packages. Tools such as pnpm and nx were then integrated to manage package dependencies and easily execute targets against multiple packages. Finally a github workflow was added to provide a basic ci/cd pipeline that triggers when a commit is pushed that only builds and deploys changed packages since the last successful build.

The full source can be found here.