Ever found yourself staring at a complex JSONata transformation buried deep inside an AWS Step Functions definition, wondering how to properly test it without spinning up the entire AWS infrastructure? Yeah, me too. That’s exactly the problem I set out to solve with this technical spike on unit testing JSONata in AWS Step Functions.
TL;DR
Embedding complex blocks of JSONata directly inside AWS Step Functions makes testing slow, complex, and frustrating. By moving JSONata templates into separate files, introducing a lightweight template manager, and running unit tests locally, you can validate transformations while you develop – no AWS deployments required.
Table of Contents
The Problem – JSONata Templates Embedded in Step Function Definition Files
AWS Step Functions recently added support for JSONata as a query language, which is fantastic for data transformations. But there’s a catch – when your JSONata templates are embedded directly in your Step Function definitions, testing them becomes a nightmare:
- No isolation: You can’t test the transformation logic without running the entire Step Function
- Slow feedback loops: Every test requires AWS infrastructure
- Debugging hell: When something breaks, good luck figuring out if it’s your JSONata or your Step Function logic
- No IDE support: JSONata embedded in JSON strings means no syntax highlighting or validation
The Solution – Separate, Test, then Integrate
The approach I developed separates JSONata templates into their own files, allows for comprehensive unit testing, and then seamlessly integrates them back into Step Functions during deployment. Here’s how it works:
Step 1: Make JSONata Templates First-Class for Easier Unit Testing
Instead of embedding JSONata directly in Step Function definitions, I created separate .jsonata files:
// src/transforms/customer-transform.jsonata
{
"customerId": $source.customer.id,
"fullName": $source.customer.firstName & " " & $source.customer.lastName,
"email": $source.customer.email,
"address": {
"street": $source.customer.address.street,
"city": $source.customer.address.city,
"state": $source.customer.address.state,
"zipCode": $source.customer.address.zipCode,
"country": $source.customer.address.country ? $source.customer.address.country : "USA"
},
"preferences": $source.customer.preferences[].{
"category": category,
"enabled": enabled,
"priority": priority ? priority : 10
},
"filter": "customerId eq '" & $source.customer.id & "'"
}
When used together with your JSONata IDE plugin of choice, this gives us proper syntax highlighting, validation, and the ability to version control our transformations independently.
Step 2: Template Management for JSONata in AWS Step Functions
I built a simple but effective template management system that handles loading and caching:
// src/transforms/index.ts
export class JsonataTemplateManager {
private static templates: Map<string, string> = new Map();
static loadTemplate(templateName: string): string {
if (this.templates.has(templateName)) {
return this.templates.get(templateName)!;
}
const templatePath = path.join(__dirname, `${templateName}.jsonata`);
if (!fs.existsSync(templatePath)) {
throw new Error(`JSONata template not found: ${templatePath}`);
}
const template = fs.readFileSync(templatePath, 'utf-8');
this.templates.set(templateName, template);
return template;
}
static getAvailableTemplates(): string[] {
const templatesDir = __dirname;
return fs.readdirSync(templatesDir)
.filter(file => file.endsWith('.jsonata'))
.map(file => path.basename(file, '.jsonata'));
}
}
// Convenience exports
export const getCustomerTransform = () => JsonataTemplateManager.loadTemplate('customer-transform');This approach provides caching, validation, and a clean API for accessing templates.
Step 3: Unit Testing JSONata in AWS Step Functions Locally
Here’s where the magic happens. With templates as separate files, we can write proper unit tests:
// test/transforms/customer-transform.test.ts
import jsonata from "jsonata";
import { getCustomerTransform } from "../../src/transforms";
describe("Customer Transform", () => {
let template: string;
beforeAll(() => {
template = getCustomerTransform();
});
it("should transform a complete customer object", async () => {
const input = {
customer: {
id: "cust-123",
firstName: "John",
lastName: "Doe",
email: "john.doe@example.com",
address: {
street: "123 Main St",
city: "Anytown",
state: "CA",
zipCode: "12345",
country: "USA",
},
preferences: [
{ category: "email", enabled: true, priority: 1 },
{ category: "sms", enabled: false, priority: 2 },
],
},
};
const expression = jsonata(template);
const result = await expression.evaluate({}, {
source: input, // Consistent variable naming
});
expect(result.customerId).toBe("cust-123");
expect(result.fullName).toBe("John Doe");
expect(result.address.country).toBe("USA");
expect(Array.from(result.preferences)).toEqual([
{ category: "email", enabled: true, priority: 1 },
{ category: "sms", enabled: false, priority: 2 },
]);
});
it("should handle missing country with default", async () => {
const input = {
customer: {
id: "cust-456",
firstName: "Jane",
lastName: "Smith",
email: "jane.smith@example.com",
address: {
street: "456 Oak Ave",
city: "Somewhere",
state: "NY",
zipCode: "67890",
// country is missing
},
preferences: [],
},
};
const expression = jsonata(template);
const result = await expression.evaluate({}, { source: input });
expect(result.address.country).toBe("USA"); // Default applied
});
});The key insight here is using a consistent `$source` variable in templates instead of relying on `$states.input`. This makes templates more testable and reusable. To support workflow variables, you can also pass them into the `expression.evaluate() function as additional bindings:
const result = await expression.evaluate({}, {
myVariable: "some value",
anotherVariable: { field: "value" },
source: input
}); Step 4: Integrating Tested JSONata Templates Back into Step Functions
The final piece is seamlessly integrating these templates back into the Step Function definition file. I use placeholder syntax in the Step Function definition:
{
"Comment": "A description of my state machine",
"StartAt": "Init variables",
"States": {
"Init variables": {
"Type": "Pass",
"Next": "Transform into Canonical",
"Assign": {
"source": "{% $states.input %}"
}
},
"Transform into Canonical": {
"Type": "Pass",
"End": true,
"Output": "{% {{CUSTOMER_TRANSFORM_TEMPLATE}} %}"
}
},
"QueryLanguage": "JSONata"
}Then during CDK deployment, I substitute the placeholders with actual templates:
// src/cdk/step-function-stack.ts
private substituteTemplates(definition: string, templates: Record<string, string>): string {
let processedDefinition = definition;
Object.entries(templates).forEach(([templateName, templateContent]) => {
const placeholder = `{{${templateName.toUpperCase().replace(/-/g, '_')}_TEMPLATE}}`;
// Escape the JSONata template for JSON embedding
const escapedTemplate = JSON.stringify(templateContent);
const embeddedTemplate = escapedTemplate.slice(1, -1);
processedDefinition = processedDefinition.replace(
new RegExp(placeholder, 'g'),
embeddedTemplate
);
});
// Validate that all placeholders were replaced
const remainingPlaceholders = processedDefinition.match(/\{\{[^}]+\}\}/g);
if (remainingPlaceholders) {
throw new Error(
`Unresolved template placeholders: ${remainingPlaceholders.join(', ')}`
);
}
return processedDefinition;
}The Results: Fast, Reliable, Maintainable
This approach delivers several key benefits:
Lightning-Fast Test Feedback
$ npm test
PASS test/transforms/customer-transform.test.ts
Customer Transform
Basic transformation
✓ should transform a complete customer object (4 ms)
✓ should handle missing country with default (1 ms)
✓ should handle missing priority with default (1 ms)
Edge cases
✓ should handle empty preferences array (1 ms)
✓ should handle null values gracefully (1 ms)
Validation
✓ should handle missing customer object (1 ms)
✓ should handle missing required fields (1 ms)
Test Suites: 1 passed, 1 total
Tests: 7 passed, 7 total
Time: 0.892 s Tests run in milliseconds, not minutes. No AWS infrastructure deployment required.
Comprehensive Edge Case Testing
The framework makes it easy to test edge cases that would be painful to reproduce in a full Step Function:
it("should handle null values gracefully", async () => {
const input = {
customer: {
id: "cust-null",
firstName: "Null",
lastName: null, // Testing null handling
email: "null@example.com",
preferences: null,
},
};
const expression = jsonata(template);
const result = await expression.evaluate({}, { source: input });
expect(result.fullName).toBe("Null null"); // JSONata handles nulls gracefully
});Integration Testing with CDK
The framework also includes integration tests that validate the CDK deployment process:
// test/integration/step-function.test.ts
describe('Step Function Integration Tests', () => {
it('should create stack without errors', () => {
const app = new cdk.App();
const stack = new StepFunctionStack(app, 'TestStack', {
environment: 'test',
});
expect(stack).toBeDefined();
expect(stack.customerProcessingStateMachine).toBeDefined();
});
it('should have valid JSON definition', () => {
const template = Template.fromStack(stack);
const stateMachines = template.findResources('AWS::StepFunctions::StateMachine');
const stateMachine = Object.values(stateMachines)[0] as any;
const definition = stateMachine.Properties.DefinitionString;
// Should be valid JSON with no unresolved placeholders
expect(() => JSON.parse(definition)).not.toThrow();
});
});Key Patterns and Best Practices for JSONata Unit Testing
Through building this framework, I discovered several important patterns:
1. Consistent Variable Naming
Always use $source in templates instead of $states.input. This makes templates more testable and allows you to control the input structure in tests.
2. Template Placeholder Convention
Use a consistent naming convention for placeholders: {{TEMPLATE_NAME_TEMPLATE}}. This makes it easy to identify and replace them programmatically.
3. Proper JSON Escaping
When embedding JSONata in JSON strings, proper escaping is crucial:
const escapedTemplate = JSON.stringify(templateContent);
const embeddedTemplate = escapedTemplate.slice(1, -1); // Remove outer quotes 4. Build-Time Validation
Validate templates during the build process to catch syntax errors early:
private validateJsonataTemplates(): void {
const templates = getAllTemplates();
const jsonata = require('jsonata');
Object.entries(templates).forEach(([name, template]) => {
try {
jsonata(template);
console.log(`✓ JSONata template '${name}' is valid`);
} catch (error) {
throw new Error(`Invalid JSONata template '${name}': ${error.message}`);
}
});
}What’s Next?
This spike proves that separating JSONata templates from Step Functions is not only possible but highly beneficial. The next steps would be:
- Template Versioning: Add support for template versioning and migration
- Performance Testing: Add performance benchmarks for complex transformations
- IDE Integration: Create VS Code extensions for JSONata syntax highlighting and validation
- Template Library: Build a shared library of common transformation patterns
Interested in testing beyond transformations? Cevo’s related blog “How to Test Step Functions State Machine Locally” walks you through using Docker and Step Functions Local for isolated workflow testing.
Takeaway: Why Unit Testing JSONata in AWS Step Functions Matters
Testing JSONata transformations doesn’t have to be painful. Treat your JSONata templates as first-class citizens, not buried strings inside Step Functions. Separating, testing, and validating them independently makes your transformations faster to debug, easier to maintain, and safer to run in production.
The complete code for this spike is available in this repository, including all the patterns, tests, and CDK integration code.
Happy transforming!
Scott has been working with public cloud platforms and technologies for over a decade, assisting customers with cloud adoption, workload migration, application modernisation, continuous improvement, cloud governance and specialised cloud workloads such as Generative AI.
