Compiler Implementation Guide

Requirements and best practices for building ITS compilers

Overview

This guide defines the requirements for implementing a compliant ITS compiler. Requirements are categorized using RFC 2119 keywords:

Compilation Process

Standard Compilation Flow

1. Load Template - Parse JSON and validate against base schema, enforcing processing limits
2. Load Extended Schemas - Fetch and validate type extensions
3. Build Type Registry - Merge types with precedence rules
4. Process Variables - Replace ${variable} references
5. Evaluate Conditionals - Include/exclude conditional content
6. Generate Instructions - Convert placeholders to AI instructions
7. Collect Reference Data - Gather and render the data sources referenced by included placeholders and object-valued references
8. Assemble Prompt - Combine introduction, instructions, reference data (when present), and template

1. Schema Validation

MUST Validate Template Structure

Compilers MUST validate templates against the base schema before processing:

MUST Validate Extended Schemas

When loading schemas from the extends array:

MUST Fail on Invalid Schemas

Compilation MUST fail if:

2. Schema Loading

MUST Support URL and Relative Paths

The extends array can contain:

SHOULD Report Loading Progress

For better user experience, report which schemas are being loaded.

MAY Implement Caching

Compilers may cache remote schemas to improve performance.

MAY Implement Security Controls

Optional but recommended security features include:

3. Type Resolution

MUST Follow Precedence Rules

Instruction types are resolved in this order (highest to lowest):

  1. customInstructionTypes in the template
  2. Last schema in extends array
  3. Previous schemas in reverse order
  4. First schema in extends array

MUST Apply Complete Override

When a type is defined in multiple schemas, the entire definition is replaced. No property merging occurs.

SHOULD Report Overrides

Compilers should inform users when types are overridden:

INFO: Instruction type overrides detected:
- 'summary' overridden by marketing-types.json (from company-types.json)
- 'list' overridden by customInstructionTypes (from its-standard-types-v1.json)

4. Variable Processing

MUST Support Variable Syntax

Replace ${variableName} references with variable values in:

MUST Support Variable Types

Handle all JSON-compatible types:

MUST Support Collection Functions

Array references accept a chain of collection functions after the path: concat(prop?), sum(prop?), avg(prop?), min(prop?), max(prop?) and top(n). They are chainable suffixes applied left to right (for example ${forecast.top(3).concat(day)}) and are valid in substitution only, never in conditional expressions. Aggregations require numeric values, and whole-number results render without a decimal part.

MUST Handle Undefined Variables

If a variable reference is undefined, compilers MUST fail with a clear error message. Both reference compilers fail in this case, and the shared invalid-template suite requires an error.

5. Conditional Evaluation

MUST Support Expression Operators

Operator Type Operators Example
Equality ==, != tone == 'professional'
Comparison >, <, >=, <= count > 5
Chained comparison a < b <= c 1 < number <= 10
Logical &&, ||, ! (word forms and, or, not accepted) includeA && !includeB
Membership in, not in type in ['A', 'B', 'C']

MUST Support Nested Conditionals

Conditionals can contain other conditionals in their content arrays.

SHOULD Provide Clear Expression Errors

Invalid expressions should produce helpful error messages.

6. Instruction Generation

MUST Process Placeholder Templates

For each placeholder:

  1. Look up the instruction type definition
  2. Apply config values to the template
  3. Wrap user description in the content wrapper
  4. Generate the complete instruction

MUST Handle Missing Config Properties

If a template references {property} but it's not in config:

Example Instruction Generation

Type Definition:

{
  "instructionTypes": {
    "list": {
      "template": "<<Replace this placeholder with a list using this user prompt: ([{<{description}>}]). Format requirements: Use {format} formatting with each item on a new line.>>",
      "configSchema": {
        "type": "object",
        "properties": {
          "format": {
            "type": "string",
            "enum": ["bullet_points", "numbered"],
            "default": "bullet_points"
          }
        }
      }
    }
  }
}

Placeholder in Template:

{
  "type": "placeholder",
  "instructionType": "list",
  "config": {
    "description": "List 5 fruits",
    "format": "bullet_points"
  }
}

Processing Steps:

  1. Replace {description} with "List 5 fruits"
  2. Replace {format} with "bullet_points"
  3. Wrap the description in content wrapper: ([{<List 5 fruits>}])

Generated Instruction:

<<Replace this placeholder with a list using this user prompt: ([{<List 5 fruits>}]). Format requirements: Use bullet_points formatting with each item on a new line.>>

7. Output Format

MUST Generate Four-Section Output

The compiled output must have these sections:

INTRODUCTION

[System prompt from compilerConfig]

INSTRUCTIONS

[Processing instructions from compilerConfig]

REFERENCE DATA

[Rendered data sources - only when the template uses reference data]

TEMPLATE

[Processed template content with instructions]

The REFERENCE DATA section is emitted only when any placeholder names a dataSource or any variable reference resolves to an object; templates using neither omit that section entirely.

SHOULD Preserve Template Formatting

Maintain line breaks, spacing, and structure from the original template.

8. Reference Data Sources

MUST Support the Reserved Data Source Keys

Compilers MUST support the reserved placeholder config keys dataSource (a variable name or array of variable names) and dataLimit (a positive integer).

MUST Render Reference Data Correctly

9. Processing Limits

MUST Enforce Configurable Limits

Compilers MUST enforce processing limits - template size, content element count, nesting depth, total variable count (nested values included), items per variable array, and text length - and MUST make them operator-configurable. A template can never raise the limits of the compiler processing it. The reference implementations expose these via configuration objects, CLI flags (JavaScript) and ITS_* environment variables (Python and .NET).

10. Error Handling

Error Type Required Behavior Example Message
Schema Validation MUST fail compilation "Template validation failed: Missing required property 'content'"
Schema Loading MUST fail compilation "Failed to load schema: https://example.com/types.json (404 Not Found)"
Version Mismatch MUST fail compilation "Schema version 2.0 incompatible with template version 1.0"
Undefined Variable MUST fail compilation "Undefined variable: ${productName}"
Invalid Expression MUST fail compilation "Invalid conditional expression: 'count >> 5'"
Missing Type MUST fail compilation "Unknown instruction type: 'custom_type'"

11. Optional Features

MAY Support Watch Mode

Recompile templates when source files change.

MAY Generate Multiple Output Formats

Besides the standard prompt format, compilers may generate:

MAY Provide IDE Integration

Language server protocol support for real-time validation.

Reference Implementations

ITS Compiler Python

The ITS Compiler Python provides the complete reference implementation of this specification. Use it to:

Installation:

# Core library (PyPI distribution: its-compiler)
pip install its-compiler

# Command-line tool providing its-compile
pip install its-compiler-cli

Usage:

# Compile a template
its-compile template.json --output prompt.txt

# Validate without compiling
its-compile template.json --validate-only

# Watch mode for development
its-compile template.json --watch

ITS Compiler JavaScript

ITS Compiler JavaScript provides a node-js implementation.

Installation:

npm install its-compiler-js

Usage:

# Compile a template
npx its-compile template.json --output prompt.txt

# Use variables and watch mode
npx its-compile template.json --variables vars.json --watch

# Strict security mode
npx its-compile template.json --strict --verbose

ITS Compiler .NET

ITS Compiler .NET provides a .NET 8 library with ASP.NET service and Azure Functions samples. NuGet publication is pending.

Shared Conformance Suite

The its-example-templates repository is the shared conformance and security suite: every reference compiler runs its templates, and new implementations should too.

Pseudo-code Compilation Flow

function compileTemplate(templatePath) {
  // 1. Load and validate template
  const template = loadJSON(templatePath)
  validateSchema(template, BASE_SCHEMA)
  
  // 2. Load extended schemas
  const types = {}
  for (const schemaUrl of template.extends || []) {
    const schema = loadSchema(schemaUrl)
    validateSchema(schema, TYPE_EXTENSION_SCHEMA)
    
    // Apply overrides
    Object.assign(types, schema.instructionTypes)
  }
  
  // 3. Apply custom types (highest precedence)
  Object.assign(types, template.customInstructionTypes || {})
  
  // 4. Process variables
  const processedContent = processVariables(template.content, template.variables)
  
  // 5. Evaluate conditionals
  const finalContent = evaluateConditionals(processedContent, template.variables)
  
  // 6. Generate instructions
  const compiledTemplate = generateInstructions(finalContent, types)

  // 7. Collect and render reference data from included placeholders
  //    and object-valued variable references
  const referenceData = collectReferenceData(finalContent, template.variables)

  // 8. Assemble final output (REFERENCE DATA section only when present)
  return assembleOutput(template.compilerConfig, referenceData, compiledTemplate)
}