Enterprsise Integration Teams: Take Note

North-star principles

• Domain before runtime: Organize everything around business domains (Orders, Inventory, Network, Billing), not around tools. Runtimes (Lambda, containers) are implementation details of domain capabilities.

• Events over calls: Prefer asynchronous events (Kafka/EventBridge) to hard synchronous chains. Use request/response only at the edges where humans or external systems need it.

• Canonical contracts: Publish and consume canonical events/DTOs with explicit versioning. Adapters handle source/target peculiarities; domains stay clean.

• Loose coupling, strong observability: Boundaries are async and decoupled; tracing, metrics, and DLQs stitch the system together operationally.

Core building blocks

1. Event Backbone (Kafka/EventBridge)

   
   

   • Topics per domain (e.g., orders.events, payments.commands, inventory.snapshots).

   • Schema registry + backward-compatible evolution (v1 → v2 with additive fields).

   • DLQs and retry topics are first-class; backpressure and replay are normal operations.

2. 
   

3. Domain Microservices (Containers, e.g., Spring Boot)

   
   

   • Own orchestration and core business logic (sagas, validations, state machines).

   • Expose stable APIs for upstreams/partners; publish domain events on state change.

   • Use outbox pattern to atomically persist state and emit events (prevents “lost” events).

4. 
   

5. Serverless Workers (Lambdas)

   
   

   • Event consumers for transformation, enrichment, notifications, scheduled jobs.

   • Edge adapters: file pulls, S3 ingest, webhooks, protocol/format bridging (CSV→canonical).

   • Great for spiky or low-duty workloads; cold starts don’t matter in async pipelines.

6. 
   

7. Canonical Services

   
   

   • A canonical model (shared nouns/verbs) with clear ownership and governance.

   • Inbound adapters: map source → canonical.

   • Outbound adapters: map canonical → target.

   • Keep canonical validation centralized (schemas, reference data, enumerations).

8. 
   

Flow patterns you’ll use (and when)

• Event choreography (default): Source adapter emits thing.created; interested domains react. Simple, scalable, minimal coupling.

• Orchestration (when needed): A domain service coordinates multi-step processes (e.g., order → payment → fulfillment) with saga semantics, compensations, and timeouts.

• Sync-to-async edge: APIs at the edge accept a request, immediately emit a command/event, and return 202 + correlation ID; progress continues asynchronously.

• Batch windows & snapshots: For legacy, run Lambda batch readers (SFTP/DB) that produce snapshot or delta events into the stream.

Contracts & data design

• Event types: .created|updated|deleted (facts) vs .requested|approved|rejected (process). Keep names semantic and consistent.

• IDs: Distinguish business IDs (e.g., orderId) from correlation/causation IDs (trace end-to-end).

• Schema evolution: Additive, nullable fields; never repurpose semantics. Use headers for routing metadata (tenant, PII class, priority).

• Idempotency: Consumers treat events as at-least-once; use deterministic keys + dedupe tables.

Reliability, failure, and recovery

• Retries: Short retry on the main topic; persistent failures route to DLQ with reason codes.

• Replay: Treat the log as truth; design consumers to replay from offsets to rebuild projections.

• Compensation: In sagas, model compensating actions explicitly (e.g., payment.refund.requested).

• Circuit breakers/timeouts on any synchronous edge calls.

Observability as a platform feature

• Unified correlation: traceId, spanId, and causationId on every event and HTTP call.

• Golden signals: per topic and per consumer group—lag, throughput, error rate, DLQ depth, p95 latency (where relevant), cost per message.

• Structured logs with event keys; metrics and logs link via trace IDs; dashboards by domain.

Security, compliance, and tenancy

• Zero trust between domains: Least-privilege IAM for publish/subscribe; no wildcard topic rights.

• Data classes & PII: Tag fields in schemas; enforce encryption in transit/at rest; mask in logs.

• Tenant isolation: Topic namespaces or headers; per-tenant quotas and consumer groups if needed.

• Secrets & rotation: Parameter store/Secrets Manager; no secrets in code or events.

Performance & cost posture

• Microservices handle steady, interactive load (apis, orchestration) with predictable autoscaling.

• Lambdas absorb spikes for enrichment/IO; pay-per-use keeps adapter costs low.

• Throughput design: Partition keys chosen for even distribution (e.g., orderId). Avoid hot keys.

• Caching strategy: Service-local caches for reference data; time-boxed and observable.

Governance & evolution

• API & schema review board (lightweight): validates naming, versioning, retention, and PII tags.

• Reusable adapter kits: Shared libraries/templates for inbound/outbound mapping, retries, metrics.

• Topic catalog: Human-readable registry of events, owners, SLAs, and consumers.

• Change playbacks: Before promoting schema changes, run consumer contract tests against a sandbox stream.

Edge integration patterns (common in enterprises)

• CDC → canonical: Debezium/CDC captures DB changes, Lambdas normalize to canonical, publish to Kafka.

• File lanes: SFTP/Share → S3 → Lambda parses → canonical events; error rows to a quarantine bucket + DLQ event.

• Webhook fences: External webhooks terminate at an API gateway → Lambda validates/throttles → publishes commands to the bus.

When

not

to use Lambda

• Long-lived connections (e.g., heavy gRPC), deep in-memory state, large dependency graphs, or strict low-latency APIs → run as a microservice.

• Anything requiring transactional composition across multiple steps → orchestrate in a service (saga), not in chained Lambdas.

The big picture (how it feels to operate)

• Day-to-day, teams ship small: a new event, a new consumer, a new adapter. Everything is observable and reversible (replay).

• Domains integrate by subscribing to each other’s facts, not by reaching into each other’s databases or endpoints.

• You keep Lambda count high where it helps (edge/utility) but ensure business truth lives in services with clear ownership.

• The platform team curates the backbone, schema registry, and guardrails, so feature teams focus on domain logic, not plumbing.

Bottom line:

For integration teams, the enterprise pattern is domain microservices for orchestration + event backbone for decoupling + Lambdas for edge/utility work, all wrapped in strong contracts, idempotency, and observability. It’s resilient, evolvable, and cost-smart—without sacrificing speed at the edges or rigor at the core.