Skip to content
flowss
Start freeChecking your session
Cloud architecture templates

AWS — Streaming Data Pipeline

Ingest to dashboard, with every edge carrying the kind of movement it really is: async where a stream sits between two services, data where bulk moves, request traffic only where something calls something. Reports clean. Drawing all three as one arrow is the commonest way an architecture diagram stops being useful.

Cloud architectureData & SchemaCloud architecture syntax docs
Template previewCloud architecture
Clickstream pipelineAn AWS architecture diagram: 11 resources in 3 scopes (0 networks, 0 availability zones), joined by 12 connections. Categories drawn: Serverless, Storage, Databases, Edge & delivery, Integration, Analytics, Observability, Clients. No error or warning across 11 resources and 12 edges, every line read; 4 of the 10 checks could not run at all — so this is a clean result on the part that could be checked, not a clean bill of health.Clickstream pipelineAWS · 11 resources · 3 scopes · 0 networks · 0 zones · 12 connectionsNo error or warning across 11 resources and 12 edges, every line read; 4 of the 10 checks could not run at all — so this is a clean result on the partthat could be checked, not a clean bill of health.Internetintern…8AccountAnalytics account8Regionus-east…8Site visitors—5+Collectorendpointaws5+Enrichmentaws5+Click streamaws1+Deliverystreamaws6+Raw data lakeaws1+Data catalogueaws6+Ad-hoc queryaws6+Warehouseaws1+Dashboardsaws6+Pipeline logsaws6+https 443invokeclick eventsbatched eventsparquetcopyscansql 5439sqllog eventsLegendServerlessStorageDatabasesEdge & deliveryIntegrationAnalyticsObservabilityClientsRequest trafficData movementAsynchronousDepends onChecks — what ran, and what could notWhether the internet reaches something that ho…ran over 3 subjects, found nothingWhether anything holding data at rest sits in …could not run here — 1 reason givenWhether anything claiming redundancy is spread…could not run here — 1 reason givenEdges that leave one network for another, or c…could not run here — 1 reason givenResources that appear in no edge at all. A leg…ran over 11 subjects, found nothingEvery id an edge or a scope names is one that …ran over 11 subjects, found nothingEach id is declared once. Two declarations mak…ran over 14 subjects, found nothingType words that did not resolve. A hole in thi…ran over 11 subjects, found nothingDeclared ranges: that each parses, sits inside…could not run here — 1 reason givenWhether every line was read and no shape cap b…ran over 14 subjects, found nothingFindings1`stream`, `lake` and `warehouse` hold data at rest and sit inside no subnet at all, so nothing in this document says whether they are publicly routable and this checkhad nothing to read.in the figure: stream, lake, warehouse2This document declares no availability zone, so the redundancy check DID NOT RUN AT ALL: nothing here claims redundancy either, so nothing was missed. Declare thezones (`zone az-a`, `zone az-b`) and put the resources inside them, and the claim becomes checkable.3This document declares no `network` scope, so there is no boundary for an edge to cross and this check DID NOT RUN over any of its 12 edges. Nothing here says thetraffic stays inside one network.4No network or subnet was declared, so there was no addressing to check.5The walk reached 3 resources from the internet and none of them holds data at rest — there was nothing here to expose. What the walk never reached is outside what thischeck can say.in the figure: visitors, collect, enrich6All 11 declared resources appear in at least one edge.in the figure: visitors (badge 5), collect (badge 5), enrich (badge 5), stream (badge 1), delivery +6 more7All 11 ids named by edges and scopes were declared somewhere in this document.in the figure: visitors (badge 5), collect (badge 5), enrich (badge 5), stream (badge 1), delivery (badge 6) +6 more8All 14 declared ids are distinct.in the figure: internet, prod, us-east-1, visitors (badge 5), collect (badge 5) +9 more9All 11 resources resolved to a type this engine knows, so every check above had the full vocabulary to read.in the figure: visitors (badge 5), collect (badge 5), enrich (badge 5), stream (badge 1), delivery (badge 6) +6 more10Every line was read and no shape cap bit: the 14 declarations above are the whole document, so the counts are totals rather than floors.

Make it your own.

title "Clickstream pipeline"
provider aws

internet {
  browser visitors "Site visitors"
}

cloud prod "Analytics account" {
  region us-east-1 {
    api-gateway collect "Collector endpoint"
    lambda enrich "Enrichment"
    kinesis stream "Click stream"
    firehose delivery "Delivery stream"
    s3 lake "Raw data lake"
    glue catalog "Data catalogue"
    athena query "Ad-hoc query"
    redshift warehouse "Warehouse"
    quicksight dash "Dashboards"
    cloudwatch logs "Pipeline logs"
  }
}

visitors -> collect : https 443
collect -> enrich : invoke
enrich ~> stream : click events
stream ~> delivery : batched events
delivery => lake : parquet
lake => warehouse : copy
catalog ..> lake
query ..> catalog
query => lake : scan
dash -> warehouse : sql 5439
dash -> query : sql
enrich ~> logs : log events