Skip to content

API Integration Patterns: How to Choose the Right One

Learn the main API integration patterns, from request/response and webhooks to batch and streaming, with trade-offs, retry rules and scraping examples.

Pim

Pim · Founder, Scrappey

15 min read

On this page (8)

The job has run cleanly for three weeks. Then one Tuesday it returns a 200 with an empty body, your loader writes a row full of nulls, and the pricing dashboard shows a competitor selling at nothing. Nobody touched your code. A vendor changed a header, the proxy rotated to a slower exit, and a timeout in the middle of a write turned one request into two records.

Most integration failures aren't parsing bugs. They come from choosing a pattern that doesn't match how long the work takes or how expensive a duplicate is. Get the timing and the failure model right and the code stays boring. Get them wrong and you spend a quarter on retry storms, dead rows and support tickets.

The patterns below are the ones teams ship, and every one gets judged against the same job: pulling data off web pages you don't control and landing it somewhere your business can use.

Six Integration Patterns at a Glance

Every integration answers one question. How does data move from a system you don't control into one you do, at the moment you need it? There are six answers worth knowing, and the only two columns that change your architecture are timing and failure cost.

Pattern How data moves Best for Main trade-off
Request/response Client sends a request and the server answers at once Lookups, on-demand scrapes, anything that finishes in seconds Caller waits, so timeouts must be handled
Polling Client checks job status on an interval Long jobs when the provider offers no callback Wasted calls; delay of up to one polling interval
Webhooks Provider sends an HTTP POST to your endpoint when an event happens Job completion, price changes, status updates Your endpoint must be public; duplicates and retries will arrive
Pub/sub Publisher sends messages to a topic and many subscribers receive copies Fanning one event out to storage, alerts and analytics A broker to run; ordering and duplicates need design
Batch Records are collected and moved together on a schedule Nightly snapshots, large catalogs, reporting Data is only as fresh as the last run
Streaming Records flow continuously over one open connection Live feeds, monitoring, real-time dashboards Needs backpressure, reconnection and cursors

Read it as a lookup. Find the row that matches your timing need (immediate, delayed, event-driven or scheduled), then check your volume. Real systems stack rows: a request starts a scrape, a webhook fires when it finishes, and a nightly batch loads the archive.

Almost nobody runs a single pattern. A 2021 survey of 950 IT decision-makers found that 99% of organizations used an integration system and 64% operated hybrid environments spanning on-premises and cloud, against 23% cloud-only and 12% on-premises-only, as reported in Postman's 2023 State of the API report. What decides how much code you write is timing and failure cost, not which protocol is fashionable this year.

Request/Response and Polling for On-Demand Data

Request/response is a phone call. You dial, you wait on the line, and an answer comes back or a busy signal does. It's the first pattern every team learns and still the one most integrations run on. Postman's 2020 State of the API survey found REST in use by 93.4% of respondents, against 34.4% for webhooks, 33.4% for SOAP and 22.5% for GraphQL (State of the API 2020).

The stack is cumulative rather than replacement-based. SOAP 1.1 arrived in 1998 built around XML messages and remote procedure calls, Roy Fielding formally defined REST in 2000, JSON landed in 2001 as a simpler interoperable data format, and OAuth 2.0 was standardized in 2012 for delegated access between applications. Older protocols don't disappear. They get an adapter in front of them.

A four-step infographic explaining request-response and polling communication patterns for web API data integration.

A scrape follows that same shape. You send a standard HTTP POST to the API endpoint with your key and a JSON payload that describes the task, and JSON comes back in the response. The request and response shapes on other data APIs follow the same rules, which is why StalkChain's guide to wallet data is worth a read if you're pulling structured records from a source rather than a rendered page.

Picking the wrong tier is the most common way teams waste a request budget.

Tier JavaScript execution Balance per request Best for
Direct HTTP Not executed 0.1 Static pages and endpoints
Full browser Executed 1 JavaScript-heavy pages

Direct HTTP routes through a residential proxy with rotating IPs, custom headers, cookies and a TLS fingerprint, but it does not execute JavaScript, so it's right for static pages, JSON endpoints and anything rendered server-side. Full browser runs the page for real, so save it for pages that build content client-side or push you through a verification flow.

Polling a long job without hammering the API

Some jobs don't finish inside a request timeout, and holding a connection open is how you get half-written results and orphaned workers. When there's no callback available, poll in a disciplined loop.

  1. Submit and store the ID. Post the job, then write the returned job ID next to the request payload before you do anything else.
  2. Start short. Two to five seconds is enough for most jobs. Tighter than that just burns calls.
  3. Back the interval off. Double it as the job drags, and cap it at a minute.
  4. Fetch once. When the status reads complete, pull the result and mark the row done in the same transaction.
  5. Stop at the deadline. Set an absolute cutoff, alert, and move the job to a failure queue.

The cost is real: every check is a request you pay for, and your data is up to one full interval stale. For the payload itself, this cURL JSON POST walkthrough shows the headers and body structure you'll reuse on every call.

Practical rule: if the answer comes back inside your request timeout, use request/response. If the job takes minutes, stop holding the connection and switch to polling or a webhook.

Webhooks, Pub/Sub and Event-Driven Delivery

A webhook is a doorbell. Nobody stands at the door all afternoon waiting. The provider posts an HTTP request to a URL you own the moment something happens, and you stop asking. When a long scrape finishes, the provider posts the completed event to your endpoint; Scrappey supports webhooks for real-time notifications, which turns a polling loop into a single listener.

A friendly robot sending data to a laptop via a webhook to visualize API integration concepts.

Receiving is where this often goes wrong. The sender guarantees an attempt, not a clean delivery, so the endpoint carries the burden.

  1. Verify the signature first. Recompute the HMAC over the raw body with your shared secret and compare it in constant time before you parse a single field.
  2. Store the raw event with its ID. Write the body to durable storage alongside the provider's event ID, before any business logic runs.
  3. Acknowledge fast. Return a 2xx quickly and do the work somewhere else. A slow handler is a handler that gets retried.
  4. Process from a queue. Let a worker pick up the stored event, so a database hiccup doesn't turn into a failed delivery.
  5. Skip IDs you've seen. Check the event ID against what you've processed and exit early if it's a repeat.

If you've never stood up a public receiver, watch the delivery mechanics end to end before you wire one into production.

Pub/sub is the bulletin board next to the doorbell. A publisher drops one message on a topic and every subscriber gets its own copy: storage writes it, alerting checks a threshold, analytics aggregates it. The publisher doesn't know or care who listens.

The failure modes are consistent. Your endpoint has to be reachable from the public internet, events arrive twice when a 200 gets lost in transit, and they arrive out of order because retries don't queue politely. Subscribe to a topic and you inherit a broker, plus decisions about ordering and deduplication.

Event-driven delivery moves the reliability burden onto the receiver. That's why the retry and idempotency rules matter most here.

A worked setup: one public endpoint receives the completed-job event, verifies the signature, stores the raw body, and publishes it to a topic. Storage, alerting and analytics each get their own copy and each deduplicate by job ID before acting on it. One flaky subscriber never blocks the other two.

Batch and Streaming for High-Volume Pipelines

Two questions decide this one. How fresh does the data need to be, and what does a stale record cost you? Everything else follows.

Take a retailer tracking 50,000 product pages. Batch collects the URLs and moves them together on a schedule: one job at 2 a.m. submits all 50,000, waits for results, and loads them into the warehouse before the morning report runs. It's efficient, cheap to audit, and easy to replay. The data is only as fresh as the last run, and a single failed batch has to be retried with per-item status so one bad URL doesn't block the other 49,999.

Streaming keeps records moving over one open connection, using server-sent events, WebSockets or chunked HTTP responses. A live price feed or a monitoring dashboard is the natural fit. In exchange you owe the system three things: backpressure, so a slow consumer doesn't drown the pipeline; reconnection that resumes from a stored cursor rather than restarting; and handling for partial messages that arrive truncated.

Batch is a scheduling problem. Streaming is a state-management problem, and state is harder than a cron entry.

Here's the sequence I'd follow.

  1. Estimate records per hour. An honest number, not the peak you hope for.
  2. Write down the freshness the business needs. "End of day" and "within a minute" are different products.
  3. Choose batch if hours of delay are acceptable. Most reporting is.
  4. Choose streaming if seconds matter. Price changes the business reacts to, or alerts that page someone.
  5. Add per-item status tracking either way. It's what makes retries surgical instead of a full re-run.

For the nightly version, post each of the 50,000 URLs to the request endpoint with its own job ID, record status per item, and retry only what failed. A streaming variant forwards each result to the warehouse the moment its job completes. If you're leaning toward the second, this walkthrough of a real-time data pipeline covers the pieces you'll need.

Gateways, Adapters and Orchestration Layers

Above the basic patterns sit three layers that decide whether your integration survives a vendor change. They're worth naming because teams rebuild them badly, one incident at a time.

An adapter translates one service's response into your own data schema. When a vendor renames a field or nests a value one level deeper, a good adapter means you edit one file instead of eight. A gateway is the single entry point for every upstream call, owning authentication, rate limits, logging and retries so each worker doesn't reimplement them.

A diagram illustrating the three layers of API integration: Orchestration, Gateway, and Adapter, with their definitions.

Orchestration puts one coordinator in charge. It scrapes the page, runs extraction, validates the result, then writes to the database, in that order, and stops when a step fails. Choreography has each service react to the event published by the step before it, with no coordinator watching.

Orchestration is easier to debug because one log shows the whole sequence and the retry logic lives in one place. Choreography scales better across teams, since each service ships on its own schedule and a broken notification step doesn't halt the write. Pick orchestration when order matters and the workflow is short. Pick choreography when several independent consumers need the same event.

Where the AI extraction step fits

The models now sit inside these layers, and many teams haven't noticed the mismatch. In the 2025 reporting covered by DevOps Digest on API strategy becoming AI strategy, 89% of surveyed developers used generative AI in daily work, but only 24% designed APIs with AI agents in mind. The testing gap is sharper still: functional and integration testing each reached 67% adoption, while contract testing sat at 17%.

Adding an agent in front of a weak contract makes the integration less reliable, not more. An agent needs stable schemas, explicit error values and bounded side effects before it can safely call anything.

That's the argument for a defined output shape. Scrappey's AI Extraction lets you describe the fields you want and receive JSON in exactly the structure you define, with no selectors to maintain, so the adapter downstream stays trivial. The same workflow hosts well in LangChain, LlamaIndex, n8n or Make, all of which Scrappey integrates with.

Every step needs a stable schema, explicit error values and validated output, so the next step can rely on what it receives.

Retries, Idempotency and Failure Rules

A POST times out. The client retries. The server creates the record twice, and now two warehouses think they own the same SKU. This is the most expensive bug in integration work, and it's entirely preventable.

Idempotency keys are the fix. The client generates one unique key that represents a single business operation, not a single HTTP attempt, and sends it with every retry. The server records the result of the first execution and returns that same result for any later request carrying the same key. Stripe's idempotency documentation is the clearest description of the contract: the API stores the status code and response body once endpoint execution begins, including 500 responses, while validation failures and concurrent conflicts can be retried because execution never started.

The practical rules that follow are strict. Generate the key from the logical operation, persist it next to the job state, bind it to an immutable payload, and reject reuse with different parameters. GET and DELETE are idempotent by HTTP semantics, so the work goes into the state-changing methods.

Classify before you retry

A retry is a guess that the failure was temporary. Make it a decision instead. Google Cloud's retry strategy guidance recommends exponential backoff when clients call its APIs directly, because progressively longer delays reduce load while the dependency recovers.

  1. Classify the failure. Network interruption, 408, 429 and selected 5xx are candidates. Auth failures, malformed payloads and deterministic 4xx need a fix.
  2. Retry only the transient class. Everything else goes to a human or a dead letter.
  3. Back off exponentially with jitter. Randomized so a thousand workers don't knock in unison.
  4. Cap attempts and set an absolute time budget. An attempt count alone lets a slow endpoint eat an afternoon.
  5. Route exhausted jobs to durable storage. A dead-letter queue keeps the endpoint from consuming all worker capacity.
Failure Retry? Action
Network reset, timeout or dropped connection Yes, with idempotency key Retry with exponential backoff and jitter; for writes, check the stored result first
408 Request Timeout Yes Retry within the attempt cap and the overall deadline
429 Too Many Requests Yes Honor Retry-After when supplied, then slow the worker
5xx server error Yes, selectively Retry a bounded number of times; open a circuit breaker if failures persist
400, 401 or 403 on a malformed or unauthorized request No Fix the payload or the credential; repeating the same request changes nothing

Open a circuit breaker when a dependency keeps failing, so the calls stop before the queue does. Then watch four numbers: retry rate, p95 and p99 latency, timeout rate, and queue age. Queue age is the one that catches problems nobody alerts on.

SDK retries vary by client and by how you construct it, so set the policy explicitly rather than assuming a library already has your back.

Scrappey's intelligent retry logic, residential proxies and reliable web access handling lift the success rate on transient access failures, but your client still needs its own budget, its own keys and its own ceiling. If rate limits are the thing biting you, this guide to handling API throttling covers the backoff side in more depth.

Securing Integrations Across Their Lifecycle

Most pattern comparisons rank REST, GraphQL, webhooks and WebSockets by latency and payload shape. That framing hides the actual risk. Security failures don't happen at the protocol layer, they happen across the whole lifecycle: an endpoint nobody inventoried, a token with more scope than it needs, an authorization decision that drifted six months ago.

The numbers make the gap plain. In the 2025 Global State of API Security from Traceable, 57% of surveyed organizations reported at least one data breach caused by API exploitation in the previous two years, while organizations tested only 38% of their APIs for vulnerabilities on average. A further 54% named API sprawl prevention as a major problem. The hard question isn't which pattern to use. It's how you govern hundreds of internal, partner, SaaS and data-access integrations without losing sight of them.

One check per stage

Discovery. Keep an inventory of every third-party endpoint your workers call, including the ones inside a script someone wrote last spring.

Authentication. Rotate credentials on a schedule and store them in a secret manager, never in a job payload or a log line.

Authorization. Give each connection a least-privilege credential, so the analytics reader cannot delete anything.

Schema validation. Reject responses that don't match the contract at the adapter, before they reach a table.

Rate limiting. Enforce it at your gateway, not in each worker, so one runaway job can't starve the rest.

Observability. Log which token called which endpoint and when. Without that trail, incident response is guesswork.

Incident response. Have a kill switch per integration, and know who to call before you need to.

Deprecation. Set a review date on every endpoint. An integration nobody owns is an integration nobody secures.

Webhooks and machine-to-machine links deserve to be treated as security boundaries, not conveniences. Each one needs a named owner, replay controls, idempotency and its own credentials. Scrappey is designed for accessing publicly available web data and powering automation workflows, and it offers a signable Data Processing Agreement for customers who process personal data through the API.

Copy this into your review: owner named, credential scoped, signature verified, schema validated, rate limit set, logs retained, kill switch tested, review date booked.

Choose Your Pattern and Test It This Week

Go back to the lookup table and find the row that matches your timing need and your volume. Then stop reasoning and measure, because one hour of testing beats another week of architecture debate.

  1. Send one request to a target page through the direct HTTP tier. Record the response time and the shape of what comes back.
  2. Repeat on a JavaScript-heavy page with the full-browser tier. Compare the output and decide whether the difference justifies the heavier call.
  3. Log every job ID with its status. If you can't answer "where is job 4471 right now", the pipeline isn't observable yet.
  4. Add the retry and idempotency rules from the failure table. Classify, back off, cap, dead-letter.
  5. Replace polling with a webhook once your endpoint verifies signatures. Not before.

A boy points at a test schedule clipboard while a laptop sends requests to a remote server.

Billing is straightforward while you test. Each API call counts as one request, a full-browser request uses 1 balance and a direct HTTP request uses 0.1 balance, and every request includes proxies, challenge handling and reliable web access handling. That last part matters more than the cost. A cheap request that gets blocked is more expensive than a full-browser call that returns clean JSON the first time.

Build the smallest version of your chosen pattern first. One endpoint, one job ID, one status check, one retry rule. Measure it, then add the next piece. Teams that get the pattern right before they get the scale right almost always finish sooner.


Scrappey gives you a single endpoint that returns clean JSON from pages that fight back, with proxies and challenge handling included in every request, so your integration work stays focused on the pattern instead of the plumbing. Start with the direct HTTP tier for static targets, move to the full-browser tier only where JavaScript runs, and let the failure rules above do the rest. You can run the first test this afternoon by signing up for a free trial at Scrappey, no card needed.

#API integration patterns#webhooks vs polling#REST API integration#idempotency keys#web scraping API

Collect web data without running the infrastructure

Scrappey handles browsers, proxies and retries behind one API. Free credits to start, no card needed.