Beyond Rate Limiting: Detecting AI Crawlers and Fake Browser Traffic at the Azure Edge

Modern website infrastructure faces an increasingly complex challenge: automated traffic that looks legitimate at the HTTP layer but behaves very differently from human users.

At KloudStack, we’ve been investigating growing volumes of crawler activity across managed WordPress environments hosted on Microsoft Azure.

What makes this interesting from an engineering perspective isn’t simply the volume of requests. It’s how automated clients distribute activity across IP addresses, impersonate real browsers and operate below conventional rate-limiting thresholds.

This article explores the traffic patterns we’ve observed, how we’re using Azure Front Door Web Application Firewall (WAF) to mitigate unwanted activity, and the architecture we’re exploring for more adaptive edge security.

1. The Infrastructure Cost of Automated Crawling

WordPress is a dynamic application.

A request that bypasses an existing cache can trigger processing across multiple infrastructure components:

• Azure Front Door routes the incoming request.
• The application processes the HTTP request through Nginx and PHP-FPM.
• WordPress loads the required application components.
• Plugins and themes execute application logic.
• MySQL processes database queries when required.
• The application generates the response.

Under normal conditions, these operations are expected.

The problem emerges when automated clients request large numbers of dynamic pages or repeatedly generate requests that cannot be efficiently served from cache.

A single crawler can cause hundreds of PHP executions in a short period. Multiply this behaviour across many sources and application infrastructure can experience substantial additional load without a corresponding increase in human traffic.

Scaling infrastructure may absorb the load, but it does not address the underlying inefficiency.

Our approach is to reduce unnecessary application requests before they reach the origin.

2. What We’re Seeing in Production

During a recent investigation, KloudStack identified a coordinated traffic pattern associated with clients claiming to use Firefox 131.

The user-agent identification appeared legitimate at first glance. The request behaviour suggested something very different.

Our investigation found:

Two distinct patterns were particularly interesting.

Pattern A: High-frequency page enumeration

Individual IP addresses would rapidly request approximately 190 pages, sometimes within only one or two minutes, before the activity stopped.

Other addresses would subsequently exhibit similar behaviour.

Traditional per-IP rate limiting can help mitigate this traffic, but rotating IP addresses allow automated clients to resume activity from another source.

Pattern B: Distributed low-rate crawling

A second pattern involved approximately ten IP addresses within a related network range.

Each source generated approximately 12–25 requests per minute.

Individually, these clients remained below a hypothetical 60-request-per-minute threshold.

Collectively, however, ten sources operating at that rate could generate 120–250 requests per minute against the application.

This exposes a fundamental limitation of independently evaluating each IP address.

A crawler doesn’t need to exceed a rate limit to create significant application load.

Browser identity is not browser behaviour

A user-agent string is supplied by the client. It does not prove that a request originated from the browser it identifies.

Real browsers generally request supporting assets, execute JavaScript and navigate websites through patterns associated with user interactions.

Automated clients may request HTML alone, enumerate URLs systematically or present a browser identity inconsistent with other transport and behavioural signals.

TLS fingerprints can provide another useful signal, but they should not be treated as definitive evidence by themselves.

The strongest classification comes from correlating multiple observations rather than relying on a single indicator.

3. Our Current Edge Security Architecture

KloudStack’s managed WordPress environments use Microsoft Azure infrastructure, with Azure Front Door Standard and custom WAF policies providing edge protection.

A simplified traffic path looks like this:

       Internet Traffic
              |
              v
    Azure Front Door Edge
              |
              v
      Custom WAF Policies
       /             \
   Blocked         Allowed
                       |
                       v
                Front Door Route
                       |
                       v
                 Azure App Service
                       |
                       v
                WordPress / PHP
                       |
                       v
                 Azure MySQL

The objective is to make blocking decisions as early as possible.

Requests denied by the WAF do not need to be processed by WordPress or generate the associated application workload.

Custom WAF rules

We use custom policies to address common application-specific threats and unwanted request patterns.

These include:

• Protecting sensitive WordPress endpoints.
• Restricting access to unnecessary execution paths.
• Applying different controls to administrative and public-facing routes.
• Blocking clearly unwanted request patterns.
• Rate limiting traffic where excessive request frequency indicates automation.

Rules must be designed around the requirements of individual customer applications.

For example, a public API, WordPress administration interface and heavily cached marketing page have different traffic characteristics.

Applying a single aggressive policy to every route risks blocking legitimate users and integrations.

4. Why Per-IP Rate Limiting Is Not Enough

Azure Front Door supports custom WAF rate-limit rules that evaluate request counts over a one-minute or five-minute period.

These rules are useful for reducing high-frequency requests, especially on endpoints that should receive relatively little traffic.

However, Azure Front Door’s native rate limiting is based on the socket IP address.

That means a group of coordinated clients can remain under individual thresholds while still creating substantial aggregate traffic.

Consider the following simplified example:

None of these clients individually exceeds 60 requests per minute.

Yet they generate 100 requests against the same application every minute.

Identifying this activity requires analysing traffic across sources rather than relying entirely on individual request limits.

A better detection model

Instead of examining only the number of requests from an IP address, we want to understand:

1. Which URLs are being requested?
2. How frequently are they being requested?
3. Are multiple sources following similar URL sequences?
4. Are clients requesting supporting browser resources?
5. Do user-agent identities match broader behavioural signals?
6. Is this activity correlated with increased application latency or resource consumption?
7. Does the traffic represent a known, desirable crawler or an unwanted automated client?

These signals provide more context for deciding whether traffic should be challenged, restricted or blocked.

5. Measuring the Value of Edge Protection

During a recent 30-day reporting period, KloudStack-managed custom WAF rules blocked more than 500,000 requests against a single customer website.

That’s an important operational metric.

It demonstrates the amount of traffic being intercepted before reaching application processing.

However, this metric requires careful interpretation.

Not every blocked request was necessarily an AI crawler. WAF telemetry may include scanning activity, unwanted endpoint access, excessive traffic and other policy-matching requests.

Similarly, the number of requests blocked does not directly translate into an equivalent reduction in CPU consumption.

The actual infrastructure benefit depends on which requests would otherwise have reached the origin, caching behaviour and the cost of processing those requests.

A more complete measurement approach correlates:

• WAF blocks and rate-limit events.
• Front Door access logs.
• Origin request volume.
• Application response times.
• App Service CPU and memory utilisation.
• PHP execution and database performance.
• Error rates and legitimate-user experience.

This creates a much stronger understanding of whether a security policy is improving hosting performance.

6. Evaluating Azure Front Door Premium

Azure Front Door Premium introduces additional WAF capabilities, including managed rule sets, Bot Manager and JavaScript challenge actions.

JavaScript challenges are particularly interesting for automated browser impersonation.

Rather than immediately blocking a matching client, the WAF can request that the browser complete a JavaScript-based verification.

Supported browsers can typically complete the challenge automatically, while some automated HTTP clients may be unable to do so.

However, JavaScript challenges require careful application design.

They aren’t appropriate for arbitrary APIs, AJAX calls, static assets or integrations that cannot execute browser JavaScript.

Geographic targeting

One approach we’re considering for Australian customer websites is applying targeted browser challenges to relevant traffic originating outside Australia.

This would allow most domestic visitors to avoid a challenge while selected overseas traffic receives additional verification.

The important limitation is that geographic matching is based on IP-derived location, not the actual physical location of the user.

Australians travelling overseas, using VPNs or accessing the internet through internationally routed corporate networks may still be challenged.

Likewise, Australian-hosted proxies may allow automated clients to appear domestic.

Geographic targeting is therefore a useful policy condition, not a reliable determination of whether a visitor is legitimate.

The commercial challenge

Premium security capabilities introduce additional infrastructure costs.

For hosting providers managing multiple customer environments, deploying dedicated Premium infrastructure for every customer may not always be commercially appropriate.

We’re evaluating the trade-offs between dedicated deployments, carefully isolated shared edge services and the additional operational controls required for multi-tenant infrastructure.

Security isolation, policy scope, change management and customer-specific requirements remain central to that assessment.

7. Toward AI-Assisted Edge Security

The next stage of our investigation focuses on making edge protection more adaptive.

Today, engineers can investigate suspicious patterns in logs, classify activity and update WAF policies.

That works, but it requires continuous operational attention.

As automated traffic becomes more distributed and sophisticated, we see an opportunity to reduce the time between detecting a suspicious pattern and responding to it.

Our proposed architecture introduces behavioural analysis and AI-assisted investigation alongside existing deterministic security controls.

         Azure Front Door
          Access / WAF Logs
                 |
                 v
         Azure Monitor / Logs
                 |
                 v
       Traffic Analysis Pipeline
                 |
        +--------+--------+
        |                 |
        v                 v
   Rule-Based         Behavioural
   Detection          Correlation
        |                 |
        +--------+--------+
                 |
                 v
          Risk Assessment
                 |
                 v
       AI-Assisted Analysis
                 |
                 v
       Policy Recommendation
                 |
                 v
       Validation & Guardrails
                 |
                 v
        Azure WAF Management
                 |
                 v
        Outcome Monitoring

This is a proposed direction rather than a production capability we’re announcing today.

Deterministic detection before AI

Not every traffic analysis task requires a large language model.

Identifying unusually high request rates, repeated URL enumeration and distributed traffic correlations can often be handled more efficiently using log queries and conventional detection logic.

The AI component becomes more useful when interpreting complex evidence, explaining unusual patterns and recommending appropriate responses.

For example, an analysis service could produce a structured assessment covering:

• Suspected automated traffic type.
• Supporting evidence and confidence.
• Affected customer environments.
• Potential application performance impact.
• Recommended mitigation.
• Expected impact on legitimate users.

This provides a basis for controlled policy adjustments rather than blindly applying new WAF rules.

Automated response requires guardrails

Allowing an autonomous agent to continuously modify production firewall policies introduces significant operational risk.

Poorly classified traffic could result in legitimate customers being blocked.

Our design principles therefore include:

• Prefer narrowly scoped, temporary mitigations.
• Apply conservative limits to automated changes.
• Keep an auditable history of decisions and policy updates.
• Validate proposed changes before deployment.
• Monitor for false positives and application errors.
• Support automatic expiry and rollback.
• Require human approval for high-impact changes.

The goal isn’t to let AI make unrestricted security decisions.

It’s to combine intelligent analysis with reliable, deterministic enforcement.

8. What We’re Building Toward

This work forms part of KloudStack’s broader direction toward an AI-native cloud operations platform.

We see edge security evolving from a collection of static WAF policies into an operational service that continuously observes infrastructure, correlates meaningful signals and recommends protective actions.

For managed WordPress environments, this means protecting more than just application security.

It means helping preserve CPU capacity, maintain responsive websites, reduce unnecessary application processing and make infrastructure spending more efficient.

As crawler activity continues to evolve, providers need to understand not just how much traffic an application receives, but what that traffic is doing and whether it should reach the application at all.

That’s the engineering problem we’re working to solve.

KloudStack Engineering

Building intelligent cloud infrastructure, managed application platforms and AI-native operations on Microsoft Azure.

kloudstack.com.au