SCRNIFY Scales to 47 Million Requests Per Year (And We're Just Getting Started)

Hey there! Laura and Heidi here from SCRNIFY. Today's story is about numbers, surprises, and why you should never assume anything in tech. Grab your coffee, because we're about to dive into some interesting discoveries about our screenshot API's capabilities! ☕

The "We Should Probably Benchmark This" Moment

So there we were, preparing for our API's release, when someone asked the obvious question: "What's our maximum capacity?" We realized we had no idea! 😅 Being the thorough developers we are (and definitely not because we were called out in a meeting), we decided to find out.

Learning From the Veterans

First stop: the internal tool team. They've been running a similar service that handled over 650 million requests last year (yeah, we were surprised too!). The conversation went something like this:

"How do you know when to add a new node to your cluster?"

Their answer was beautifully simple: "If the cluster memory utilisation goes above 80% for more than 24 hours, we prepare a new node."

"What about CPU and network bandwidth?"

They laughed: "Don't forget the IO... but 98% its the memory and the other 2% are bugs in the code :P"

This was our first clue that we might be in for some surprises.

Time to Get Scientific(ish)

We created a benchmark script to test different scenarios. Starting with example.com (because everyone starts with example.com), we gradually moved to more complex sites. Here's what we found.

Here's the script we used to test using Apache Benchmark:

#!/bin/bash

# Configuration
API_KEY="your_api_key"
BASE_URL="https://api.scrnify.com/capture"
ENCODED_TEST_URL="https%3A%2F%2Ftheverge.com"  # Pre-encoded URL of the test site
CONCURRENT_REQUESTS=20
TEST_DURATION=120  # Duration in seconds
OUTPUT_DIR="benchmark_results"

# Create output directory
mkdir -p "$OUTPUT_DIR"

# Construct the full URL
FULL_URL="$BASE_URL?key=$API_KEY&type=image&format=png&width=1920&height=1080&url=$ENCODED_TEST_URL"

echo "Starting benchmark with $CONCURRENT_REQUESTS concurrent requests for ${TEST_DURATION} seconds"
echo "Target URL: $FULL_URL"
echo "Results will be saved in $OUTPUT_DIR"

# Run Apache Benchmark and save results
ab -c $CONCURRENT_REQUESTS -t $TEST_DURATION -n 999999999 "$FULL_URL" > "$OUTPUT_DIR/ab_results.txt"

# Display results
cat "$OUTPUT_DIR/ab_results.txt"

echo "Benchmark complete! Check $OUTPUT_DIR/ab_results.txt for detailed results."

This handy script sets up the required parameters, runs the benchmark, and saves the outputs in the specified directory.

The Simple Stuff: example.com

Requests per second: 4.95
Memory Impact: +3GB
CPU Impact: +20%

Not bad! But real-world websites are a bit more... demanding.

The Heavy Hitter: theverge.com

Requests per second: 1.57
Memory Impact: +6GB
CPU Impact: +37%
Failed Requests: ~14%

(Some requests were blocked - turns out websites don't love being repeatedly screenshot. Who knew? 🤷‍♀️)

The News Giant: cnn.com

Requests per second: 4.72
Memory Impact: +3GB
CPU Impact: +25%

(Until they caught onto us and started blocking requests - we see you, CNN! 👀)

Plot Twist: It's Not the Memory!

Remember how the internal team said memory was their bottleneck? Well, surprise! Our cluster's memory barely flinched (9GB to 12GB on a 1TB cluster - basically a rounding error). The real bottleneck? CPU usage jumped by up to 37%!

Why? Remember our previous article where we talked about using Brave? Unlike the internal tool's modified browser setup, we start fresh for each request. It's like making a new cup of coffee for every sip - inefficient but super clean!

Finding the Sweet Spot

After much testing (and probably getting on some websites' blocklists), we found our optimal configuration:

• 20 concurrent requests
• ~3 requests per second
• Consistent response times
• Less than 2% failure rate

Being conservative (because we like sleeping at night), we're comfortable saying we can handle 1.5 requests per second sustainably. That's where our 47 million requests per year number comes from - and that's just the beginning!

Why These Numbers Matter

Our setup is different from the internal tool for good reasons:

1. Fresh browser instance = perfect isolation
2. No cached state = consistent results
3. Clean environment = predictable behavior

Yes, this impacts performance, but we believe the trade-offs are worth it. Security and consistency over raw speed - that's our motto! (Well, it is now, we just made it up.)

Scaling to Infinity (and Beyond!)

Here's the best part: these numbers are just our current cluster capacity. Need more? We just add nodes. It's like adding more coffee machines to a café - simple and effective!

The internal tool's 650 million requests last year? Yeah, we can get there. We're just warming up!

What's Next?

We're planning more detailed benchmarks with controlled test sites (no more getting blocked! 🚫), but these real-world tests give us confidence in our system's capabilities.

Want to help us push these limits? Head over to SCRNIFY and start capturing! We're ready for whatever you throw at us (just maybe not 47 million requests all at once, okay? 😅).

Cheers, Laura & Heidi 🇦🇹

P.S. Yes, we did get temporarily blocked by several websites during this testing. No websites were harmed in the making of this article!

P.P.S. Special thanks to the internal tool team for their insights. Your "memory rule" led us down quite the interesting path! 🙏