How I cut an SEO rank tracker's costs by 39% through an architecture redesign

I reduced an SEO rank tracker's daily cost from $0.28 to $0.17 without removing features, tracking fewer keywords, or sacrificing its historical data. That is a 39.3% reduction.

The savings came from one architecture decision: stop paying multiple times for exactly the same information.

That was the issue I found in a custom SEO rank tracker I built around DataForSEO. The application organized projects, keywords, domains, and historical rankings. It did its job, but its architecture tied every external request to an individual project.

The inefficiency became clear whenever the same keyword appeared in more than one project.

If three projects tracked “locksmiths Barcelona,” the system submitted three separate requests. Yet the Google results page we needed to analyze was identical in every case.

Fixing this required more than optimizing a function. I needed to change the system's unit of work. That change saves $0.11 per day—approximately $3.30 per month or $40.15 per year when the application runs every day.

The outcome

After the redesign, the application:

• Reduced its daily spend from $0.28 to $0.17—a 39.3% reduction.
• Preserved every tracking and comparison feature.
• Retained all 19,294 historical rankings.
• Can use one SERP to calculate results for multiple projects.
• Can add projects that use existing keywords without immediately triggering another external request.

The savings did not come from delivering less value. They came from eliminating duplicated external work and moving project-specific domain comparisons into the application.

The problem: one request per project, keyword, and domain

In the previous architecture, each project owned its tracking tasks. Conceptually, the cost model looked like this:

Project + keyword + domain = one API request

For example:

“locksmiths Barcelona”

Project A → one request
Project B → another request
Project C → another request

This duplication did more than increase API usage. It split the same source data across separate external tasks, repeated logic across multiple endpoints, and made consistent historical tracking harder to maintain.

The review also uncovered related issues:

• Credentials and tokens stored in the codebase.
• Inconsistent historical dates.
• Redundant or partially broken functions.
• No single source of truth for a SERP response.
• Limited protection against concurrent or repeated requests.
• A structure that became more expensive as new projects were added.

The system modeled each project's need, but not the resource we were actually purchasing: one search results page for one keyword.

The key decision: make the keyword the unit of retrieval

I redesigned the flow so that each unique keyword generates one external request:

Unique keyword = one API request

Once DataForSEO returns the complete SERP, the application stores it once and performs every domain comparison internally. The same response can serve all projects associated with that keyword.

Keyword: “locksmiths Barcelona”
             ↓
One DataForSEO request
             ↓
Complete SERP stored once
             ↓
Project A ranking
Project B ranking
Project C ranking

This separated two responsibilities that had previously been coupled:

1. Retrieving a SERP from an external provider.
2. Calculating where each project's domain appears within that SERP.

The first operation has an external cost and should not be duplicated. The second happens inside the application and can be repeated as many times as needed.

Building a global, deduplicated keyword catalog

Exact string matching was not enough to support the new design. Differences in capitalization, accents, whitespace, or punctuation could create records that looked different while representing the same search.

I introduced a global keyword catalog and normalized every term before storing it:

• Convert it to lowercase.
• Remove accents.
• Normalize whitespace.
• Remove irrelevant punctuation.

As a result, these inputs:

cerrajería barcelona
Cerrajeria Barcelona
CERRAJERÍA   BARCELONA

all share the same normalized key:

cerrajeria barcelona

Normalization is not merely a data-cleaning improvement here. It prevents the system from paying for duplicate requests caused by formatting differences.

The new data architecture

The redesigned model uses five main entities:

• keywords: the global, deduplicated search-term catalog.
• projects: the projects that require rank tracking.
• project_keywords: the relationship between a project, a keyword, and the domain to locate.
• serp_runs: every DataForSEO request, including status, payload, responses, attempts, errors, and timestamps.
• project_rankings: the project-specific results calculated from a stored SERP.

This creates a clear source of truth. serp_runs records what the external provider returned, while project_rankings stores how that response applies to each project.

It also improves traceability. If a ranking looks unexpected, I can identify the request that produced it, inspect the original response, and follow the calculation that generated the project-specific result.

Asynchronous processing without duplicate requests

DataForSEO does not always return the final result when a task is first submitted, so I separated the workflow into two scheduled processes.

A daily publishing cron selects only keywords that:

• Are active.
• Belong to at least one active project.
• Do not already have a pending request.
• Have not been queried that day.

A second cron runs every five minutes and checks submitted or pending requests. If a result is not ready, the request stays pending. Once complete, the process stores the full SERP, calculates every associated project ranking, and stops polling it.

I also added database locks to prevent two cron executions from processing the same work concurrently.

Deduplication is therefore enforced throughout the request lifecycle, not only in the database schema.

One SERP, multiple rankings

When a complete response arrives, the internal processor:

1. Iterates through the organic results.
2. Normalizes the returned domains.
3. Compares them with the domains configured for each project.
4. Selects the best position for each domain.
5. Stores the exact ranking URL.
6. Explicitly records when a domain was not found within the requested search depth.

One response can produce as many rankings as there are project associations for that keyword.

The stored response can also be reused later. If a new project starts tracking a keyword with an existing SERP, the application can calculate its ranking immediately without another DataForSEO request. Adding projects that reuse known keywords therefore has an external marginal cost close to zero.

Migrating the history without losing data

The redesign also had to preserve the existing dataset. The previous database contained:

• 7 projects.
• 84 tracking tasks.
• 57 unique keywords.
• 19,294 historical snapshots.

After migration, the new system retained:

• All 7 projects.
• All 57 normalized keywords.
• All 84 project-keyword-domain relationships.
• All 19,294 historical rankings.
• 19,268 shared historical runs.

No snapshot was discarded, and external identifiers from different keywords were never combined.

One complication was that the old captured_at values were unreliable. I reconstructed the timeline using valid created_at timestamps instead. Migrated runs were explicitly marked as legacy data so they could never trigger new DataForSEO requests.

Preserving and improving the product experience

Reducing API usage could not come at the expense of useful reporting. The redesigned application preserves and expands its historical comparisons:

• Current and immediately previous ranking.
• Positive or negative movement.
• Seven-day and 30-day comparisons.
• Comparison against a specific date.
• Up to 100 historical points per tracked item.
• The exact URL found in each measurement.
• Distinct pending, completed, not-ranking, and error states.

The interface displays gains in green, declines in red, and unchanged positions in a neutral color. Rankings are ordered from best to worst, with pending items placed at the end.

Treating security as part of the redesign

The architecture review also created an opportunity to address risks unrelated to API cost:

• Credentials were removed from the repository.
• Local configuration was excluded through Git.
• Write endpoints were protected with an administrative token sent through a request header.
• Cron processes became CLI-only.
• Public reads and administrative operations were separated.
• Database locks and status checks were added to prevent duplicate external requests.

The result does not merely make fewer calls. It is also safer, easier to audit, and easier to maintain.

The main lesson

The most important improvement was not a particular SQL query, cron job, or function. It was recognizing that the system had chosen the wrong unit of work.

When retrieval was tied to a project, every new project could multiply API consumption. Once retrieval was tied to a unique keyword and domain matching moved inside the application, adding projects no longer implied the same growth in external requests.

Whenever several consumers need the same information, it is worth asking whether they truly need to retrieve it separately—or whether they can share one source and derive their own results from it.

In this case, that question transformed a collection of isolated tracking tasks into a shared, reusable architecture designed to scale. The measurable result was a reduction from $0.28 to $0.17 per day—39.3%—with no loss of data or functionality.