Screener.in Search API: A Performance Checkup! 🔎

Hello everyone! So I’ve been digging into fundamental analysis lately, using a popular stock screening website to help me find promising companies. I’m a firm believer in understanding the “why” behind every number, and this extends to the tools I use. You know how it is – you start with the big picture, but the developer in me always wants to peel back the layers and see how the sausage is made.

It all started innocently enough. I’m using the filter’s search feature to find companies that match my specific criteria. On the surface everything looks good, clean and efficient. But my curiosity got the better of me. I started typing a search query, and out of habit, I opened the browser’s network tab to watch the API calls. That’s when I noticed something strange. Each keystroke sends a new API call to the server! This means that if I enter “KPI Green Energy”, the filter will send requests for “k”, “kp”, “kpi”, “kpig” and finally “kpigreen” respectively. Every query is sent to the backend without any delay or optimization, resulting in redundant network requests.

I did some timing analysis to see what was going on and this is what I found:

• API call duration: The server itself looks very fast. I see response times of 164 milliseconds (ThrotdlingURLLoader::OnReceiveResponse), 88 milliseconds (MojoURLLoaderClient::OnReceiveResponse), and 81 milliseconds (ResourceRequestSender::OnReceivedResponse). This indicates that the backend is handling requests efficiently.

• XHR activities: The real culprit is on the client side (my browser). The rapid succession of XHRReadyStateChange and XHRLoad events confirmed my suspicions: there is no intelligent throttling or delay mechanism. Essentially, the browser sends each keystroke to the server immediately without any waiting time.

I found some serious issues with the way the stock screener searches work. It’s like a leaky faucet wasting water (bandwidth and server power). This is the problem:

⚠️Question:

• No debounce: Every time you type a letter, Search sends a message to the server. This is too much news! It’s like sending a text message for each letter in a word instead of sending the entire word at once. This makes the server work too hard and wastes network speed.

• No throttling: There is no speed limit on how fast these messages can be sent. Type it fast enough and you’ll flood the server with requests.

• Redundant calls: Many messages sent are incomplete words (such as “k”, “kr”, “kri”) that no longer matter once you’ve finished typing. It’s like writing every unfinished sentence you start before finally completing a sentence.

• No client cache: If you make a mistake and backspace, it will send the same message again instead of remembering what it already knows. Like forgetting what you just said.

📊 Effect:

1. Server load is too high: The server was overwhelmed with requests, most of which were incomplete queries.
2. Unnecessary online activity: The network is full of redundant data transmission, consuming bandwidth.
3. slow response time: A constant stream of requests may result in slower responses because the server is busy handling unnecessary calls.

💡Recommendations:

De-bouncing is like taking a “wait and see” approach. It does not respond immediately to each keystroke, but waits for a specified period of inactivity before performing a search.
Introduce an anti-shake timer to delay API calls until the user pauses typing (for example, 300 milliseconds).

let debounceTimer; 

const searchInput = document.getElementById("input");

searchInput.addEventListener("input", (event) => {
  const inputValue = event.target.value;
  const delay = 500; // Set delay to 500ms (0.5 seconds)

  // 1. Clear Any Existing Timer:
  //    If the user is still typing, we cancel any pending search.
  clearTimeout(debounceTimer);

  // 2. Start a New Timer:
  //    After the delay, execute the search only if the user stopped typing.
  debounceTimer = setTimeout(() => {
      // 3. Perform the Search Action:
      //    The searchTasks function is called only when no input has been received for 500ms
      console.log("Performing search for:", inputValue); 
      searchTasks(inputValue);  // Replace this line with actual search call
  }, delay);

});

// Dummy search function - Replace this with your actual search implementation
function searchTasks(query) {
  // Here you would perform the actual search logic
  // Example: Fetch data from API using 'query'
  console.log("Searching for: ", query);
}

How it works in practice

1. The user starts typing.
2. Each character entered triggers an input event.
3. For each event, the previously scheduled timeout is cleared and a new timer is set.
4. If the user pauses for 500 milliseconds, the setTimeout callback will run and the searchTasks function will be called with the current input values.
5. If the user continues typing, the timeout will continue to be cleared and reset, preventing searchTasks from being called until typing has stopped for 500 milliseconds.

• Limit call rate

Throttling is like implementing “gatekeeping.” It limits the rate at which the function is allowed to execute. Even if the user continues typing quickly, new search requests are only sent after the specified delay, limiting the number of requests.
Limit the number of API calls within a specific time period (for example, one call per second).

let lastSearchTime = 0; // Timestamp of the last search
const searchInput = document.getElementById("input");

searchInput.addEventListener("input", (event) => {
    const currentTime = Date.now();
    const throttleDelay = 700; // Throttle delay (700ms)

    const inputValue = event.target.value.trim();

  // 1. Check if enough time has passed since the last search
    if (currentTime - lastSearchTime >= throttleDelay) {
    // 2. Perform Search Action
        console.log("Throttled search for: ", inputValue);
        searchTasks(inputValue); // Execute the search logic
        lastSearchTime = currentTime; // Update last search time
    } else {
        console.log("Throttled, not searching for: ", inputValue);
    }
});

// Dummy search function - Replace this with your actual search implementation
function searchTasks(query) {
    // Here you would perform the actual search logic
    // Example: Fetch data from API using 'query'
    console.log("Searching for: ", query);
}

How it works in practice

1. The user starts typing.
2. Each character entered triggers an input event.
3. For each event, the current time is compared to the last time the search was triggered (lastSearchTime).
4. If the time difference is greater than throttleDelay, the search logic (searchTasks function) is executed and lastSearchTime is updated.
5. If the time difference is less than throttleDelay, the search is skipped to prevent multiple calls to the searchTasks function. This ensures that the server is not flooded with search requests when users are actively typing, and reduces server load.

Client-side caching stores previous search results directly in the user’s browser. If a user searches for the same term (or a similar term for which cached results are available), the results can be retrieved directly from the browser rather than making a new server call.
Example: If the user enters krishna, cache the results for krish and kri for faster retrieval.

let searchCache = {}; // Initialize the cache
const searchInput = document.getElementById("input");

searchInput.addEventListener("input", async (event) => {
    const query = event.target.value.trim();

    if (!query) {
        // If empty query clear the results and return early.
        updateSearchResults([]);
        return;
    }

    const cachedResult = getCachedResult(query);
    if (cachedResult) {
       console.log("Using cache for query:", query);
       updateSearchResults(cachedResult);
       return;
    }

    console.log("Making API call for query:", query);
    // Simulate API call. In reality this would be an actual network call
    const results = await fetchSearchResults(query);

    if(results) {
        cacheSearchResults(query, results);
        updateSearchResults(results);
    }
});

function getCachedResult(query) {
    if(searchCache[query]) return searchCache[query];

    for (let i = query.length - 1; i > 0; i--) {
      const prefix = query.substring(0, i);
      if (searchCache[prefix]) {
          console.log("Using cached results for prefix: ", prefix);
          return searchCache[prefix];
      }
    }
    return null;
}

function cacheSearchResults(query, results) {
   // Store results for current query and all of its prefixes.
    for (let i = 1; i <= query.length; i++) {
        const prefix = query.substring(0, i);
        searchCache[prefix] = results;
    }
    console.log("Cache updated: ", searchCache);
}

// Simulate fetching results from an API (replace with actual API call).
async function fetchSearchResults(query) {
    return new Promise(resolve => {
        // Simulate API delay
        setTimeout(() => {
            const mockResults = generateMockResults(query);
            resolve(mockResults);
        }, 500);
    });
}

function updateSearchResults(results) {
    const resultsContainer = document.getElementById("results");
    resultsContainer.innerHTML = ""; // clear previous results

    if (results && results.length > 0) {
       results.forEach(result => {
          const resultItem = document.createElement("li");
          resultItem.textContent = result;
          resultsContainer.appendChild(resultItem);
      });
    } else {
        const noResults = document.createElement("li");
        noResults.textContent = "No results found";
        resultsContainer.appendChild(noResults);
    }
}

function generateMockResults(query) {
    const results = [];
    for (let i = 1; i <= 5; i++) {
        results.push(`${query} Result ${i}`);
    }
    return results;
}

how it works

1. User types search input.
2. The input event listener is triggered.
3. Call the getCachedResult function to check the cache.
4. If the result is in the cache, the result is retrieved and displayed directly.
5. If the results are not in the cache, the fetchSearchResults function is called to fetch the search results from the API, which is simulated in our sample code.
6. The results of the query and all its prefixes are then cached using cacheSearchResults and displayed to the user using the updateSearchResults function.

focus

1. Prefix caching: The code caches not only the results of the complete query, but also the results of each prefix. This helps make searches more responsive as users type.
2. Mock API: To demonstrate, I simulated an API call using setTimeout. You should replace it with your actual API call function.
3. Simple Cache: This is a very basic implementation. For practical applications, consider:
4. Cache Expiration: Results should expire after a certain amount of time.
5. Maximum size: Limit the size of the cache to prevent memory problems.
6. Cache expiration policy: Develop a policy on how and when items are removed from the cache.
7. Empty query handling: Added additional checks to handle empty queries by clearing results

How to use

1. Replace the mock fetchSearchResults function with actual API call logic.
2. Make sure there is an HTML element with the id input for the input field, and an element with the id results to display the search results.

• Send important messages
  If possible, send one large message that contains all of your search information rather than sending a lot of smaller messages.

Optimizing search functionality is critical to building efficient applications. Debounce, throttling, client-side caching, and bulk requests are powerful techniques that can significantly improve the performance of your search system.
This synergy results in faster responses, more efficiency, and an overall better user experience. It reduces server and network load, ultimately providing users with a smooth and efficient experience.

I hope the developers of Screener.in read this article. It looks like they missed some important website tips that make searching faster and easier to use. They should check and improve.

What do you think of these insights? Let me know! 💭