Building a Search Engine: Overcoming Challenges and Solutions

Building a Search Engine: Overcoming Challenges and Solutions

Table of Contents:

1. Introduction
2. Building the Server Side
3. Extracting Keywords
4. Handling Plurals
5. Downloading Websites
6. Web Scraping Challenges
7. VPN and IP Changes
8. Handling Heap Allocation Error
9. Dealing with Crashes
10. Conclusion

🔍 Building a Search Engine: The Journey of Challenges and Solutions

Introduction Have you ever wondered how search engines like Google index millions of websites and quickly provide relevant search results? In this article, we'll explore the journey of building a search engine from scratch, uncovering the challenges and solutions encountered along the way. Join me as we dive into the intricacies of web scraping, SEO optimization, and the art of indexing.

Building the Server Side The adventure begins by constructing the server side of our search engine. While JavaScript may have its perks, its effectiveness became questionable as I tackled the initial implementation. Frustration kicks in as the shaking takes longer than expected. Why couldn't it just download from the internet like a normal package manager? Despite these initial setbacks, I persevered and created sample pages to match with the intended solution.

Extracting Keywords To make the search engine truly functional, extracting keywords from web pages becomes a crucial step. I extracted every word from the page's text and counted their occurrences. Words with more than one occurrence were considered for query matches. Additionally, keywords included in the page title and description received higher priority. However, as I delved deeper into the system's mechanics, it became evident that the approach had flaws.

Handling Plurals The next obstacle in our search engine journey was handling plurals. It quickly became apparent that a simple word matching approach would not be sufficient. For example, a search for "kitten" should ideally match websites with the term "kittens" as well. Initially, I explored manually appending endings to nouns using a dictionary API. However, performance issues led me to find a downloadable dictionary. After some painstaking efforts, success was achieved. The engine now recognized plurals efficiently, significantly enhancing its functionality.

Downloading Websites Now that our search engine had a solid groundwork, it was time to download thousands of websites from the internet. However, this endeavor was not without its challenges. I acquired a file containing a million of the most visited domains, prioritizing results based on popularity. Web scraping, the common method utilized, involved fetching websites, grabbing their links, and recursively scraping those pages. But a single wrong function import threw us into a half-hour debugging frenzy before realizing the mistake.

Web Scraping Challenges As I progressed with basic web scraping and ditched my internet distractions, I became aware of the need to scrape websites without getting banned. A VPN came to the rescue, automatically changing my IP every five minutes. However, a heap allocation error reared its head due to the number of simultaneous requests being made. My initial solution involving queues and promises proved ineffective. After some trial and error, I realized the flaw in my assumption and addressed the issue using a stack overflow approach.

VPN and IP Changes Thanks to the VPN, the retrieval of approximately 3000 sites resumed without crashes. The scraping process started, offering a tantalizing glimpse of accomplishment. But as with any project, nothing runs entirely smoothly. Soon enough, we encountered a bug in the no GS module, resulting in crashes after a short period. Determined to find a solution, I turned to the JavaScript community for help.

Handling Heap Allocation Error Debugging became a crucial part of the process as we tackled the persistent crashing issue. Initially, the culprit seemed to be the infinite recursive calls. But as it turned out, a bug in no GS caused concern. In fact, adding a URL string before another string inexplicably appended a random string from the URLs' end. With a touch of frustration, I sought assistance from the JavaScript community, ultimately uncovering the bug's connection to memory and read stream.

Dealing with Crashes Our search engine journey encountered another hurdle when most of the requested websites returned erroneous responses. It became apparent that these sites were protected by cloudflare and had activated bot-fighting mode. Sadly, there seemed to be no workaround, leaving us with another setback. Thankfully, we weren't YouTubers and didn't have to deal with the repercussions.

Conclusion In conclusion, building a search engine from scratch is a challenging endeavor. From building the server side to tackling web scraping complexities, each step brought its own share of obstacles and solutions. While JavaScript proved to be inconvenient at times, perseverance and resourcefulness led to overcoming hurdles. With this knowledge, we approach future projects armed with practical insights and an understanding of the intricate world of search engine development.

Highlights:

• The journey of building a search engine from scratch
• Challenges faced in server-side development and solutions found
• Extracting keywords and assigning priority for query matches
• Handling plurals using downloadable dictionaries
• Downloading websites and overcoming web scraping challenges
• Effectiveness of VPN and IP changes in avoiding bans
• Dealing with heap allocation errors and crashes
• The bug discovered in the no GS module
• Challenges of cloudflare protection and bot-fighting mode
• Lessons learned and the future of search engine development

FAQ:

Q: Is JavaScript the best choice for building a search engine? A: While JavaScript has its advantages, it may not always be the most suitable option for certain tasks within search engine development.

Q: How did you handle website scraping without getting banned? A: By utilizing a VPN and regularly changing our IP, we were able to navigate the web and scrape websites while minimizing the risk of getting banned.

Q: Did you encounter any other major challenges during the development process? A: Yes, apart from handling heap allocation errors and crashes, dealing with websites protected by cloudflare and their bot-fighting mode posed significant challenges.

Q: What were your key takeaways from this project? A: Perseverance, resourcefulness, and the understanding that debugging is an essential part of the development process were the main takeaways from this search engine journey.

Resources:

• Node.js
• JavaScript
• Cloudflare