As we continue our journey through time, in order to better understand how search engines have evolved and how they work nowadays, we take the baton from the article under the title of “Search engines – The history of their evolution”, where we referred to some of the greatest scientists – pioneers of search engines and the gradual evolution of the “web” from a simple link collection to a more sophisticated indexing and search system in the early 1990s.
At the same time, the Robots Exclusion Standard was created, which set standards and conditions for how search engines should and should not index the content of web pages.
It enabled website owners – administrators to be able to specify which type of content they wanted to be indexed by search engines and which type of content they wanted to ignore.
A webmaster could now choose whether to block search engines from indexing their entire website or specific pages of it.
By the end of 1993, three new search engines were created, but none of them had the performance that would allow them to capture the interest of users.
Jumpstation collected titles and headers from pages, indexed them and allowed the user to search them using linear search.
www worm indexed page titles and links, but displayed them in the order in which it had indexed them.
The third, called Repository Based Engineering (RBSE), had no ranking system at all!
The main disadvantage of all three was that in order to search for something, you had to know its exact name!
However, six undergraduate students at Stanford University were building a search engine that would rank results based on statistical analysis of word correlations. This search engine would eventually become known as Excite.
As the internet grew, two other students from the same university (Stanford) had devised a way to use the – then difficult to use – internet, to win in a virtual fantasy basketball league.
Their names , Jerry Yang and David Filo.
They were the creators of what we all know today as Yahoo!
The two students, electrical engineers, used the internet, which was still essentially a collection of files, to find reliable sports information.
The two quickly realized that to use the internet more efficiently, users needed a directory to help them navigate the information scattered across the web.
So, they started to create a directory, in which they manually categorized the websites they found, into categories and subcategories.
Users could browse this directory and discover new information and websites without knowing anything about them (as was required by the previously mentioned search engines)
This directory was called, David and Jerry’s Guide to the World Wide Web, it was the first website where users could more easily find the information they were looking for.
As David and Jerry’s Guide to the World Wide Web gradually gained a lot of popularity, they realized they needed a catchier name.
So, they created Yahoo!
But the name alone was not enough, they needed funding. Today search engines like Yahoo! and Google are financial powerhouses and generate billions, but back then no one had found a way to make money from the internet.
In the early days, the internet was not used for business and commerce, it was just a place for the free flow and sharing of information.
As is always the case in human history with anything new, there were several conflicts about the imminent commercialization of the internet, on one hand there were traders and investors who saw a great economic potential in the internet, on the other hand there were the romantic supporters of a utopian reality of a non-commercial environment.
Dave and Jerry faced the dilemma of accepting funding and therefore, advertising that would help grow their platform, but at the same time, would potentially alienate their users.
Naturally, they ended up accepting the advertising, finding that not only did it not negatively affect their users, but on the contrary, the audience using their platform was constantly expanding.
This attracted the attention of other businesses, which also realized that there is great economic potential in the Internet.
In the meantime, “Excite” upgraded its algorithm and evolved more into what we know today as a search engine.
The competition between “Excite” and “Yahoo!” escalated!
This was of course for the benefit of the users because they were constantly discovering new services and freebies like email to attract and retain more users on the platforms they had created.
Greed was lurking and eventually made itself felt, as most searches did not show results based on users’ searches but on the payment of advertising companies, resulting in many cases in irrelevant or undesirable results.
Users needed a better way to search online.
It was then that a new search engine emerged, by two more students from the same university that helped create Yahoo! and Excite, Larry Page and Sergey Brin, would create the search engine we know today as Google.
Google started with the idea that websites are just part of a huge popularity contest, and the more popular the website is, the more people quote it as a link to recommend it to others.
Therefore, the greater the number of links a website has leading to it, the better it should be for users, and the higher the ranking it should have in their new search engine.
Yes, well you recognized it, we are talking about the well-known backlinks, which even today are one of the most important factors in the ranking of websites in every search.
In their 1998 university paper, they determined that, in essence, Google translates a link from page a to page b as a vote from page a to page b.
Google evaluates how important a page is by the votes it receives. This has become the most important part of Google’s algorithm, known as PageRank.
Back in the day, to make sure your website ranked higher than your competitor’s website, all you needed was more links than your competitor had.
Something that subsequently proved to be easily manipulated and in recent years has been described as black hat seo practice.
So, over the years, Google has modified its algorithm to attribute less value to some things and more value to others.
But in the early days of the internet, before people started taking advantage of the algorithm, this made search results extremely user-friendly.
Users started pouring into the new search engine, and almost brought down Stanford University’s internet.
After that, Larry and Sergey requested to move Google off campus grounds.
In order to continue running the search engine and evolve its algorithm, they naturally needed funding. No one was willing to invest because they felt it was just another search engine.
An investor was introduced, who had also invested in Excite, who tried to convince Larry and Sergey to work with Excite instead of continuing to develop Google.
The two agreed to meet with Excite and at that time, they offered Excite almost a million dollars to buy it.
Excite declined the offer, and the two continued on their own, eventually finding the coveted funding.
But investors were pushing for the commercialization of the search engine, which the creators of Google were not opposed to, but they wanted to differentiate Google from the respective portals of Excite and Yahoo!
They wanted the Google homepage to have only their logo and the results they would provide to their users would be simple and easy to understand without banners and “bright” ads.
They ended up keeping the page ad-free until they could come up with a new creative way, that would not affect the “user friendliness“.
Meanwhile, the founder of a Los Angeles startup, ideaLab, was looking for a solution to the problem of online advertising.
His name was Bill Gross. Bill realized that every time a user entered a query into a search engine, they were telling that search engine exactly what they were interested in and what items they could potentially buy.
This information was extremely valuable for marketers and advertisers.
Bill also realized that search engines could sell this information and specific search queries to prospective buyers.
This would allow businesses to make sure their brand was associated with specific keywords by buying a link to their website every time someone typed a relevant keyword into a search engine.
Canon for example, could pay a lot of money to have itself appear every time someone typed the word camera.
There were several who again disagreed with this idea and claimed that such a thing could never work.
Gross saw it as a new type of yellow pages, where on every page you could find paid ads for whatever you were looking for.
Finally, in 1998 he created overture.com which was an instant success.
Gross’s page immediately caught the attention of Google creators Larry and Sergey, who thought it could be an excellent guide for their own search engine.
They met with Bill and discussed ways to combine Bill’s successful platform with their own excellent search engine.
For reasons that were never disclosed, no deal was eventually struck between the two parties and eventually in 2000 Google launched its own version of paid advertising under the name Adwords, a service similar to the one provided by Overture.
Gross went on the offensive, suing Google for “copying” and the two sides reached an out-of-court settlement, with Larry and Sergey giving Gross a large number of Google shares in return.
Google changed the way it presented ads in comparison to its other “organic” results, and this ensured that they would remain useful and relevant to its users’ searches.
This feature of Google, which it retains to this day, not only set it apart from other engines, it also gave it the largest market share and changed the future of online advertising.
In the following excellent video from the Data Is Beautiful channel you can see in time lapse how the use of search engines has changed over the years.