Did you ever wonder how Google indexes things, and how it valuates websites?

Here is few facts that are not available to public:

How Google is scraping web?

Lets divide Google Spiders (bots to index content of WWW) to 4 main groups.

1) Backlinks Spiders - this creatures are most basic ones, and came out from initial creation of Google. They are somehow Highway builders for other Spiders. They are build like viruses - they will follow link, then they will scrape for more links, and multiply themselves, and follow all links from that page. They love to scrape code and backlinks. They will send back to server information about and backlinks (if they don't see any value in they will scrap some random content, but that will give you ---). If they will catch you on using dead link, they will give you penalty, as you will kill one of them. Google hates when you kill their spiders.
Those spiders have limited ability to reproduce themselves as well (limited by server abilities), that explains why some websites get most pages indexed in one move, and some just 1-2 pages. (it is possible there are 2 spiders - one that scrapes for backlinks, and one for meta, impossible to determine).

2) Content Spiders - they are creation of first Google evolution. They are scraping all websites content. There is some insight that they can actually understand, and recognize text. They will know if article is spun, if article is about something, that is related to your keywords/website. They are "heavy" spiders, and require outstanding amount of resources. They use Backlink Spider highways to move around web. They love constantly changing content. They are so heavy, so they don't want to be forced to scrape same content every time they will enter website. If you will delete one of the links after backlinks spider will index you, but before content will get to you, you need to expect huge damage in your SEO, as they are so heavy to run, that Google will issue penalty when you kill them.

3) Picture Spider - another effect of Google Pictures evolution. They love pictures, and they will actually be able to somehow see some of the pictures (not details, but colours, patterns and will recognize poorly adult content) and read pictures related tags to rank them. They move on highways, and use Content Spiders information to find pictures.

4) Video Spider - another experiment from Google. This smart fellas will do some amazing things. As of now they have no ability to see video itself, so they will use Content Spiders information to find video files, and scrape Thumbnail, tags, rating, and comments, to decide whether this is good video. They have been created in YouTube era (why Google needs YouTube? Test test test).

Enough about spiders. Only Google knows how much more crazy creatures they have.

About Google Dance:

What is Google dance?

Simply - every spider from Google family have power to valuate website. Based on that your website will go up, or down.
Lets say on maximum of 1.000.000 points new website will start having "0" points.
After Backlink Spider visit it has counted that you build a lot of authority backlinks, and you have good Onsite SEO. So this important Spider will give you 257.654 points. You rank on page 2 after this.
Now content Spider. It found no valuable content. All copy-paste. Content Spider do not like this, it will give you -57.567 points. After few days you dropped to page 17 because of that.
Now another spiders came in, and re-valuate your website. Some will give you +, some will give you -. After 2 weeks of dancing around (you deleted some link, get some crappy backlinks, or Xrumer spam from bad list? -12.867 from Backlink Spider) you will get to page 3. Does this make sense now?

What is Google Sandbox/Penalty?

Whenever you will get more minuses, than pluses, you will get hit by Sandbox (for example Google found out that you have duplicate content farm pointing to your site. Backlinks Spider couldn't find out, but content one would). Penalty? If you will get sudden drop of ranking (for example from 300.000 points to 150.000 points) it means for Google you messed up = penalty. Remember that Google works using script. You can say - it was just 100.000 backlinks from free Xrumer list, its not that much! - Google says: SPAM.

Why there is no 1 Spider To Rule them ALL? Well simple answer = cost. They cant afford to have 1 Million servers. They are dividing work between spiders, so servers can handle this (yes, Google is using regular servers too). Then all information is send back to super network. Amazingly huge network creating one big computer (computer "42?"). This network calculates position based on score you get.

How determine if website has authority?

Simple. Position in Google is determined by this factors:

1) Authority
2) Backlinks
3) Content
4) Other less important stuff.

What is authority?

Its simple:

2+3 + 4 = 1



Is Page Rank any important anymore?

Yes and No. Pagerank was original way to determine authority. It is not anymore, as it is using old, and not updated ways to do this. It is using simplest way of 2 + 3 = 1 to determine its value without including +4 and partially new "3" updates in this, so it will be highly inaccurate.


Is No Follow any good?

It is, but not as much as Do Follow, as it will not be scraped by Content Spider, so it will bring less value, and it wont create Highway for other Spiders (they are not allowed to enter it).


What is hot these days?

800+ words article single directory submission (Ezine is still alive, just don't spam articles to 100 directories, as Google now sees this as SPAM!), high AUTHORITY backlinks, Social Backlinks, Viral Marketing, News Feeds, Closed Websites backlinks (ones requiring payment, or difficult to get in to).


Is Google smart?

People behind it - extremely. I worked for Google (few days only) and I had the possibility to see head programmers. I am pretty sure all head programmers are freaking millionaires. Google as a search engine - no. It is just a script. And every script has a loophole. I found 2 exploits last week alone. Just think outside of the box. How to please all Spiders.
I was working for Symantec long time (Norton Anti-virus) and I know for example that they hired CIA guys (some hacking stuff), and young hackers Prodigy to work for them. Google guys for sure too, and they will be pissed after seeing this 


Perfect Website?

Having around 10 links on each page. External links to authority sites that are anyhow under same category. Few videos, and 2+ pictures for each articles. 80% of unique content. Some big citations. Only naturally looking backlinks. Some backlinks poiting pictures. 50% of inner pages have relevant backlinks poiting to it. Maximum of 40% of forum backlinks, 30% of blog backlinks, 20% profile backlinks. Minimum of 30% authority backlinks. Articles/content of various word count. From 300+ all up to 1.500+. Minimum 30-40 words of unique content on every page.


How accurate is this info?

5 years of "no bullshit" tests. I never read any SEO ebook. In my life I read only 2 ebooks (one about HYIP and one about Mastercard today - waste of time BTW), so my mind is clear. No Guru bullshit, or Google blog. Just raw information filtered and exposed.


Why Google wont give this info away?

Would you shoot yourself in foot? It is OBVIOUS they will keep in secret all the juicy info, and feed people with lies.

Disclaimer: Only Google knows whats the story about their Algorithm. This data is just raw expression to help you understand whats going on around. We cant have accurate number of "points", or Spiders names etc. Also sorry for all typos, its late here. Enjoy!  

Thanks goes to ExtraWinner from BHW for this great post.