I decided to scan the top 25K websites on the internet The purpose was to see who all were still implementing the meta keyword tag. As I started crawling these websites the experiment became bigger and bigger. Below I present some of my findings.
I used the top 25K list from Quantcast. Approximately 1120 sites had a hidden profile so I could scan only 23880 websites.
1) Response times
Approximately 1352 websites failed either a DNS lookup or the connection timed out (slower than 20 seconds).
Here is a table for the response times of these sites.
RESPONSE TIME (SECONDS)
URL ( TOTAL)
% OF TOTAL.
1
9170
38.4
2
7380
30.9
3
3393
14.21
4
1218
5.1
5
818
3.43
6
196
0.82
7
65
0.27
8
44
0.18
9
187
0.78
10
31
0.13
Approx 92% of the sites opened up in the first 5 seconds.
less than 1000 websites were above 5 seconds.
Response codes
5.6% (1352 took longer than expected or had no response.
63.85% had 200 status ok. (2xx)
28.57% used a redirect. ( 3xx )
1.68% (402) websites had a not found error (4xx)
0.23% had a server error (5xx)
Bonus Analysis on analytics software.
I also crawled their web page to find out how many were using Google tag manager. I also checked for others such as
Google Tag Manager
Google Analytics
Doubleclick
Marketo
Adobe Analytics
This is a relative percentage markup as not each site was tracked or followed. Approx 13K sites provided some sort of a js code to detect what system they were using.
