Online User Agent Switcher
Get webpage contents from while setting the User Agent. The URL is requested to the server and the server responds back like the request was sent via browser or crawler.
There are two alternate ways to set your default user agent using php ini_set function or via .htaccess file.
Set via PHP
ini_set('user_agent', 'Opera/9.80 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/12.16');
Set via .htaccess
php_value user_agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
List of Search Engine Web Crawlers
| Googlebot (Desktop) | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| Googlebot (Smartphone) | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
| Googlebot News | Googlebot-News |
| Googlebot Images | Googlebot-Image/1.0 |
| Googlebot Video | Googlebot-Video/1.0 |
| Google AdsBot | AdsBot-Google (+http://www.google.com/adsbot.html) |
| Google Apps | AdsBot-Google-Mobile-Apps |
| Google AdSense | Mediapartners-Google |
| Bingbot | Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) |
| MSNBot | msnbot/2.0b (+http://search.msn.com/msnbot.htm) |
| MSNBot-Media | msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) |
| AdIdxBot | Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm) |
| BingPreview | Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b |
| YandexBot | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) |
| Baiduspider | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) |
| Yahoo! Slurp | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
| DuckDuckBot | DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html) |
About the Author
Joydeep Deb is a Senior Digital Marketer and Project Manager with 15+ years of experience in Digital Marketing, Lead Generation, Online Brand Management, Marketing Campaigns, Project Management, Search Engine Optimization (SEO), Search Engine Marketing (SEM), PPC, eMail Marketing, Web Analytics, Web Technologies, Web Design and Development.
With an MBA in Marketing. IIM Calcutta Alumini. Lives in Bangalore, Karnataka - India.