Get webpage contents from while setting the User Agent. The URL is requested to the server and the server responds back like the request was sent via browser or crawler.
There are two alternate ways to set your default user agent using php ini_set
function or via .htaccess
file.
ini_set('user_agent', 'Opera/9.80 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/12.16');
php_value user_agent Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
Googlebot (Desktop) | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
Googlebot (Smartphone) | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
Googlebot News | Googlebot-News |
Googlebot Images | Googlebot-Image/1.0 |
Googlebot Video | Googlebot-Video/1.0 |
Google AdsBot | AdsBot-Google (+http://www.google.com/adsbot.html) |
Google Apps | AdsBot-Google-Mobile-Apps |
Google AdSense | Mediapartners-Google |
Bingbot | Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) |
MSNBot | msnbot/2.0b (+http://search.msn.com/msnbot.htm) |
MSNBot-Media | msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) |
AdIdxBot | Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm) |
BingPreview | Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b |
YandexBot | Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) |
Baiduspider | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) |
Yahoo! Slurp | Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
DuckDuckBot | DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html) |
Joydeep Deb is a Senior Digital Marketer and Project Manager with strong experience in Digital Marketing, Lead Generation, Online Brand Management, Marketing Campaigns, Project Management, Search Engine Optimization (SEO), Search Engine Marketing (SEM), PPC, eMail Marketing, Web Analytics, Web Technologies, Web Design and Development.
With an MBA in Marketing. IIM Calcutta Alumini. Lives in Bangalore, Karnataka - India.