User Agent String best practices

Are you programming a web browser, web crawler or some other software that makes web requests? Here's a guide on making sure your user agent serves you, your users and the internet well.

  • Don't put actual links ("<a href...") in the user agent.

    Lots and lots of spam bots do this; some websites output a list of "recently seen" user agents on their site, so spam bots have tried to game this by putting links they want visited (in order to drive up the Google PageRank) in the user agent.

    It is a dead give away that you're not being legitmate; in fact, the parser/API will always detect a user agent like this as being abusive.

    Don't include links with the "a href"

    However it's very common for bots (crawlers, analysers etc) to include a link to an informative page about the bot, so that curious webmasters can find out more... just don't enclose that link in anchor tags.

  • Include the full version number

    Always include the full version number of your software, don't abbreviate it to the major version.

    Firefox breaks this rule, so instead of reporting 50.1.2 in it's user agent, it always reports 50.0. The rationale the developers give for this is that by showing your full version number it somehow helps attackers know if your browser is vulnerable to a particular exploit; however this doesn't really make sense. If you were malicious, you could just attempt the attack regardless and it would either work or not.

    The problem is that it prevents sites like from determining if you're actually up to date or not. As such, we've had to scale back our version checking for Firefox - because now we only know if you're running the correct major version, not the revision as well.

  • Don't include "bot-like" fragments unless you're really a bot

    Unless your software is a bot or crawler, don't put the fragments "bot", "crawler" or "spider" anywhere in the user agent. Many web masters will apply filtering they want applied to bot software based on whether the user agent contains fragments, and so if you're making an actual "web browser", your users may run into these filters by mistake.

  • Let us know!

    Let know about your new software! We are always happy to add detection for developers who contact us to let us know of the software that they're developing. Please include a user agent (including examples of any variations) along with basic explanations of each and any unique fragments so that we can test that we're detecting it the right way.

    Doing this will ensure that the thousands of companies who use our API will know what software their users are using.