Checking User Agents for problems

User Agents are just "free-text" HTTP Header fields; there is absolutely no standard format for them and any software that makes web requests (browsers, bots, scripts, malware etc) is free to send it's user agent string in whatever form it likes.

And while web browsers and bots send their user agents in a wide variety of ways, most are still fairly predictable... But when crawlers, scrapers, badly written scripts, and attackers come into the picture, things get pretty wild, very quickly!

Being able to spot these oddities can help you identify risky behavior on your site and help reduce fraud and security problems.

Learn about the ways we detect that there is a problem with a user agent:

User agent checking

It's likely that most user agent strings you've seen follow a pretty standard format; they open with a Mozilla/5.0 fragment, and then maybe some Windows NT or Android or Macintosh fragments, then a Chrome/, Firefox/ or Safari/ fragment and so on. Or perhaps there's an obvious fragment like Googlebot/2.1.

In other words, while fairly "standard" user agents are still a bit of a mess, they're not that crazy.. right?


You'd be forgiven for thinking that, but actually, User Agent Strings can come in a literally unlimited number of formats, and since they are just strings of text, often generated by malfunctioning or incorrect code (at best) and malicious and dangerous code (at worst), if you are doing anything with user agents, it's important to handle them correctly.

You can also use user agent to glean extra information about who or what is sending the request; this can help you identify traffic which should be blocked or at least investigated further.

All of the paid API plans get full access to these fields to help you identify many, many types of problems with the user agents accessing your systems.

Find out about how we sanitize user agents

As well as checking them for problems, we also sanitize/normalize/fix user agents too.

Stay safe and secure

Our user agent checks help you and your systems stay safe and free of junk user agents.

Use the API to identify malicious user agents, weird user agents, restricted user agents, and user agents with spam in them.

Get started now

The API is free to use and easy to set up, so why not get started right now.

Do you have a question? Get in touch! We'd love to help you.