• The Curse of the URL Shorteners: How Safe Are They?

    URL shortening services have become all the rage on the Internet. These services take a long URL as input and produce a short, easy to use, URL as an output. Simple! By virtue of their ease of use, millions of Internet surfers use them to post messages on twitter. In fact, URL Shortening services like bit.ly have garnered so much attention that even giants like Google and Microsoft have jumped onto the URL shortening bandwagon.

    Case in point:

    These URL shortening services are godsend for Internet surfers tired of copying and pasting long, ugly looking, URLs. But hold on a minute! All is not hunky dory in URL Shortening Land.

    Due to processes inherent to “URL Shortening,” the original URL an Internet surfer might like to shorten is, for all purposes, being obfuscated. Is this a problem? Yes. Why, you ask? Consider the fact that people, not even necessarily tech-savvy ones, have learned to double check the links present in their emails and on websites. They even have help from various browser plugins, but in general, users are smartening up. When these same people see “shortened” links, they have no way to make a judgment call on whether visiting the link is safe, or not. For example, you may recognize www.stopthehacker.com as being a benign, safe to visit link, but what about bit.ly/oJMrP or bit.ly/dc38ze?

    Articles published from credible sources, like ISC SANS, show that URL shortening services, when compromised, can provide an excellent mechanism for malicious hackers to infect unsuspecting visitors. Criminals use these services to bypass Google’s Safe Browsing service, which is used by popular browsers.

    To combat this growing menace, URL shortening services have partnered with security companies to identify malicious URLs and websites. Some of them even use the SURBL blacklists to identify if someone has tried to link to a malicious website.

    This article attempts to identify the effectiveness of security measures put in place by the various URL shortening services.

    This experiment answers the following questions:

    • Do URL shortening services have any kind of security measures in place?
    • How effective are these security measures?

    The 25 URL shortening services evaluated in this article are listed below:

    We compare 25 URL shortening services listed below. Each URL shortening service is analyzed to measure the effectiveness of their security measures. We use a two stage process to evaluate the security implemented by each service.

    snipr.com
    budurl.com
    bit.ly
    short.to
    twurl.nl
    chilp.it
    fon.gs
    ub0.cc
    snurl.com
    fwd4.me
    short.ie
    a.gd
    hurl.ws
    kl.am
    to.ly
    hex.io
    tr.im
    cli.gs
    urlborg.com
    is.gd
    sn.im
    ur1.ca
    tweetburner.com
    tinyurl.com
    snipurl.com
    

    Experiment methodology:

    An initial corpus of 932 websites was obtained from Malware Patrol a well respected source of information about malware infected websites, which receives nearly 3,500,000 hits/month. This experiment was conducted between February 2nd and February 4th, 2010.

    For each URL obtained from Malware Patrol, we attempt to create shortened URLs for each site domain and full URL using each of the 25 services.

    We denote a service as Stage 1 Compliant if it appears to use a security service or blacklist to identify malicious domains and does not allow a user to create a shortened link to any infected domain. Does the URL shortening service allow a user to create a URL pointing to a malicious domain (e.g. http://www.badsite.dom)?

    We denote a service as Stage 2 Compliant if it uses a security service or blacklist to identify malicious domains and does not allow a user to create a shortened link to any infected domain or malicious full URL hosted on that domain. Does the URL shortening service allow a user to create a URL pointing to a malicious link hosted on a malicious domain (e.g. http://www.badsite.dom/badfolder/badfile)?

    We present the most interesting results in brief:

    • Approximately 68% of URL shortening services were Stage 1 Compliant.
    • Approximately 56% of URL shortening services were exclusively Stage 2 Compliant.
    • Approximately 52% of URL shortening services were both Stage 1 Compliant and Stage 2 Compliant (see graph below).

    Observations on specific URL shortening services:

    • bit.ly seems to favor blocking malicious domains rather than specific links.
    • fwd4.me, hurl.ws and urlborg.com seem to favor blocking malicious links rather than specific domains.
    • bit.ly failed to qualify as Stage 2 Compliant due to 0.5% of tested URLs.
    • fwd4.me failed to qualify as Stage 1 Compliant due to 9.8% of tested URLs.
    • hurl.ws failed to qualify as Stage 1 Compliant due to 0.3% of tested URLs.
    • urlborg.com failed to qualify as Stage 1 Compliant due to 0.3% of tested URLs.

    Venn Diagram depicting URL filtering capabilities of URL shortening services. Only about half of the most popular URL shortening services are effective at blocking malicious URLs.

    Stage 1 Compliant and Stage 2 Compliant services:

    budurl.com
    cli.gs
    fon.gs
    hex.io
    is.gd
    kl.am
    sn.im
    snipr.com
    snipurl.com
    snurl.com
    to.ly
    tr.im
    ub0.cc
    

    Deeper security issues remain:

    It seems that popular services like bit.ly, which do try to use blacklists in order to prevent malicious hackers from using their services and pointing to bad websites, can still be easily fooled by chaining together shortened URLs created by another service. We have found that if a malicious user can create a shortened URL using a service that does not implement blacklist checks or is not effective, then a service like bit.ly can be tricked into redirecting the visitor via the malicious shortened URL to a malicious domain. Effectively, users can be redirected to a malicious site regardless of bit.ly performing all its checks. See the appendix for an example below (wget log).

    Conclusion:

    This limited experiment shows that URL shortening services have a long way to go before Internet users can trust them to deliver safe links. About half of the most popular URL shortening services seem to be somewhat effective at blocking access to well known malicious URLs that can be found on blacklists. It remains to be seen if these URL shortening services can improve and provide a safer web experience for their users.

    Appendix

    Wget log example:

    In this example, a malicious link (hxxp://wywg.ccsfyb.cn/wywg/txer) has been shortened using ow.ly (hxxp://ow.ly/Zyv3). Then, this shortened URL is fed to bit.ly. The shortened bit.ly URL (hxxp://bit.ly/5s4YhP) is created successfully and blacklist checks are no longer effective.

    $ wget -O demonstrate_bit.ly_exploit http://bit.ly/5s4YhP
    --scrubbed--  http://bit.ly/5s4YhP
    Resolving bit.ly... 168.143.174.29, 128.121.234.46, 128.121.254.129, ...
    Connecting to bit.ly|168.143.174.29|:80... connected.
    HTTP request sent, awaiting response... 301 Moved
    Location: http://ow.ly/Zyv3 [following]
    ---scrubbed--  http://ow.ly/Zyv3
    Resolving ow.ly... 75.101.155.42
    Connecting to ow.ly|75.101.155.42|:80... connected.
    HTTP request sent, awaiting response... 301 Moved Permanently
    Location: http://wywg.ccsfyb.cn/wywg/txer [following]
    ---scrubbed--  http://wywg.ccsfyb.cn/wywg/txer
    Resolving wywg.ccsfyb.cn... 98.126.11.178
    Connecting to wywg.ccsfyb.cn|98.126.11.178|:80... connected.
    HTTP request sent, awaiting response... 301 Moved Permanently
    Location: http://wywg.ccsfyb.cn/wywg/txer/ [following]
    ---scrubbed--  http://wywg.ccsfyb.cn/wywg/txer/
    Reusing existing connection to wywg.ccsfyb.cn:80.
    HTTP request sent, awaiting response... 403 Forbidden
    -scrubbed-- ERROR 403: Forbidden.
    
      • Social comments and analytics for this post…

        This post was mentioned on Twitter by hkrnws: URL shortening services: not secure at all http://bit.ly/d421Dm

        Posted by uberVU – social comments on February 19th

      • I tried to implement everything in your article: Malware check, short url double dipping check.

        hxxp://séó.com/

        Posted by Matthew Callis on February 19th

      • Yes, obfuscating the URL does have some negative implications. But since we are doing this anyhow, why not make it short, suspicious and frightening?

        hxxp://www.shadyurl.com/

        ceo

        Posted by C. Enrique Ortiz on February 19th

      • [...] than the shortened URL will consistently refer to the same resource; that the reference cannot be hijacked and the service provider will remain in business (see shuurl.com). These problems are not within [...]

        Posted by Can You Trust That Web Site? (URL Shortener edition) « Aggressive Virus Defense on February 19th

      • I wonder how http://safe.mn/ would compare. it is a URL shortener focused on security. One of the check is done against the Malware Patrol list.

        Posted by Julien on February 19th

      • SnipURL has had peek.snipurl.com (and variants: peek.st.im) for many years. This allows the clicker to see where the directing will happen to. E.g., http://peek.st.im/new-york or http://peek.snipr.com/new-york

        Their spam control is fabulous too, since 2006. Just paste the stuff online: http://snipr.com/site/reportspam

        And naturally they not only check for malware etc through the common dbs such as the safebrowsing API from goog.

        There’s a pretty good reason the team has been around since 2001 I suppose. I’m an avid fan.

        Posted by Eric on February 20th

      • Wow! never thought it was so easy to fool bit.ly

        Posted by anon on February 20th

      • [...] starts pre-filtering malicious links, as we have shown in another article that services like URL-shorteners are a thorn in the side of security policies. As new services like posterous come alive in the [...]

        Posted by How secure is posterous’s posting policy – stopthehacker.com – Jaal, LLC on July 14th

      • [...] http://www.stopthehacker.com/2010/02/19/analyzing-url-shorteners/ http://blog.mxlab.eu/2009/07/17/shortened-urls-the-real-dangers-behind-and-how-to-avoid-troubles/ http://www.pcworld.com/businesscenter/article/184677/url_shortening_frenzy_comes_with_security_risks.html [...]

        Posted by Bit.ly and URL Shorteners Assist With Email Spam | Stop Wordpress Hackers on November 12th

      • [...] http://www.stopthehacker.com/2010/02/19/analyzing-url-shorteners/ [...]

        Posted by [MonoURL] The Curse of the URL Shorteners: How Safe Are They? « Skull’s blog on March 3rd

      • Has anybody tried this one : http://32go.co – it looks quite good safety wise but I’ve never heard of it before so I don’t know how reliable it is.

        Posted by Max on June 3rd

      • [...] http://www.stopthehacker.com/2010/02/19/analyzing-url-shorteners/ [...]

        Posted by On Link Shortening Services OR Spammers Are Winning : Hunter Dyar on July 23rd