Too many security products trade on fear, uncertainty, and doubt among customers and the media. At the same time, giving a positive review to a flawed product risks putting people’s privacy and even their safety in danger. This is especially true for virtual private networks, or VPNs. To review these products, we consider a VPN’s security, performance, and features to produce reviews that are factual and useful to our readers.
This is harder than it sounds. If we relied only on our hands-on observations, we’d miss the features that make each service unique. If we relied entirely upon objective measurements, it would be trivial for a vendor to game the system by inflating particular stats. Combining the two—subjective observations and objective measurement—is messier, but leads to better and more comprehensive analysis.
Our readers may not always agree with our conclusions, but we strive to include enough information in our reviews that readers could form their own opinions, too. In fact, we encourage them to do so.
A Note About Ethics
In an era of fake news, phony reviews, and mounting concern over pay-for-play content, we believe it is important for readers to understand how our company earns money and how our reviews are written. At the top of every review on PCMag—VPN or otherwise—is the following statement:
PCMag reviews products independently, but we may earn affiliate commissions from buying links on this page.
In practice, this means that PCMag may earn a commission either from the company whose product has been reviewed, or some other entity. It’s a common practice among review sites. This is entirely separate from our editorial process, and handled by a completely different staff. By design, reviewers do not have any knowledge of the specific ways in which a particular review is monetized. Nor do reviewers receive a cut of that monetization. Reviewers, full-time or freelance, are paid for their work and do not earn a commission for the reviews they produce.
Importantly, companies, even those who have affiliate relationships with PCMag, do not dictate the outcome of reviews. Our reviewers value their reputations, and we would not stake them on what amounts to bribery. Moreover, PCMag’s reviewers are held accountable by a code of conduct that explicitly forbids accepting any gift of significant value from vendors.
Why are we repeating our code of ethics here? First of all, we’re proud of it, and it bears repeating. Second, it seems important to point out for the VPN space. Security software in general attracts readers that are extremely concerned with this sort of thing. The VPN market in particular seems to be awash with suspicion, some of it the result of paranoia, some created by pay-for-play “review” sites, and some—purportedly—stoked by VPN companies themselves. Rest assured, we hear it when readers worry about the ethics of VPN reporting, and it’s important to us, too.
In short, our affiliate relationships have no effect on our testing process.
Pricing and Plans
Keeping in mind our readers’ concern with price, when we review VPNs, we always opt for the monthly package and report that price in the reviews. We feel this gives the best apples-to-apples price comparisons. While nearly all VPN services will offer a discount when you sign up for a long-term subscription, our goal is to report the base-level price for each service and not the discounts you might get for signing up for a year.
Another reason we choose the monthly plans is because we want to encourage readers to start with a short-term subscription for a VPN. Too often we have received emails from readers who spent $60 (or more) on a year-long subscription to a VPN only to discover it doesn’t work for them. It’s far better to try out a service for a month or three, and decide to spring for a long-term, discounted billing plan only when you’re certain you want to keep it. Consider the extra money you spend up front to be a down payment against buyers’ remorse.
With more and more VPN services popping up, companies have begun adding more and varied features to their offerings in order to stand out. In PCMag’s review of each service, we strive to report as many features as we can but to focus on the ones we believe are the most significant and reflect the value of a service. The number of devices the VPN service allows you to connect simultaneously, for example, is a concrete measurement of value and a point we always report.
In our reviews we also note whether or not a service provides ad-blocking or malware blocking with their service, and if the VPN allows BitTorrent or other P2P services on its network. We do not test the efficacy of ad blocking or malware blocking, partly because it is not part of a core VPN product but also because we feel stand-alone tools address those concerns better.
We also check to see if Netflix is accessible while connected to a US server on each VPN service. It’s not unusual for streaming services to block VPNs, as it allows them to enforce regional licensing agreements. Star Trek: Discovery, for example, is available on Netflix outside the US, but you’ll need a CBS All Access account to watch it in the US. Does this VPN work with Netflix? is a question that anyone who writes about VPNs hears all the time.
Each VPN review also notes the most significant add-ons available from a VPN service. These usually include static IP addresses, additional simultaneous connections, and so on. We generally do not test these add-ons, and instead focus on the core product being sold to consumers.
Server Numbers and Distribution
Our VPN reviews include a current count of the number of servers provided by the VPN company. The number of servers can give you a rough sense of how robust a VPN service is. That’s because with more servers available, the company can connect fewer people to each server. That means a bigger slice of the bandwidth pie for each person assigned to a given server.
This figure, however, is just part of the story. Most VPN companies spin up new servers to meet demand as necessary, causing the precise number of server to change often. It also doesn’t make sense for a small company with only a few thousand subscribers to have as many servers available as a company with a million subscribers. A company might also seek to inflate their server count by using numerous virtual servers, which we explain in greater detail below. We try to balance these considerations in our reviews.
In addition to the number of servers, we also look at how many different server locations are available and how widespread those locations are. We call this “geographic diversity,” and give preference to services with lots of servers in many different parts of the world. This is particularly important to frequent travelers and users overseas, since a VPN server closer to their computer will likely mean a faster and more reliable connection. For users in the US, more VPN server locations mean more opportunities to spoof your location.
We do not test each and every connection to ensure it is functional. This is one of the places where we must assume companies are telling us the truth about their products. However, we do investigate if we find a server is unavailable during testing.
Most VPN companies offer servers in Asia (sometimes excluding China, as explained below), Australia, Canada, the US, and Western Europe. Better services include a few servers in Africa, Eastern Europe, the Middle East, South America, and Southeast Asia. We give preference to services with robust offerings in Africa and South America, two areas generally underserved by VPN companies.
Unsurprisingly, there are limitations to this information, as well. For the sake of clarity, we have not noted a difference between rented servers and servers owned outright by the VPN company. We don’t generally penalize services for relying on so-called virtual servers. A virtual server functions like a server in a given country without having to physically be in that country. We value VPN services that make clear where servers are actually located, and prefer services that minimize their use of virtual servers that appear to be in locations other than their host machine. This won’t matter to most people, but if you’re concerned about having your data in a specific region, knowing which are legit servers and which are virtual can be an important differentiator.
VPNs and Censorship
VPNs are valuable tools for everyone, but especially people living or working in countries whose governments have chosen to restrict information and punish dissent. Given those stakes, we believe it would be unethical to choose a service that would be “best” for circumventing censorship. We will not, for example, write a story about the Best VPN for China.
Additionally, we do not believe our testing is currently adequate for making this determination in the first place. It is our hope that experienced organizations will come forward to identify technology and practices that can be used to safely evade censorship without putting individuals at risk.
We appreciate and elevate companies that contribute to a free and open society. We also note as to whether or not a VPN company offers servers in countries with particularly restrictive internet policies. Our understanding is that connecting to one of these servers from within the country will not circumvent censorship, but would provide some modicum of privacy and security to the user—particularly for visitors to the country.
There is a false dichotomy in digital security between a product’s ease of use and the value it provides. We frequently see commenters dismiss a product (generally one they have never used) as worthless because it looks pretty. That ignores a very real truth about humans: no one is going to do a difficult or annoying product just because it might protect them from hypothetical threats. A well-designed security product that average users can actually use is better than a perfect security tool that is only accessible via the command line.
When we review VPNs, we go through the setup process for each service. We also take time to poke around settings, and see how easy it is to perform certain functions. It’s important that readers, like you, have a sense of what using a given product will be like from reading our reviews.
Sometimes, an excellent user experience takes a mediocre product and makes it better. Sometimes, a poor user experience undercuts the value of an otherwise stellar offering. In general, we place great emphasis on a product being easy to use and accessible to users with all levels of experience. At the same time, we cannot deny the importance of technical excellence, especially when it’s combined with value.
VPN is a mature technology, but it’s hardly a static one. There are several different means for creating a VPN connection (you can even create your own VPN), but not all of them are equal.
In our reviews, we give preference to the services that offer OpenVPN. This open-source protocol has been picked over by volunteers, helping to quickly find and fix potential issues. It also has a reputation among professionals for providing better speeds and more reliable connections. IKEv2 is another good choice, as it uses newer and more secure technology than older protocols.
The other protocols out there are either older or held in less high regard. With these availability of excellent tools, it’s a mark against a VPN company if they can’t offer them.
In the future, we expect the WireGuard VPN protocol to be increasingly important. For now, we consider it to be an interesting bonus, but not a mainstream technology that’s ready for mass use.
Some services, such as VyprVPN, have started to deploy their own VPN protocols. Most are built on established tools, so it’s not as foolhardy an idea as rolling your own encryption protocol (looking at you, Telegram).
To properly evaluate proprietary VPN protocols requires resources and expertise well beyond our means. Similarly, we have no way to evaluate how companies have implemented existing, established VPN technology. We must therefore rely on the work of independent security researchers to uncover bad practices. As a rule, we assume that VPN vendors are good actors, operating in good faith, until we have reason to believe otherwise.
VPN Speed Testing
Most of our readers seem concerned with the security of a VPN, but also the impact on internet speeds. That’s understandable, since most VPNs increase your latency and slow your overall internet connection. Why this happens is simply a product of taking your internet traffic and running it through extra steps.
In late 2018, PCMag moved its offices. This had several unavoidable consequences for our VPN speed testing. First, a change in physical location impacts speed test results, especially when a VPN is active. Second, our new office provides a gigabit FiOS connection for testing—far more bandwidth than was available to us in the past. More bandwidth is always good, but it does mean that our testing represents ideal conditions, and not what the average consumer is likely to experience.
Because of this new network connection, we found that the impact from VPNs on download and upload speeds increased. This is likely because our bandwidth is now far greater than anything a VPN can provide. We have always cautioned readers that our speed test results are best used for comparison and are likely to differ greatly from what readers experience. This more true now than ever, but we also believe this upgraded testing environment allows us to better measure the best possible service a VPN vendor can provide.
To find the fastest VPNs, we compare median test results from Ookla with the VPN active, and then when the VPN is inactive, in order to find a percent change. The Ookla test returns results for latency, upload speeds, and download speeds, so those are the metrics we use as well. We run the Ookla tool ten times, with the VPN on and ten times with the VPN off and then take the median of these results.
Note that Ookla is owned by PCMag’s publisher, Ziff Davis.
This methodology differs from previous years. In the past, we performed five tests, discarded the highest and lowest values, and then took a mean-based average of the remaining three. We then found a percent change from these averages. We moved to this new methodology because it greatly simplified our processes and preserved the sometimes-mercurial results we observed.
Given the significant changes in technology and methodology, we believe it would be unfair to compare between results gathered under the new and old testing regimens. As such, we have re-tested all of the VPN services that PCMag had already reviewed as of December 1, 2018. All of this testing was performed on a Windows PC. This ensures that all Windows VPN app comparisons can be made within the same context.
These tests have limitations. Issues with the internet connection we use in testing could affect the results. For this reason, we perform speed tests in between results and compare it to historical speed data for our specific connection, to ensure that it is operating within expected parameters.
Background system processes on our test computers and smartphones could muck up test results. We strive to avoid this as much as possible.
Most significantly, despite gathering numerous test results, it is still only a single point of data and not enough to give a definitive judgment on a service’s overall network performance. Consider that when PCMag does our Fastest Mobile Network survey of wireless providers, we test constantly over the course of several days and across several states. To create a truly accurate picture of VPN performance, we would have to replicate the scope and scale of that testing, which is far more expensive, far more time-consuming, and requires tools that currently do not exist.
Additionally, VPN performance may depend greatly on the VPN server you connect to. Some VPN apps have fine-grained server selection tools that let you choose a specific server over and over again, but not all of them do. When testing, we grant our analysts the leighway to halt and restart testing if the results appear anomalous. Analysts may also specify a specific VPN server, if they feel the VPN app’s choice of servers is causing anomalous results. All testing for a VPN service is carried out in a single session. When a disruption requires an analyst to stop a test and return a different day, the previous results are discarded and the test begun fresh.
Because of these limitations, PCMag presents its speed testing not as the final word in a VPN’s performance but instead as a snapshot. It is meant to say that at this given day and time, this VPN performed this way.
Trust and Privacy
When its product is active, a VPN company has the same level of insight as your ISP has into your online activities. Because of that, it’s important that you trust the VPN company you sign up with, and that you are comfortable with the potential pitfalls using a VPN might entail.
The steps that VPN companies take to protect your information vary. Some, like Private Internet Access, issue users a semi-random username, as part of an effort to obfuscate individual identity on their service. Others operate under legal jurisdictions that allow them to avoid retaining information, or handing it over to law enforcement. For example: ProtonVPN operates out of Switzerland, and NordVPN is under the legal jurisdiction of Panama.
We feel it would be overly reductive and unfair to definitively state that a company does or does not do a good job protecting user privacy. Instead, we present the facts as we have them along with our analysis and the context gathered from over 30 VPNs. We will always say what we think, but a reader whose priorities differ from ours will have the information to make their own decision.
Some consumers refuse to use VPNs based in the US, out of concern that these companies will be compelled to hand over information to law enforcement. It is worth noting, however, that the US does not have any mandatory data retention laws that would require companies to keep certain information on hand. The UK, however, does have such rules.
Some VPN companies operate out of Hong Kong and may be subject to government pressure to which we are not privy. Other companies may list their offices as existing in one country, but actually operate out of another. We think this information is important to report and include it in our reviews. We also acknowledge that the legal frameworks and ethical practices of other countries vary widely.
We are also wary of xenophobia in the guise of seeking the best and most secure option. Rumormongering is not unheard of in the security industry, nor is using baseless fears over race, class, and other factors. For example: China and Russia have been accused of numerous cyberattacks against the US, and are known for fostering oppressive environments domestically. Because of this, some consumers refuse to use security products from these countries, believing that they are inherently compromised. By the same token, the US government is responsible for the largest and perhaps most intrusive intelligence gathering operation in the world (if the information from Edward Snowden is to be believed), and has even intercepted domestically made products in transit to install malicious software. Yet US products are often regarded as more trustworthy—by US customers, at least.
For the time being, we are hesitant to penalize a product for its country of origin alone. Instead, we present the information we collect, provide context, and encourage readers to make their own choices. In the future, we hope to develop tools for our readers and reviewers to better assess the track record of individual companies and countries for security and privacy.
Our testing assumes that the VPN companies we review are good actors, operating in good faith. We rely heavily on the work of security researchers who have unmasked some of the worst behavior among VPN providers, and on the robust security community that is quick to point out the flaws in any product. We also value efforts toward transparency by VPN vendors, by responding to our questions, by publishing the results of third-party audits, by disclosing interactions with law enforcement, and by participating in accountability evaluations such as those provided by the Center for Democracy and Technology.
PCMag freely admits that this is inadequate. What the VPN industry needs is similar to what is already common for antivirus products: robust third-party testing and easily deployed tools to confirm that the product is performing as advertised. We hope to encourage the creation and dissemination of such tools and standards in the future.
DNS and IP Leak Tests
In 2019, we’ve opted to include testing to see if VPN services leak IP address or DNS information. To confirm that the IP information is not revealed, we use the Ookla speedtest tool to see if the IP address and ISP identification information changes when the VPN is in use.
We validate that information and check for DNS leaks using the online DNS Leak Test tool. The provides a list of which DNS servers are being used. By comparing the list with and without a VPN, we can see if DNS information is leaking out.
VPNs Beyond Windows
Smartphones have, for the most part, replaced laptops as the go-to mobile digital device. Instead of just connecting laptops to unsecured Wi-Fi networks, you now connect your phones and tablets, too. So, as you might guess, we believe that evaluating VPNs on mobile platforms is important.
In general, we review mobile VPNs the same as desktop VPNs. Most of the features are the same, and we do take pains to highlight the differences. While our desktop VPN speed tests are carried out via wired Ethernet connection, the mobile speed tests are done over Wi-Fi. During these tests, we deactivate the cellular radio in order to reduce variables that could affect the results.
One major difference between mobile VPNs and desktop VPNs is the selection of available protocols. It’s much more common to find OpenVPN on Windows and Android, less likely in macOS VPNs, and outright rare on iOS. That’s because Apple requires developers to jump through additional hoops if they want to use OpenVPN in their iOS app, or in an app they want distributed through the macOS App Store.
Fortunately, more developers are taking the effort to include OpenVPN in their iPhone VPN apps. We try to reflect that extra effort in our reviews. However, this will hopefully no longer be a differentiator in the future and including OpenVPN will be the norm instead of the exception.
Mobile devices have a different design language, since you interact through a touch screen instead of a keyboard and mouse/trackpad. A successful mobile product will be visually and functionally similar across all platforms, but tailored for each—whether it’s a VPN for Android or iPhone. We see ease of use is a major criterion of an excellent VPN, and this is especially true for mobile VPN apps.
The Evolution of VPN Testing
At PCMag, we strive for reviews that are meaningful, based on testing that is reproducible. We eschew including unnecessary information or meaningless testing data that would confuse, rather than elucidate, the reader. Walking this line is always a tradeoff, and for VPNs it is no different.
As always, we will adapt and improve our testing as well as our reviews as the products change, but also the landscape around them. Perhaps a new technology will completely upend what makes a VPN worthy. Whatever the case, the VPN reviews you read here on PCMag will always be as accurate and useful as we can make them.
Tech Savvy, Enthusiast, Graphic Designer (Aspiring WEBDEV), Samsung/Pixel Lover, Occasional Blogger – Business -Family Man… Can help on Tech-related issue because is a passion to me