YouGov cuts storage needs by 70% with MongoDB’s data compression

Survey firm uses less capacity for more data with MongoDB 3.0

YouGov has reduced its storage needs by 70 per cent after migrating to the latest release of MongoDB's database.

The polling company collects between 2GB and 4GB of new data every hour, but at peak times the amount can be three times as high, and first turned to the NoSQL database in 2010 to store current and archived data which comprises respondents' answers to questions and their activity during an interview.

But with its recent upgrade to MongoDB 3.0, it used the compression in the database's WiredTiger engine to drastically reduce its storage requirements, which were growing as it captured ever more information.

The 3.0 release allows customers to choose between three engines to power their database - its standard read-intensive engine, a write-intensive engine and an in-memory engine.

WiredTiger powers the write-intensive engine, allowing developers to build applications that deliver between seven and 10 times better write-throughput performance while dramatically compressing data.

Jason Coombs, executive technical director at YouGov, said: "Given that we've moved to SSDs for storage, this saving is fantastic. That feature alone has pushed us to upgrade faster than we would have otherwise."

YouGov now uses MongoDB to power dozens of its applications, but the key system it supports is Gryphon, a survey system the company built itself.

All survey data enters MongoDB via Gryphon, before being sent to one of YouGov's other applications, like brand perception tool BrandIndex or consumer behaviour tracker Pulse.

Gryphon used to sit on YouGov's own database, called FastStore, but it couldn't scale globally, so the firm selected MongoDB five years ago over a rival product also being used within the company based on Microsoft SQL Server.

"That stack could not provide the speed of innovation or performance required by the Gryphon product," said Coombs.

Now, YouGov uses MongoDB Enterprise Advanced, a version of the database that includes advanced software, support, certifications, and licenses. 

It also uses MongoDB Cloud Manager to run, automate and back up its database, benefitting from a cluster of five shards, two of which are in the US and two of which are in Europe.

"It's less about break and fix support, and more about proactive, consultative services and tools such as planning upgrades or advice on schema design for new apps," said Coombs.

Featured Resources

Virtual desktops and apps for dummies

An easy guide to virtual desktop infrastructure, end-user computing, and more

Download now

The total economic impact of optimising and managing your hybrid multi-cloud

Cost savings and business benefits of accelerating the cloud journey

Download now

A buyer’s guide for cloud-based phone solutions

Finding the right phone system for your modern business

Download now

What’s next for the education sector?

A new learning experience

Download now

Most Popular

KPMG offers staff 'four-day fortnight' in hybrid work plans
flexible working

KPMG offers staff 'four-day fortnight' in hybrid work plans

6 May 2021
Hackers use open source Microsoft dev platform to deliver trojans

Hackers use open source Microsoft dev platform to deliver trojans

14 May 2021
How to move Windows 10 from your old hard drive to SSD
operating systems

How to move Windows 10 from your old hard drive to SSD

30 Apr 2021