Dutch software makes supercomputer from laptop

Two astronomers from the University of Groningen (The Netherlands) developed a software library that can effortlessly generate visualisations based on hundreds of millions of data points. Maarten Breddels and Jovan Veljanoski initially developed their software to handle the enormous quantity of data from the Gaia mission.

However, the software can also show patterns in other large data files. The software is open source, and free to use. The researchers explain the ins and outs in an article that has been accepted for publication in the journal Astronomy and Astrophysics.

Breddels and Veljanoski call their software vaex, which stands for visualisation and exploration of big tabular datasets. The software can generate visualisations of billions of data points in only one second. This makes it interactive. It behaves similarly to Google Maps. When panning or zooming, one needs to wait shortly for an updated or more detailed map to appear. However, Google Maps run on fast, powerful servers, while vaex works on a laptop.

The power of vaex lies in the combination of several smart techniques. First, it utilises a smart algorithm that maximises all available computing power. Then, it read only the required data from the hard disk and sends it directly to the main memory of the computer. Finally, it is extremely memory efficient, and the working memory does not store unnecessary copies of the data.

Breddels has showcased vaex live at several conferences. As an example, he used a dataset consisting of 1 billion entries related to the Yellow Cab taxis in New York City. He shows which taxi rides are the most lucrative, and where the taxis should wait in any part of the day to maximise their profit. This example shows how vaex can be interesting and beneficial for general applications outside of astronomy.

The software, including the documentation and tutorials, is available for free via here

"Vaex: Big Data exploration in the era of Gaia." Maarten A. Breddels and Jovan Veljanoski, 2018. To appear in Astronomy and Astrophysics.

Related Links
Astronomie at NL
Space Technology News - Applications and Research

Tweet

Thanks for being there;
We need your help. The SpaceDaily news network continues to grow but revenues have never been harder to maintain.

With the rise of Ad Blockers, and Facebook - our traditional revenue sources via quality network advertising continues to decline. And unlike so many other news sites, we don't have a paywall - with those annoying usernames and passwords.

Our news coverage takes time and effort to publish 365 days a year.

If you find our news sites informative and useful then please consider becoming a regular supporter or for now make a one off contribution.

SpaceDaily Monthly Supporter
$5+ Billed Monthly

paypal only

SpaceDaily Contributor
$5 Billed Once

credit card or paypal

Microsoft sinks data centre off Scottish island
London (AFP) June 6, 2018
US tech giant Microsoft has submerged a data centre off the Orkney archipelago in northern Scotland in a project to save on the energy used to cool the servers on land, the firm said Wednesday. The Northern Isles data centre consists of a 40-foot (12.2 metre) long white cylinder containing 864 servers - enough to store five million movies - and can lie on the seabed for up to five years. An undersea cable brings electricity, from Orkney's renewable energy network of wind turbines and tidal pow ... read more