Friday, December 16, 2011

IBM Braces for Processing an Exobyte of Data

Perhaps this helps us grasp the computing power available to us now and what we have been waiting for.

My own needs are for one billion converging equations in each cycle in order to properly simulate the exact curvature around and internal to one neutron.  It would be great if it could be done simultaneously to provide real time imaging.

It has been a long wait but I suspect we have gotten there.

IBM braces itself for 'deluge' of data from telescope which outputs an 'exabyte' of data a day - more traffic than the whole internet

Telescope is 10,000 times faster than any radio telescope today

Huge array of dishes connected by optical fibres 

£1.3 billion machine will come to life in 2016

Last updated at 8:37 AM on 6th December 2011

The £1.3 billion Square Kilometer Array is one of the most ambitious science projects ever undertaken - and the biggest telescope every built.

It will consist of dishes spread over a square kilometer of land in either South Africa or New Zealand - and will scan the sky 10,000 times faster than any existing telescope.
Signals received by the SKA will be transferred to a central high performance supercomputer by optical fibres. The rate at which the vast quantities of data will be transferred will far exceed the data rates of current internet traffic - but IBM is already designing machines to digest it.

The Square Kilometer Array will generate so much data that no supercomputer on Earth could deal with it - every day, it will generate an 'exabyte' of data, more than the world's daily internet traffic

Astronomers and engineers from more than 70 institutes in 20 countries are designing the SKA - it will be 50 times more sensitive, and will survey the sky 10,000 times faster, than any other telescope

The Square Kilometer Array will create three-dimensional maps of 'cosmic magnets' to understand how they stabilise galaxies, influence the formation of stars and planets, and regulate solar and stellar activity

The amount of data captured by the dishes is near-impossible to even imagine - enough to fill 15 million of the largest-capacity iPods every day.

Currently, most astronomy projects are managed 'manually' - ie by astronomers using comptuers, but 'picking' which data to home in on. The SKA will generate so much data this will be impossible. 

But IBM has already prototyped software that can 'digest' it -- and says that the technology could allow companies to oversee telecommunications systems and transport networks more effectively than they do today. 

Concept imagery of the Square Kilometer Array - the dishes will be connected by a network of fibre optic cables built to move more information than the entire internet does today

The SKA will scan the sky 10,000 times faster than any other telescope - and will help scientists unravel mysteries such as 'dark energy' by mapping the cosmic distribution of hydrogen, and tracking 'young' galaxies

Working with Dr Melanie Johnston-Hollitt, a radio astronomer from Victoria University in Wellington, IBM constructed the Information Intensive Framework (IIF) prototype to automate key elements of the work currently undertaken manually by scientists.

'The Information Intensive Framework prototype tested several new concepts and is IBM’s first attempt to tackle the data intensive challenge faced by astronomy,' said Dougal Watt, Chief Technology Officer, IBM New Zealand, and Chair of NZ SKA Industry Consortium'. 

'While developed with SKA in mind, the results are also applicable to other organisations faced with a ‘data deluge’. These range from individual manufacturing plants and telecommunications companies to whole transport networks and healthcare systems.'

Radio telescopes detect radio-frequency signals from space. They can reveal areas of space that may be obscured with cosmic dust. The SKA will the biggest and fastest ever built

Dr Johnston-Hollitt said: 'Undertaking research on exa-scale datasets will force radio astronomers into a new, as yet, unexplored habit of automated processing, imaging and analysis. 

'We will need new solutions to fully realize the vast scientific potential.' 

No comments: