OVERVIEW
Cheap and readily available Internet for all has made the amount of data generated by the world
rise exponentially. Scientists, data curators, researchers, etc. on intensive studies and work came
to a wonderful discovery- that these stupendous and copious amounts of data pack with it the
key to unlocking the future. In a world and market which is getting totally customer-oriented,
these petabytes and exabytes of random and heterogeneous data under all that mess somewhere
contain the answer of all sorts of questions- from the biggest of astrophysics-related stuff to
human behavior. However, the biggest milestone in digging those answers is to come up with a
methodology or framework that can dig through all those worthless pieces of junk in order to
strike upon pure gold.
rise exponentially. Scientists, data curators, researchers, etc. on intensive studies and work came
to a wonderful discovery- that these stupendous and copious amounts of data pack with it the
key to unlocking the future. In a world and market which is getting totally customer-oriented,
these petabytes and exabytes of random and heterogeneous data under all that mess somewhere
contain the answer of all sorts of questions- from the biggest of astrophysics-related stuff to
human behavior. However, the biggest milestone in digging those answers is to come up with a
methodology or framework that can dig through all those worthless pieces of junk in order to
strike upon pure gold.
THE SOLUTION
To answer this problem of dealing and handling these large units of data, the world saw the
inception of the field of big data. And this same field today is counted amongst one of the most
promising and emerging fields in the world of computation. Big data is certainly a great example
of the phrase, “Modern problems require modern solutions” (those with a strong meme game
would definitely feel that).
inception of the field of big data. And this same field today is counted amongst one of the most
promising and emerging fields in the world of computation. Big data is certainly a great example
of the phrase, “Modern problems require modern solutions” (those with a strong meme game
would definitely feel that).
While making use of the most basic concepts of Cloud Computing and Distributed Computing,
Big Data is more than capable of handling terabytes of data at a time while making use of the same
computational resources that we have had for a long time now. What makes it a perfect answer for
handling all sorts of data in the most effective and efficient manner is its ability of being highly
scalable. However, if we go to the very basics of the types of storage and their fetching speed, we
would come to a realization that manipulating, storing, and retrieving such large amounts of data
from even bigger data-centers and data-lakes using conventional methods will prove to be a bottleneck in terms of the time consumed. And so, with the current database and network
technologies at our disposal, what would be the most effective and detrimental of solutions
to this problem?
Big Data is more than capable of handling terabytes of data at a time while making use of the same
computational resources that we have had for a long time now. What makes it a perfect answer for
handling all sorts of data in the most effective and efficient manner is its ability of being highly
scalable. However, if we go to the very basics of the types of storage and their fetching speed, we
would come to a realization that manipulating, storing, and retrieving such large amounts of data
from even bigger data-centers and data-lakes using conventional methods will prove to be a bottleneck in terms of the time consumed. And so, with the current database and network
technologies at our disposal, what would be the most effective and detrimental of solutions
to this problem?
DATA CACHING
Again, the problem of dealing with the high fetching and retrieval times of data from data-centers,
we again go back to the very basics- to the basics of computer manufacturing. Just like we have 4
layers of storage in a computer system: the register memory, the cache, the RAM, and finally the
hard-disk, the introduction of a smaller and faster piece of memory equipment in between should
certainly aid us in our quest. And in fact, this is exactly the solution that proved itself to be the most
efficient of solutions. By introducing an intermediate buffer, the transfer and search of data would
first be done at this intermediate cached location in order to cut down the access time of the data
in every instance. Even for everything that it gave, this procedure of in-process caching also proves
to be cumbersome in a lot of instances- particularly when multiple instances are launched at the
same time or when the size of applications becomes extremely large.
we again go back to the very basics- to the basics of computer manufacturing. Just like we have 4
layers of storage in a computer system: the register memory, the cache, the RAM, and finally the
hard-disk, the introduction of a smaller and faster piece of memory equipment in between should
certainly aid us in our quest. And in fact, this is exactly the solution that proved itself to be the most
efficient of solutions. By introducing an intermediate buffer, the transfer and search of data would
first be done at this intermediate cached location in order to cut down the access time of the data
in every instance. Even for everything that it gave, this procedure of in-process caching also proves
to be cumbersome in a lot of instances- particularly when multiple instances are launched at the
same time or when the size of applications becomes extremely large.
The answer to the above bottleneck was later found in a concept known as distributed caching-
something beyond our scope of explanation at the moment. So you could research on it for a bit
yourselves.
something beyond our scope of explanation at the moment. So you could research on it for a bit
yourselves.
Resource Box
Despite being one of the best technology and frameworks of modern times, the world of big data
is still in dire need of major reforms to make things even more efficient. As such, receiving a data
science course from proper and experienced professionals is certainly the way to go for all the
technical people out there. Don’t think much and join a course today.
is still in dire need of major reforms to make things even more efficient. As such, receiving a data
science course from proper and experienced professionals is certainly the way to go for all the
technical people out there. Don’t think much and join a course today.
- Click here to know more about Data Analytics Course
Level 16, 1 Sentral,, Jalan Stesen Sentral 5,, KL Sentral,KL Sentral 50470 Kuala
Lumpur, Malaysia
Lumpur, Malaysia
phone no: 011-3799 1378
