{"id":303,"date":"2015-09-11T04:04:08","date_gmt":"2015-09-11T04:04:08","guid":{"rendered":"http:\/\/139.162.15.227\/?p=303"},"modified":"2020-03-02T13:07:42","modified_gmt":"2020-03-02T04:07:42","slug":"big-data-data-warehousing-smart-storage","status":"publish","type":"post","link":"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/","title":{"rendered":"Big Data, Data Warehousing &#038; Smart Storage"},"content":{"rendered":"<h3 class=\"underline\">Big Data Platform Considerations<\/h3>\n<p>This article highlights some considerations to be taken into account when designing a Big Data Platform in terms of Database selection and Hardware choice.<\/p>\n<h3 class=\"underline\">The Challenge<\/h3>\n<p>To create a scalable data architecture that meets the requirements to serve a particular data set within the required speed, size and budget constraints.<\/p>\n<h3 class=\"underline\">RDBMS vs NoSQL vs HADOOP<\/h3>\n<p>Hadoop and No SQL databases are great but in cases where the data structure is complex with a deep hierarchy of complex object relations there are some significant advantages of using a traditional RDBMS such as:<\/p>\n<ul>\n<li><strong>Simplicity:<\/strong><br \/>\nIt is very easy to navigate a hierarchal structure using basic SQL<\/li>\n<li><strong>Talent Pool:<\/strong><br \/>\nIts easier to hire a DBA experienced in SQL based DB&#8217;s than others<\/li>\n<li><strong>Ease of Use:<\/strong><br \/>\nData Scientists are able to run complex analyses with SQL which are often complicated to translate into code.<\/li>\n<\/ul>\n<p>For the purpose of this article we will focus on an RDBMS based solution.<\/p>\n<h3>Lack of Parallelisation<\/h3>\n<p>One of the limitations of traditional RDBMS&#8217; is the lack of true MPP (Massive Parallel Processing) capability for large queries.<br \/>\nTo address this a number of different approaches have been taken led by enterprise solutions:<\/p>\n<h5>Netezza<\/h5>\n<p>An older example of attempts at achieving fast databases through smart storage techniques on spiinning disks and parallelism done at a low level.<\/p>\n<p>Netezza stored the &#8220;hot data&#8221; on the outer tracks of the disk platter to leverage the faster access times and used the inner tracks for non hot-data and duplication.<\/p>\n<p>Parallelism was achieved through dividing the query execution in smaller worker tasks and combining the outputs into the final result.<\/p>\n<h5>Netezza<\/h5>\n<p>Netezza was acquired by IBM in 2010<\/p>\n<h5>GreenPlum<\/h5>\n<p>GreenPlum was acquired by EMC in 2010<\/p>\n<h5><strong>Teradata<\/strong><\/h5>\n<p>Teradata was acquired by EMC in 2010<\/p>\n<p><strong><\/strong><\/p>\n<p><strong><\/p>\n<h5>EMC<\/h5>\n<p><\/strong><strong><\/strong><br \/>\nDell acquired EMC in 2016<\/p>\n<h3>The Enterprise Market Solutions<\/h3>\n<p>Getting meaningful analytics from Data Warehouses requires a significant amount of processing &#8211; One can imagine a sort or aggregation on a billion row table of numeric values by date for example.<\/p>\n<p>Despite the Schema designer&#8217;s best efforts to keep the database as clean and as close to first normal form as possible &#8211; there is no getting around the fact that in most cases we will be waiting for that single CPU thread to execute the operation sequentially.<\/p>\n<p>Enabling parallel operations on a database is not something like an all or nothing &#8211; it can enabled on selected operations &#8211; where appropriate if you like.<\/p>\n<h5>Meaningful data insights on large datasets using a traditional DBMS!<\/h5>\n<ul>\n<li><i class=\"uk-icon-globe\"><\/i>Rapid Response<\/li>\n<li><i class=\"uk-icon-globe\"><\/i>Low Latency<\/li>\n<li><i class=\"uk-icon-globe\"><\/i>Scalable User<\/li>\n<\/ul>\n<p>Through levereging significantly improved web capabilities of HTML5, Avinton has delivered rich web based applications which allow users to visualise a wealth of information on their assets in the field such as:<\/p>\n<ul>\n<li>Configuration Parameters<\/li>\n<li>Connectivity to other nodes<\/li>\n<li>Performance Metrics<\/li>\n<li>Alarm Status<\/li>\n<\/ul>\n<h3>GIS Analytics<\/h3>\n<p>Analytics on the geographical data allows to aggregate statistics geographically by area, region, city or country as well as user defined areas. Our geo platform enables analyses against demographic datasets such as population density, land use type and such. This data can either be overlayed on the map or reported in table form.<\/p>\n<h3>Overlays<\/h3>\n<p>Heat maps and other calculated data sets such as analyses results or external layers can be overlayed onto the map. Through animations of the data over a time series one is able to visualise trends in the changes within the spatial data.<\/p>\n<h3>Map Data Support<\/h3>\n<p>Map data of any form can be imported into the system. Vector transformation to the base projection is trivial and we have in-house tools for raster reprojection if it so happens that there are more than one base projection in the geo data.<\/p>\n<h3>GIS Consulting<\/h3>\n<p>Avinton provides end to end GIS software development, delivery and support. We also provide an advisory service for those looking to get support with their application stack definition and which frameworks and application stack will best suit their needs.<\/p>\n<h3>Fleet Tracking &amp; Management<\/h3>\n<p>GPS data from personnel, vehicles or assets in transit allows near realtime tracking and visualisation. This data can be hooked into existing call centre or internal operations software to leverage the location awareness.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This article highlights some considerations to be taken into account when designing a Big Data Platform in terms of Database selection and Hardware choice.<\/p>\n","protected":false},"author":2,"featured_media":954,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8,42],"tags":[9],"class_list":["post-303","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tech-articles","category-42","tag-big-data"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.8 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Big Data, Data Warehousing &amp; Smart Storage - Avinton Japan<\/title>\n<meta name=\"description\" content=\"This article highlights some considerations to be taken into account when designing a Big Data Platform in terms of Database selection and Hardware choice.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Big Data, Data Warehousing &amp; Smart Storage - Avinton Japan\" \/>\n<meta property=\"og:description\" content=\"This article highlights some considerations to be taken into account when designing a Big Data Platform in terms of Database selection and Hardware choice.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/\" \/>\n<meta property=\"og:site_name\" content=\"Avinton Japan\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Avintons\/\" \/>\n<meta property=\"article:published_time\" content=\"2015-09-11T04:04:08+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-03-02T04:07:42+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/avinton.com\/wp-content\/uploads\/2015\/09\/WebHostingSolution.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"850\" \/>\n\t<meta property=\"og:image:height\" content=\"400\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"James Cauchi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@AvintonJapan\" \/>\n<meta name=\"twitter:site\" content=\"@AvintonJapan\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"James Cauchi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/\",\"url\":\"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/\",\"name\":\"Big Data, Data Warehousing & Smart Storage - Avinton Japan\",\"isPartOf\":{\"@id\":\"https:\/\/avinton.com\/en\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/avinton.com\/wp-content\/uploads\/2015\/09\/WebHostingSolution.jpg\",\"datePublished\":\"2015-09-11T04:04:08+00:00\",\"dateModified\":\"2020-03-02T04:07:42+00:00\",\"author\":{\"@id\":\"https:\/\/avinton.com\/en\/#\/schema\/person\/aa5bcc7a7c363ca85c0eeb6a7c2c594b\"},\"description\":\"This article highlights some considerations to be taken into account when designing a Big Data Platform in terms of Database selection and Hardware choice.\",\"breadcrumb\":{\"@id\":\"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/#primaryimage\",\"url\":\"https:\/\/avinton.com\/wp-content\/uploads\/2015\/09\/WebHostingSolution.jpg\",\"contentUrl\":\"https:\/\/avinton.com\/wp-content\/uploads\/2015\/09\/WebHostingSolution.jpg\",\"width\":850,\"height\":400},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/avinton.com\/en\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Big Data, Data Warehousing &#038; Smart Storage\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/avinton.com\/en\/#website\",\"url\":\"https:\/\/avinton.com\/en\/\",\"name\":\"Avinton Japan\",\"description\":\"Tailored Solutions and Consulting in AI and Big Data\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/avinton.com\/en\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/avinton.com\/en\/#\/schema\/person\/aa5bcc7a7c363ca85c0eeb6a7c2c594b\",\"name\":\"James Cauchi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/avinton.com\/en\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/24fff15ecfe40a23480c47de1acb5c69cc3aa019d6f6cd36353cee85ac20a9e7?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/24fff15ecfe40a23480c47de1acb5c69cc3aa019d6f6cd36353cee85ac20a9e7?s=96&d=mm&r=g\",\"caption\":\"James Cauchi\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Big Data, Data Warehousing & Smart Storage - Avinton Japan","description":"This article highlights some considerations to be taken into account when designing a Big Data Platform in terms of Database selection and Hardware choice.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/","og_locale":"en_US","og_type":"article","og_title":"Big Data, Data Warehousing & Smart Storage - Avinton Japan","og_description":"This article highlights some considerations to be taken into account when designing a Big Data Platform in terms of Database selection and Hardware choice.","og_url":"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/","og_site_name":"Avinton Japan","article_publisher":"https:\/\/www.facebook.com\/Avintons\/","article_published_time":"2015-09-11T04:04:08+00:00","article_modified_time":"2020-03-02T04:07:42+00:00","og_image":[{"width":850,"height":400,"url":"https:\/\/avinton.com\/wp-content\/uploads\/2015\/09\/WebHostingSolution.jpg","type":"image\/jpeg"}],"author":"James Cauchi","twitter_card":"summary_large_image","twitter_creator":"@AvintonJapan","twitter_site":"@AvintonJapan","twitter_misc":{"Written by":"James Cauchi","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/","url":"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/","name":"Big Data, Data Warehousing & Smart Storage - Avinton Japan","isPartOf":{"@id":"https:\/\/avinton.com\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/#primaryimage"},"image":{"@id":"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/#primaryimage"},"thumbnailUrl":"https:\/\/avinton.com\/wp-content\/uploads\/2015\/09\/WebHostingSolution.jpg","datePublished":"2015-09-11T04:04:08+00:00","dateModified":"2020-03-02T04:07:42+00:00","author":{"@id":"https:\/\/avinton.com\/en\/#\/schema\/person\/aa5bcc7a7c363ca85c0eeb6a7c2c594b"},"description":"This article highlights some considerations to be taken into account when designing a Big Data Platform in terms of Database selection and Hardware choice.","breadcrumb":{"@id":"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/#primaryimage","url":"https:\/\/avinton.com\/wp-content\/uploads\/2015\/09\/WebHostingSolution.jpg","contentUrl":"https:\/\/avinton.com\/wp-content\/uploads\/2015\/09\/WebHostingSolution.jpg","width":850,"height":400},{"@type":"BreadcrumbList","@id":"https:\/\/avinton.com\/en\/blog\/2015\/09\/big-data-data-warehousing-smart-storage\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/avinton.com\/en\/"},{"@type":"ListItem","position":2,"name":"Big Data, Data Warehousing &#038; Smart Storage"}]},{"@type":"WebSite","@id":"https:\/\/avinton.com\/en\/#website","url":"https:\/\/avinton.com\/en\/","name":"Avinton Japan","description":"Tailored Solutions and Consulting in AI and Big Data","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/avinton.com\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/avinton.com\/en\/#\/schema\/person\/aa5bcc7a7c363ca85c0eeb6a7c2c594b","name":"James Cauchi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/avinton.com\/en\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/24fff15ecfe40a23480c47de1acb5c69cc3aa019d6f6cd36353cee85ac20a9e7?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/24fff15ecfe40a23480c47de1acb5c69cc3aa019d6f6cd36353cee85ac20a9e7?s=96&d=mm&r=g","caption":"James Cauchi"}}]}},"_links":{"self":[{"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/posts\/303","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/comments?post=303"}],"version-history":[{"count":10,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/posts\/303\/revisions"}],"predecessor-version":[{"id":23337,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/posts\/303\/revisions\/23337"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/media\/954"}],"wp:attachment":[{"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/media?parent=303"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/categories?post=303"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/avinton.com\/en\/wp-json\/wp\/v2\/tags?post=303"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}