Our Big Data Cluster will turn your Big Data into value. Big Data is information assets that can be utilized to derive Business Intelligence, which in turn is the enabling of enhanced decision making, insight discovery & process optimization on the business level. This involves the generation, collection, processing & analysis of your data.
There are many tools already in the market from vendors big and small, but most are targeted at specific processes only and therefore do not cover the entire workflow. Moreover, the choice of such tools can be very difficult, not to mention the high learning costs associated to utilize each of them.
Our Big Data Cluster is an industry-first, cloud-native, scalable solution to centralize the end-to-end workflow. Not only is it suitable for the development and deployment of AI systems, but it can also be used as a virtual private cloud platform.
Avinton has a standing history of success in the telecommunication industry, which with the rise in use of smartphones and connected mobile devices is generating massive amounts of data every day. Our solution has been tailored to fully cope with Big Data in all its forms.
Our solution features a fully-fetched AI platform with optional GPU support. Creating AI models is just the tip of the iceberg; deploying it in a scalable & reliable manner in user systems is usually the most cumbersome part. Using containers to manage the total workflow is our approach.
We make use of the latest yet proven open-source technologies. Whereas software solutions from vendors usually incur costly license fees, these are backed by developers from all over the world and are publicly available, helping to reduce long-term costs for platform owners.
Studies reveal that less than 30% of raw data owned by businesses is usable as-is. Our solution includes pipelines to collect data from various sources, and connectors to parse them into structured form. We also offer AI consulting services to validate the data that you already own.
Clusters comprise of multiple nodes & applications that must all be monitored around the clock. Our solution handles everything from the automatic collection of system metrics, to the generation of dashboards and handling of alerts.
The platform is fully orchestrated by Kubernetes, which makes it lightweight, scalable & self-healing. Our solution is cloud-native, but users will have the choice to deploy it on cloud, on premise or on virtual machines.
Protection of data from both unauthorized users and accidental loss is critical. Our solution comes with SSO (Single Sign On) capabilities to restrict access to both data and applications by users. In terms of data integrity, the distributed file system handles replication for fault tolerance.
Avinton’s solution is already deployed in full production at a global telecommunication company, and is acting as the core platform for its rapid transition to a data-driven enterprise.
The telecommunication industry of today is all about Big Data; mobile devices and network equipment are churning massive amounts of data every day. Corporate strategies must be implemented to turn this data into business value.
Our customer, seeing the opportunity to utilize their data assets, needed an end-to-end platform to collect, accumulate, process & analyze data, and fast.
Avinton, with its long-standing history of success handling data in the telecommunication industry, was awarded the full scope of work from design to deployment.
Within three months, the Big Data Cluster was deployed on-premise at the customer’s data center.
Currently, Avinton is responsible for the overall maintenance of the platform, as well as the development of custom web applications that run inside the cluster.
The platform is now collecting data from various sources, from internal servers to external cloud storage. The data is processed and used for the training of AI models, and visualization dashboards running as custom web applications.
In addition, Avinton has supported the customer’s transition to a data-driven enterprise through the delivery of training in data engineering & data analysis.
Following the success of the initial deployment, we have initiated the design of a second deployment in another region which will handle data at petabyte scales.
Find answers to some frequently asked questions about our Big Data Cluster.
As with any solution, there is Capital Expenditure (CAPEX) and Operational Expenditure (OPEX). Our solution utilizes the latest yet proven open-source technologies, which do not require hefty license fees to use like most vendor software.
At Avinton, we adhere to the open-source ethos that those who use such technology should give back to the open-source community in the form of enhancements, since it is through this positive cycle of give-and-take that technology can keep evolving.
This is a very common issue that most enterprises are confronted with when handling Big Data. Studies reveal that less than 30% of raw data owned by businesses is usable as-is. This is where data wrangling comes in, and is usually the most important and time-consuming part of AI-related projects.
Our Big Data Cluster can be packaged with customized data connectors, which can be used to parse your data into the desired format. For example, you may have log data in XML format which you would like in CSV format in order to train your model. With the schema and source data, we can create the connectors to suit this need.
Again, this is a very common issue when handling Big Data. The traditional approach would be the use of bash scripts, Perl scripts, SQL-stored procedures and the like to pull data from source. However, with the advent of Big Data and IoT, updating such scripts and procedures is simply not scalable. The value of data depreciates quickly, and so a real-time solution is needed.
With our solution, users can make use of an intuitive GUI which allows for data pipeline configuration. These pipelines can collect data from various sources, and even perform some data wrangling on-fly by making use of highly customized processors which handle inflows of data.
Big Data is all about the generation, collection, processing & analysis of data, and each process will be dependent on different applications. The task of managing these applications may indeed sound daunting.
However, within our solution each application is running as a container, and these containers are fully orchestrated by Kubernetes. Centralized configuration and self-heal are just some of the benefits of this approach.
Again, the fact that our solution comprises of applications running on containers fully orchestrated by Kubernetes ensures that the architecture is very scalable to handle such changes.
Should there be a need for more physical resources like storage or CPU, it is a simple matter of adding nodes to the existing cluster. The performance of applications will not be affected by the expansion.
The task of monitoring the health of not just the applications, but the actual nodes on which they are running is critical for the stability of the entire system.
Centralized monitoring is one of the defining features of our solution. Custom dashboards can be created to monitor standard health metrics (like CPU and memory usage) and logs, for both the applications and nodes. Alerting rules can also be implemented, to notify administrators of any health issues detected.
Yes, our solution is cloud-native. However, users will have the choice of deploying the platform on cloud, on premise, or on virtual machines.
Should users decide to go forward with the on premise approach, they will have the choice of selecting their preferred hardware vendor.
Our solution is fully compatible with AI projects from end-to-end. There will be no need to prepare separately third-party AutoML applications, which tend to be limited in terms of modelling capacity and expensive to use.
Moreover, the use of container-native workflows enables the deployment of AI models at scale, so that they can be used reliably by users accessing the system.
For users who have data at hand but are not confident as to whether it is sufficient to generate AI models, we can provide AI consulting as a separate service. This will usually involve a short PoC (Proof of Concept) using sample data sets provided
Protecting data from unauthorized users is critical. Our solution has SSO (Single Sign On) capabilities, which ensure that users can securely authenticate with multiple applications with minimal hassle; a single set of credentials can be used for all applications.
In terms of data integrity, which refers to the accuracy and consistency of data, the distributed file system on which the data is stored handles replication for fault tolerance. Files are stored as a sequence of blocks, which are replicated multiple times across the nodes inside the cluster.
Avinton has been handling various data-related projects in the telecommunication domain long before the term Big Data gained the degree of recognition that is seen today. Around half of our ongoing projects are AI / Big Data related.
All of our development is done in-house; we never use subcontractors. Therefore, we are confident that the solutions delivered to our customers are always of the highest quality and reliability.
We have a team of engineers rich in both age and ethnic diversity, with expertise in various domains of IT. This enables us to answer specific customer requests, such as those in the Case Study where we developed a custom GIS web application as part of the Big Data Cluster package.
Avinton is involved not only in the development of IT solutions, but also the provision of technical consulting to various clients, from small companies (50~100 employees) to multinational enterprises and local governments. The team is managed by our lead consultant with over 15 years of experience in more than 10 different countries. Some of our AI / Big Data related consulting services include:
– Custom Data Connector Development
– Custom Data Pipeline Development
– Data Visualization Application Development
– AI Modelling & Training
– PoC for Data Verification
– Technical Training (Avinton Academy)
We will be happy to provide you with more information.