BigData

Hive ANALYZE TABLE Command – Table Statistics

Hive uses cost based optimizer. Statistics serve as the input to the cost functions of the Hive optimizer so that it can compare different plans and choose best among them. Hive uses the statistics such as number of rows in tables or table partition to generate an optimal query plan. Other than optimizer, hive uses mentioned statistics in many other ways. In this post, we will check Apache Hive table statistics - Hive ANALYZE TABLE command and some examples. Uses of Hive Table or Partition Statistics There are many ways…

Comments Off on Hive ANALYZE TABLE Command – Table Statistics
January 11, 2019
General

Steps to Optimize SQL Query Performance – Best Practices

We pay lots of attention to improve the performance of the web application, but ignore back-end SQL performance tuning. Even experts like application architects and the developer does not have an idea on how databases process the SQL queries internally. This could be because of lack of SQL and database knowledge. In this post, we will check best practices to optimize SQL query performance. How to Select SQL Query for Optimization? Identifying the query to optimize is a crucial step. Even today's most advance SQL engines require optimization. Simple SQL query tweak may increase the…

Comments Off on Steps to Optimize SQL Query Performance – Best Practices
January 9, 2019
Netezza

Steps to Generate and Load TPC-DS Data into Netezza Server

The TPC-DS benchmark model decision support system of a retail product supplier. This includes various queries and data maintenance. Database schema, data population, queries, data maintenance model and implementation rules have been designed to be broadly representative of new or modern decision support systems. In this post, we will discuss steps to generate and load TPC-DS data into Netezza or puredata systems server. What is TPC-DS? The Transaction Processing Performance Council (TPC) is a benchmark model decision support system. Several major firms are member of TPC. You can get more…

Comments Off on Steps to Generate and Load TPC-DS Data into Netezza Server
December 28, 2018
Netezza

Different Methods to Display Netezza Table Statistics

IBM Netezza or Puredata Systems uses the cost based optimizer to determine best methods determine redistribution, scan, join, join orders. Optimizer uses statistics to generate optimal execution plan. You need collect statistics on tables or databases regularly. In my other post, Netezza Generate Statistics: A Guide and Best Practices, we have discussed some of best practices to generate statistics on Netezza tables. In this post, we will discuss different methods to display Netezza table statistics with some examples. What Are Table Statistics? Tables statistics are nothing but information about each column…

Comments Off on Different Methods to Display Netezza Table Statistics
December 20, 2018
Apache Spark

Spark SQL EXPLAIN Operator and Examples

Spark SQL uses Catalyst optimizer to create optimal execution plan. Execution plan will change based on scan, join operations, join order, type of joins, sub-queries and aggregate operations. In this article, we will check Spark SQL EXPLAIN Operator and some working examples. Spark SQL EXPLAIN Operator Spark SQL EXPLAIN operator provide detailed plan information about sql statement without actually running it. You can use the Spark SQL  EXPLAIN operator to display the actual execution plan that Spark execution engine will generates and uses while executing any query. You can use this execution plan…

Comments Off on Spark SQL EXPLAIN Operator and Examples
December 11, 2018
Apache Spark

Create Pyspark sparkContext within python Program

In my other article, we have seen how to connect to Spark using JDBC driver and Jaydebeapi module. Hadoop cluster like Cloudera Hadoop distribution (CDH) does not provide JDBC driver. You either have to create your own JDBC driver by using Spark thrift server or create Pyspark sparkContext within python Program to enter into Apache Spark world. Spark Context or Hive Contex SparkContext or HiveContex is entry gate to interact with Spark engine. When you execute any Spark application, driver program initiates context for you. for example, when you start…

Comments Off on Create Pyspark sparkContext within python Program
December 7, 2018
Netezza

How to Connect Netezza using JDBC Driver and working Examples

Netezza is one of the widely used MPP database. You connect to it by using various methods and programming languages. Netezza supports ODBC, OLEDB and JDBC drivers for connections. Connection to Netezza using JDBC driver is easy and one of the widely used method. In this article, we will check how to connect Netezza using JDBC driver and some working examples. Netezza JDBC Driver Netezza provides JDBC driver, you can use that driver from any programming language that supports JDBC connections such as Java, Python etc. You can download JDBC…

Comments Off on How to Connect Netezza using JDBC Driver and working Examples
December 2, 2018
Apache Spark

Execute Pyspark Script from Python and Examples

As Apache Spark is gaining popularity, most of organizations are trying to integrate existing big data ecosystem to Spark so that they can utilize speed and distributed computation power of Apache Spark. In my earlier post, I have discussed various Methods to Access Hive Tables from Apache Spark to access Spark from from Python. In this post we will discuss how to execute pyspark script from Python with an working examples. Python Pyspark Python is widely used programming language and easy to learn. Well, you can access Apache Spark within python…

Comments Off on Execute Pyspark Script from Python and Examples
November 26, 2018
BigData

Methods to Access Hive Tables from Python

Apache Hive is database framework on the top of Hadoop distributed file system (HDFS) to query structured and semi-structured data. Just like your regular RDBMS, you access hdfs files in the form of tables. You can create tables, views etc in Apache Hive. You can analyze structured data using HiveQL language which is similar to Structural Query Language (SQL). In this article, we will check different methods to access Hive tables from python program. Methods we are going to discuss here will help you to connect Hive tables and get…

Comments Off on Methods to Access Hive Tables from Python
November 16, 2018
Apache Spark

Methods to Access Hive Tables from Apache Spark

Now a days, with growing data size, Apache Spark is gaining importance. It is open-source general purpose and lightning fast distributed computing framework. Apache Spark is 100 times faster compared to Hadoop technologies. Considering its speed, you can use Apache Spark to access Hive metastore and process required data. In this post, we will check methods to access Hive tables from Apache Spark. Why Apache Spark? As mentioned earlier, Apache Spark is 100 times faster compared to Hadoop and more than 10 times faster than accessing data from disks. Spark…

Comments Off on Methods to Access Hive Tables from Apache Spark
November 16, 2018