General

Set and Use Environment Variable inside Python Script

It is somewhat difficult when it comes to setting and using bash environment variables in python script file. The same step is very easy and straight forward using shell script. In this post, we will check one of the method to set and use environment variable inside python scrip file. Note that, steps mentioned in this post helps only if you are setting and using that variable inside same process i.e. in same python script. There is no way you can modify bash script from python and use that variable…

Continue ReadingSet and Use Environment Variable inside Python Script
Comments Off on Set and Use Environment Variable inside Python Script
October 24, 2018
BigData

Steps to Connect HiveServer2 from Python using Hive JDBC Drivers

HiveServer2 has a JDBC driver and It supports both embedded and remote access to HiveServer2. Usually, remote HiveServer2 is recommended for production environment as it does not require direct metastore or HDFS access to be given to Hive users. In this article, we will check steps to Connect HiveServer2 from Python using Hive JDBC Drivers. Steps to Connect HiveServer2 from Python using Hive JDBC Drivers Hive JDBC driver is one of the widely used method to connect to HiveServer2. You can use the Hive JDBC with Python Jaydebeapi open source module.…

Continue ReadingSteps to Connect HiveServer2 from Python using Hive JDBC Drivers
Comments Off on Steps to Connect HiveServer2 from Python using Hive JDBC Drivers
September 21, 2018
BigData

Execute Hive Beeline JDBC String Command from Python

To perform any analysis, you need to have data in place. To collect data, you may have to connect your application to different data source. In this article, we will discuss on one of such approach to execute Hive Beeline JDBC string command from Python application. This is one of the simple and easy approach to connect to Kerberos HiveServer2 using Beeline shell. I was working on one of the machine learning project to predict query execution time on Hadoop Hive cluster. We were gathering various features from the HiveQL…

Continue ReadingExecute Hive Beeline JDBC String Command from Python
Comments Off on Execute Hive Beeline JDBC String Command from Python
September 4, 2018
Apache Spark

Apache Spark Architecture, Design and Overview

Apache Spark is a fast, open source and general-purpose cluster computing system with an in-memory data processing engine. Apache Spark is written in Scala and it provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. Apache Spark architecture is designed in such a way that you can use it for ETL (Spark SQL), analytics, machine learning (MLlib), graph processing or building streaming application (spark streaming). Spark is often called cluster computing engine or simply execution engine. Apache Spark is one of most…

Continue ReadingApache Spark Architecture, Design and Overview
Comments Off on Apache Spark Architecture, Design and Overview
July 5, 2018
Netezza

Netezza Stored Procedure ARRAY Variables and Examples

The ARRAY data type is a composite data value that consists of zero or more elements of a specified specific data type. Netezza nzplsql allows you to define ARRAY types along with other scalar variables. In this article, we will check Netezza stored procedure ARRAY variables, declaration and examples. Netezza Stored Procedure ARRAY Variables You can define and use VARRAY in Netezza stored procedures. You can insert values to array variables, increase array size in case if length is exceeded and remove elements from array. ARRAY variables are allowed anywhere…

Continue ReadingNetezza Stored Procedure ARRAY Variables and Examples
Comments Off on Netezza Stored Procedure ARRAY Variables and Examples
June 6, 2018
BigData

Apache Hive Grouping Function, Alternative and Examples

Most of the relational databases supports Grouping function to segregate super-aggregated rows.  Apache Hive Support for SQL grouping function was added in Hive 2.3.0. But who are using lower version of Hive will have difficult time in porting SQL queries that are written using grouping functions. In this article, we will check Apache Hive Grouping function alternative and examples. Grouping Function In general, the grouping function indicates whether an expression in a GROUP BY clause is aggregated or not for a given row. The value 0 represents a column that is part…

Continue ReadingApache Hive Grouping Function, Alternative and Examples
Comments Off on Apache Hive Grouping Function, Alternative and Examples
June 4, 2018
Netezza

Netezza Data Transformation Functions and Examples

You must secure the data in case if you are working on various sensitive and confidential data. Netezza does provides various transformation functions that you can use to change the data representation. In this article, we will check Netezza data transformation functions and some examples. Netezza Data Transformation Functions The data transformation functions in Netezza are used to change data into different representation for purpose of security (encryption and decryption), space saving (compression) and save time to transmit data across various medium. IBM used industry standard algorithms to build these…

Continue ReadingNetezza Data Transformation Functions and Examples
Comments Off on Netezza Data Transformation Functions and Examples
May 19, 2018
Netezza

Netezza Phonetic Matching Functions and Examples

Netezza nzsql language support phonetic matching functions, that is, you can encode names into phonetic representations. These functions are useful to match names that are based on their pronunciation and reduce misses that might result from spelling variations. In this article, we will check types of Netezza phonetic matching functions and usage with some examples. Netezza Phonetic Matching Functions There are two types of Phonetic matching functions supported in Netezza:  SoundEx NYSIIS algorithm  Double Metaphone (dbl_mp) algorithm There are three helper functions in Double Metaphone:  Primary Metaohine (pri_mp)  Secondary Metaphone…

Continue ReadingNetezza Phonetic Matching Functions and Examples
Comments Off on Netezza Phonetic Matching Functions and Examples
May 16, 2018
BigData

Step by Step Guide Connecting HiveServer2 using Python Pyhive

Data plays important role in every decision-making process. You may have to connect to various remote servers to get required data for your application. This article explains how to connect Hive running on remote host (HiveSever2) using commonly used Python package, Pyhive. In this article, we will check step by step guide Connecting HiveServer2 using Python Pyhive. There are lot of other Python packages available to connect to remote Hive, but Pyhive package is one of the easy and well-maintained and supported package. There is a option to connect to…

Continue ReadingStep by Step Guide Connecting HiveServer2 using Python Pyhive
2 Comments
May 4, 2018
Netezza

Netezza nz_migrate Script Usage and Examples

If you have upgraded you Netezza server to higher capacity box then want to migrate data from one netezza server to other, you can use the IBM provided Netezza nz_migrate utility. You can either mention the tables to be migrated or you can migrate all tables available in source database. Netezza nz_migrate script is one of the best Netezza utility that provides lot of option to use while migrating data from one server to other. nz_migrate optionally creates destination database if it is not already present.  All the tables mentioned in the…

Continue ReadingNetezza nz_migrate Script Usage and Examples
2 Comments
April 25, 2018