Apache Spark

Spark SQL INSERT INTO Table VALUES issue and Alternatives

Spark SQL is gaining popularity because of is fast distributed framework. The Spark SQL is fast enough compared to Apache Hive. You can create tables in the Spark warehouse as explained in the Spark SQL introduction or connect to Hive metastore and work on the Hive tables. Not all the Hive syntax are supported in Spark SQL, one such syntax is Spark SQL INSERT INTO Table VALUES which is not supported. You cannot use INSERT INTO table VALUES option in spark. We will discuss the alternate approach with some examples.…

Comments Off on Spark SQL INSERT INTO Table VALUES issue and Alternatives
July 7, 2019
Apache Spark

Python Pyspark Iterator-How to create and Use?

An iterator is an object in Python representing a stream of data. You can create an iterator object by applying the iter() built-in function to an iterable dataset. In python, you can create your own iterator from list, tuple. For example, the list is an iterator and you can run a for loop over a list. In this article, we will check Python Pyspark iterator, how to create and use it. Python Pyspark Iterator As you know, Spark is a fast distributed processing engine. It uses RDD to distribute the…

Comments Off on Python Pyspark Iterator-How to create and Use?
July 4, 2019
BigData

Hive UDF using Python-Use Python Script into Hive-Example

Hadoop provides an API so that you can write user-defined functions or UDFs using any of your favorite programming language. In this article, we will check how to how to create a custom function for Hive using Python? that is nothing but creating Hive UDF using Python. What is Hive? Hive is a data warehouse ecosystem built on top of Hadoop HDFS to perform batch and ad-hoc query execution on large datasets. Apache Hive can handle petabyte of data. The Hive is designed for OLAP. It is not suited for OLTP…

Comments Off on Hive UDF using Python-Use Python Script into Hive-Example
July 4, 2019
Apache Spark

Register Python Function into Pyspark – Example

Similar to UDFs in the hive, you can add custom UDFs in pyspark spark context. We have discussed "Register Hive UDF jar into pyspark" in my other post. We have discussed, how to add udf present in jar to spark executor later we register them to Spark SQL using create function command. In this article, we will check how to register Python function into Pyspark with an example. Register Python Function into Pyspark Python is one of the widely used programming languages. Most of the organizations using pyspark to perform…

Comments Off on Register Python Function into Pyspark – Example
July 1, 2019
Apache Spark

Register Hive UDF jar into pyspark – Steps and Examples

Apache Spark is one of the widely used processing engine because of its fast and in-memory computation. Most of the organizations use both Hive and Spark. Hive as a data source and Spark as a processing engine. You can use any of your favorite programming language to interact with Hadoop. You can write custom UDFs in Java, Python or Scala. To use those UDFs, you have to register into the Hive so that you can use them like normal built-in functions. In this article, we check check couple of methods…

Comments Off on Register Hive UDF jar into pyspark – Steps and Examples
July 1, 2019
Vertica

Vertica Derived Table and Examples

In a data warehouse environment, there are many places where you need to derive tables to meet certain requirements such as calculating columns, renaming table columns etc. You can use derived tables in place of temporary tables. In this article, we will check Vertica derived tables and how to use them in SQL queries. Vertica Derived Table A derived table in Vertica is basically a sub-query which is always in the FROM clause of a SQL query Statements. The reason it is called a derived table is because it essentially functions as a table as…

Comments Off on Vertica Derived Table and Examples
June 24, 2019
BigData

How to Update Impala Table? – Steps and Examples

Cloudera Impala and Apache Hive provide a better way to manage structured and semi-structured data on Hadoop ecosystem. Both frameworks make use of HDFS as a storage mechanism to store data. The HDFS architecture is not intended to update files, it is designed for batch processing. i.e. process huge amount of data. But most of the organizations are maintaining a data warehouse on traditional relation databases like Netezza, Teradata, Oracle, etc. When they migrate their data warehouse to Hadoop ecosystem, they might want to have a design similar to that…

Comments Off on How to Update Impala Table? – Steps and Examples
June 18, 2019
BigData

How to Execute HBase Commands from Shell Script? – Examples

Shell scripting is one of the widely used scripting language to automate day to day activities. Usually, Linux shells are interactive, they accept command as input from users and execute them. However, it will become repetitive as you have to type in all commands each time on terminal. Instead, you can bundle those commands in shell script. In this article, we will check how to execute HBase Commands from Shell Script with an example. Why Shell Script is Required? There are many reasons to use Linux shell scripting: It helps…

Comments Off on How to Execute HBase Commands from Shell Script? – Examples
June 6, 2019
BigData

Methods to Access Impala Tables from Python

Apache Impala is an open source massively parallel processing (MPP) SQL query engine for data stored on bigdata Hadoop. Cloudera Impala provides fast, interactive SQL queries directly on your Apache Hadoop data stored in HDFS or HBase. In addition to using the same unified storage platform, Impala also uses the same metadata, SQL syntax (Hive SQL), ODBC driver, and user interface as Apache Hive. In this article, we will check different methods to access Impala tables from python program or script. The methods we are going to discuss here will help you to connect Impala…

Comments Off on Methods to Access Impala Tables from Python
June 4, 2019
BigData

Impala Delete from Tables and Alternative Steps

Data warehouse stores the information in the form of tables. You may have to delete out-dated data and update the table’s values in order to keep data up-to-date. These performance critical operations are critical to keep the data warehouse on bigdata also when you migrate data from relational database systems.  In this article, we will check Impala delete from tables and alternative examples. Impala Delete from Table Command Cloudera Impala version 5.10 and above supports DELETE FROM table command on kudu storage. This command deletes an arbitrary number of rows…

Comments Off on Impala Delete from Tables and Alternative Steps
June 2, 2019