Compatible versions to train and deploy XGBoost using PySpark — Following article gives a walkthrough of the steps to be taken to use XGBoost with PySpark on AWS EMR. XGBoost does not provide a PySpark API in Spark, it only provides Scala and other APIs. Hence we will be using a custom python wrapper for XGBoost from this PR. We…

Installation steps simplified — Below steps have been tried on WSL on a Windows 10 laptop, with two different Spark versions (2.4.5 and 3.1.2). I have used WSL but the steps will work on Ubuntu machine as well. Installing Prerequisites PySpark requires Java version 7 or later and Python version 2.6–3.7 for Spark 2.x.x and Python…

A non-exhaustive, but fairly large list of product recommendation strategies in Fashion Retail (ecommerce) Personalization Personalization consists of tailoring a service or a product to accommodate specific individuals, or groups/segments of individuals. Almost every organization today uses, or wants to use, personalization to improve customer satisfaction and sales. When it comes…

Part 2 of the Rolling Window approach series. Quick Recap When building a predictive model, often times the ask is to predict what will happen next, or what will happen in next X days or X weeks. …

How-to guide on setting up Airflow on Linux machine and creating a basic workflow using BashOperator, PythonOperator and MySqlOperator — Home No more command-line or XML black-magic! Use standard Python features to create your workflows, including date time…airflow.apache.org Airflow is a platform created by the community to programmatically author, schedule and monitor workflows. Airflow is a powerful tool when it comes to deploying and monitoring workflows. It uses standard Python features to create workflows, including date time formats for scheduling and loops to dynamically generate tasks. …

Following article walks through the flow of a clustering exercise using customer sales data. It covers following steps: Conversion of input sales data to a feature dataset that can be used for clustering Performing clustering exercise Profiling the clusters, and Setting up a regular scoring process to assign cluster labels…

Installation steps simplified (and automated to certain extent…) — Below steps have been tried on 2 different Windows 10 laptops, with two different Spark versions (2.x.x) and with Spark 3.1.2. Installing Prerequisites PySpark requires Java version 7 or later and Python version 2.6 or later. Java To check if Java is already available and find it’s version, open a Command Prompt…