“pip install pyspark”: Getting started with Spark in Python

I have worked with spark and spark cluster setup multiple times before. I started out with hadoop map-reduce in java and then I moved to a much efficient spark framework. Python was my default choice for coding, so pyspark is my saviour to build distributed code.

But a pain point in spark or hadoop mapreduce is setting up the pyspark environment. Having java, then installing hadoop framework and then setting up clusters …. blah blah blah…

I was looking for a simple one step setup process in which i can simply just do one click/command setup and just get started with coding in my local system. Once the code is ready I can simply run the job in a pre-setup cluster. (say over cloud)

So this article is to help you get started with pyspark in your local environment.

Assuming you have conda or python setup in local.

For the purpose of ease, we will be creating a virtual environment using conda. Simply follow the below commands in terminal:

conda create -n pyspark_local python=3.7

Click on [y] for setups.

conda activate pyspark_local

To ensure things are working fine, just check which python/pip the environment is taking.

which python
which pip
pip install pyspark

And voila! Its done!

Now that you have a pyspark setup. Let us write a basic spark code to check things.

We will we reading a file in pyspark now. So, create a sample.txt with some dummy text to check things are running fine.

Simply run the command to start spark shell: (you can do the same in python notebook as well)


Now let us run the below code.

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .master('local') \ 
    .appName('firstapp') \
    .config('spark.come.config.option','some-value') \

# make sure the path of the file is correct
text_f = spark.read.text('sample.txt')

Hope this will get you excited for running spark code in python!

For more: https://spark.apache.org/docs/latest/quick-start.html

Quiz: Do you know what is the role of SparkSession here? Comment below.


Topic: Matrix Rotation

In this section, I will be discussing a simple yet not so intuitive approach to solving matrix rotation problem. So the problem statement is to rotate a NxN matrix clock wise 90 degrees in an efficient way. I can think of it as Rubic’s Cube one side rotation.

Now what all changes when a matrix is rotated 90 degrees:

  • row 1 becomes last column,
  • row 2 becomes last second column,
  • row last becomes first column

Now I approach this problem in the following way…

  • Take one outer boundary at a time. Just the boundary.
  • Top right is at 0,0 index
  • Top left is at 0, len(A)-1 index
  • Bottom right is at len(A)-1, len(A)-1 index
  • Bottom left is at len(A)-1, 0 index

Now simply swap these four cells clock wise. top-left with top-right, top-right with bottom-right, bottom-right with bottom-left, and finally bottom left with top-left. Now just move one step ahead in clockwise direction.

When you are done with outer boundary movement, take a step into one layer inner circle, and perform above swaps the same way!

Illustration to show movement of swaps.

Below is the python code for same.

class Matrix_Rotation:
    # @param A : list of list of integers
    # @return the same list modified
    def rotate90Clockwise(self, A):
        for steps in range((len(A)/2)):
            for move_ahead in range(len(A)-1-steps*2):
                temp = A[len(A)-1-move_ahead-steps][steps]
                A[len(A)-1-move_ahead-steps][steps] = A[len(A)-1-steps][len(A)-1-move_ahead-steps]
                A[len(A)-1-steps][len(A)-1-move_ahead-steps] = A[steps+move_ahead][len(A)-steps-1]
                A[steps+move_ahead][len(A)-steps-1] = A[steps][steps+move_ahead]
                A[steps][steps+move_ahead] = temp
        return A

Can you find this code’s complexity? Comment below.

Installing Auto-Sklearn Properly using python 3.5+

auto-sklearn installation requires python 3.5 or higher. In addition, it also has dependencies on the packages mentioned here: https://raw.githubusercontent.com/automl/auto-sklearn/master/requirements.txt

Better approach is to have a python 3.5+ environment. And then using pip install auto-sklearn.

  • Check which version/path are you using – which pythonwhich pip
  • Install python 3.5 or higher, if you don’t have it already: steps to follow
  • Once you have the correct version of python installed, set up a virtual environment of the python3.5. Follow the code to setup a virtual environment:

python3 -m pip install --user virtualenv

source env/bin/activate

  • Finally call pip install auto-sklearn


  • In case you are using anaconda, following command will start your virtual env:conda update conda #Update your current version of condaconda create --name py35 python=3.5 #creat e a virtual env for python 3.5source activate py35 #activate the environment

Post your query here, again in case you are not sure of the steps.

Why it is important to be respectable to co-workers?

Coming from my personal experience, when I first started working, I was a low performer throughout. I used to push myself to go work every morning. My managers felt I was too naive. In order to get the right work and grow, I shifted teams. But the performance was always low.

I am so weird! I am not performing up to the mark! Everyone else is working in the same office. Why is it that my performance is low? It’s definitely because I am not professionally strong. I am a terrible employee!

I remained in my first job for a few years, accepting and blaming myself for my below-average performance. And finally quitting with the mindset that its me who needs to work harder. But I wanted to be someone on the top!

Moving forward with my ambitious mindset, I finished my graduate school and came to another workplace.

Now, the second workplace was my dream job. I was highly motivated since I had worked so hard to get the job. But then as I moved ahead with my work, my performance did not move up at all. A co-worker who was really aggressive with work had a role to play here. Many times he unknowingly, abused or simply rejected any proposals I had for our project at the office. In fact, the arguments and frictions were visible on the floor, in meeting rooms, in front of other co-workers. Simultaneously my personal life was also taking a major shift. It started to become distressed to be at work. And obviously, my performance was terrible.

It’s only when the co-worker left our workplace, I found my existence at work. Now when no-one was putting me down and no-one was telling me that “the problem is YOU” (me), I had more ideas to express. I was more vocal, more understanding on projects and ownership at work.

I watched Christine Porath’s  ted talk on incivility the other day. And now the dots connect. I believe that it is extremely important to respect people, in-general, and even more so at workplace. Watch the talk and let me know your thoughts on it.

Have you experienced issues at workplace?

Who do “you” want to be? It is a simple question. And whether you want to answer this or not, you are answering it everyday through your actions.

Athematik: My art page

Although i do not consider myself an artist, but art and color inspires me. I paint sometimes to refresh my mind. It is just so amazing to use colors! So last year around this time I started painting just as my weekend hobby. But then I started to like it. Here are some of them. Critics are most-welcome! Also, you can become a Patreon for me and support me.

Click to reach my Patreon page

Project Soli: Your hands are the only interface you’ll need


Graphics Source: Project Soli

While randomly scrolling through tech videos on YouTube, I came across the video of Project Soli at Google Advanced Technology & Projects Group (ATPG).

This technology reminded me of early design ideas of Pranav Mistry called the sixth sense technology.

But I am curious to know more about this. How does it work? Let’s explore…

What is Project soli?

Project Soli is capturing actions by hand into the virtual world. It is a new sensing technology that uses miniature radar to detect touchless gesture interactions. The premise of this technology is that the hand is an ultimate input device. It is very natural for us to use it. Capturing the capabilities of the human hand is what Project Soli envisions.

How does it function?

Soli is a purpose-built interaction sensor that uses radar for motion tracking of the human hand.

Imagine how hand gestures work while clicking a button or sliding a screen, or rolling a dial.

It is pretty fascinating to see how this works. It works by emitting electromagnetic waves in a broad beam.

Objects within the beam scatter this energy, reflecting some portion back towards the radar antenna. Properties of the reflected signal, such as energy, time delay, and frequency shift capture rich information about the object’s characteristics and dynamics, including size, shape, orientation, material, distance, and velocity.

Soli tracks and recognizes dynamic gestures expressed by fine motions of the fingers and hand. In order to accomplish this with a single chip sensor, we developed a novel radar sensing paradigm with tailored hardware, software, and algorithms. Unlike traditional radar sensors, Soli does not require large bandwidth and high spatial resolution; in fact, Soli’s spatial resolution is coarser than the scale of most fine finger gestures. Instead, our fundamental sensing principles rely on motion resolution by extracting subtle changes in the received signal over time. By processing these temporal signal variations, Soli can distinguish complex finger movements and deforming hand shapes within its field.

Source: Project Soli

I think it has incredible possibilities. Well for simple use-cases could be a virtual AC remote control, a presentation slide changer, music volume rise or low controller. More exciting could be a musical instrument, passcode detector, fancy door opener and closer. I always imitate door opening and closing in lifts! Just for fun. This makes me believe I can actually do it for all doors! Ha! I believe this is amazing.

When I saw it, I immediately felt like signing up for it’s updates or maybe build applications for it! However, due to some error on a web page, I couldn’t sign up.. 😦 Well I will try again. But in case you know more about it, please share it here. Else what are your thoughts on this? Isn’t this too exciting?