“pip install pyspark”: Getting started with Spark in Python

I have worked with spark and spark cluster setup multiple times before. I started out with hadoop map-reduce in java and then I moved to a much efficient spark framework. Python was my default choice for coding, so pyspark is my saviour to build distributed code.

But a pain point in spark or hadoop mapreduce is setting up the pyspark environment. Having java, then installing hadoop framework and then setting up clusters …. blah blah blah…

I was looking for a simple one step setup process in which i can simply just do one click/command setup and just get started with coding in my local system. Once the code is ready I can simply run the job in a pre-setup cluster. (say over cloud)

So this article is to help you get started with pyspark in your local environment.

Assuming you have conda or python setup in local.

For the purpose of ease, we will be creating a virtual environment using conda. Simply follow the below commands in terminal:

conda create -n pyspark_local python=3.7

Click on [y] for setups.

conda activate pyspark_local

To ensure things are working fine, just check which python/pip the environment is taking.

which python
which pip
pip install pyspark

And voila! Its done!

Now that you have a pyspark setup. Let us write a basic spark code to check things.

We will we reading a file in pyspark now. So, create a sample.txt with some dummy text to check things are running fine.

Simply run the command to start spark shell: (you can do the same in python notebook as well)


Now let us run the below code.

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .master('local') \ 
    .appName('firstapp') \
    .config('spark.come.config.option','some-value') \

# make sure the path of the file is correct
text_f = spark.read.text('sample.txt')

Hope this will get you excited for running spark code in python!

For more: https://spark.apache.org/docs/latest/quick-start.html

Quiz: Do you know what is the role of SparkSession here? Comment below.


Why it is important to be respectable to co-workers?

Coming from my personal experience, when I first started working, I was a low performer throughout. I used to push myself to go work every morning. My managers felt I was too naive. In order to get the right work and grow, I shifted teams. But the performance was always low.

I am so weird! I am not performing up to the mark! Everyone else is working in the same office. Why is it that my performance is low? It’s definitely because I am not professionally strong. I am a terrible employee!

I remained in my first job for a few years, accepting and blaming myself for my below-average performance. And finally quitting with the mindset that its me who needs to work harder. But I wanted to be someone on the top!

Moving forward with my ambitious mindset, I finished my graduate school and came to another workplace.

Now, the second workplace was my dream job. I was highly motivated since I had worked so hard to get the job. But then as I moved ahead with my work, my performance did not move up at all. A co-worker who was really aggressive with work had a role to play here. Many times he unknowingly, abused or simply rejected any proposals I had for our project at the office. In fact, the arguments and frictions were visible on the floor, in meeting rooms, in front of other co-workers. Simultaneously my personal life was also taking a major shift. It started to become distressed to be at work. And obviously, my performance was terrible.

It’s only when the co-worker left our workplace, I found my existence at work. Now when no-one was putting me down and no-one was telling me that “the problem is YOU” (me), I had more ideas to express. I was more vocal, more understanding on projects and ownership at work.

I watched Christine Porath’s  ted talk on incivility the other day. And now the dots connect. I believe that it is extremely important to respect people, in-general, and even more so at workplace. Watch the talk and let me know your thoughts on it.

Have you experienced issues at workplace?

Who do “you” want to be? It is a simple question. And whether you want to answer this or not, you are answering it everyday through your actions.

Project Soli: Your hands are the only interface you’ll need


Graphics Source: Project Soli

While randomly scrolling through tech videos on YouTube, I came across the video of Project Soli at Google Advanced Technology & Projects Group (ATPG).

This technology reminded me of early design ideas of Pranav Mistry called the sixth sense technology.

But I am curious to know more about this. How does it work? Let’s explore…

What is Project soli?

Project Soli is capturing actions by hand into the virtual world. It is a new sensing technology that uses miniature radar to detect touchless gesture interactions. The premise of this technology is that the hand is an ultimate input device. It is very natural for us to use it. Capturing the capabilities of the human hand is what Project Soli envisions.

How does it function?

Soli is a purpose-built interaction sensor that uses radar for motion tracking of the human hand.

Imagine how hand gestures work while clicking a button or sliding a screen, or rolling a dial.

It is pretty fascinating to see how this works. It works by emitting electromagnetic waves in a broad beam.

Objects within the beam scatter this energy, reflecting some portion back towards the radar antenna. Properties of the reflected signal, such as energy, time delay, and frequency shift capture rich information about the object’s characteristics and dynamics, including size, shape, orientation, material, distance, and velocity.

Soli tracks and recognizes dynamic gestures expressed by fine motions of the fingers and hand. In order to accomplish this with a single chip sensor, we developed a novel radar sensing paradigm with tailored hardware, software, and algorithms. Unlike traditional radar sensors, Soli does not require large bandwidth and high spatial resolution; in fact, Soli’s spatial resolution is coarser than the scale of most fine finger gestures. Instead, our fundamental sensing principles rely on motion resolution by extracting subtle changes in the received signal over time. By processing these temporal signal variations, Soli can distinguish complex finger movements and deforming hand shapes within its field.

Source: Project Soli

I think it has incredible possibilities. Well for simple use-cases could be a virtual AC remote control, a presentation slide changer, music volume rise or low controller. More exciting could be a musical instrument, passcode detector, fancy door opener and closer. I always imitate door opening and closing in lifts! Just for fun. This makes me believe I can actually do it for all doors! Ha! I believe this is amazing.

When I saw it, I immediately felt like signing up for it’s updates or maybe build applications for it! However, due to some error on a web page, I couldn’t sign up.. 😦 Well I will try again. But in case you know more about it, please share it here. Else what are your thoughts on this? Isn’t this too exciting?

Amazing Grace

I just came across this wonderful experience of watching Rear Admiral Grace M. Hopper  on David Letterman Show! And trust me, I am just talking to my self since then , Where on earth you were Ankita till now! Why you haven’t seen this video till now! I think everyone on this planet who believes in science must try and know more about Amazing Grace! Thank you Mam (Rear Admiral Grace M. Hopper) for making the first compiler! If you wouldn’t have done that, this post or my identity couldn’t have been possible!

To all readers:

  • Please read more about her: https://en.wikipedia.org/wiki/Grace_Hopper
  • Enjoy the below video

Semantic Web, and the Vision of World Wide Web

Hi Innovators!

The Evolution of internet, started with the PC Era, rising and constantly evolving towards Intelligent web can be observed these days in every part of the technology. We want machines to be more and more intelligent day by day. For us Siri is not enough! We want real life JARVIS!

In one of our college lectures, we came across this really interesting comparison of Semantic Web with Web 2.0. Have a look!

As a technology fanatic, “Tim Berners lee” is  definitely my idol! In the year 2001, he wrote an article in Scientific American on Semantic Web.  Its a must read! So have a look!


Imagine what we probably do via the Zomato’s Link(In Zomatos App) with Uber Taxi is what he(TIm Berners Lee) envisioned in the year 2001!

Well I am definitely enthralled by the power of WWW!  Tell us about your take on this.

Tick Tock on the “Quartz” Clock!

As a young naughty kid, I always used to get fascinated by all crazy engineered things and i used to stop only when i had opened them for not to be fixed back again. 😉 One time i broke our wall clock by opening it, only to be never functioning again. Another time it was TV’s remote control. Next was those hand held game with 9999 in 1 brick games console I had. And of-course I never admitted on doing those things ;). But recently my curiosity went towards the Clocks and time-keeping devices. I wonder what keeps these timer running. Does it have some chip with some program running? Then I found out the secret lies in its name! The QUARTZ Clock. BTW even I thought Quartz was a brand name. 😛 Well, this post is about the working of quartz clock. Lets unroll the secret scroll. As Wikipedia says, “A quartz clock is a clock that uses an electronic oscillator that is regulated by a quartz crystal to keep time.” Yes there is a quartz crystal resonator just like a fork that creates these oscillations. Now my curiosity was why Quartz? Why not some other material?

Chemically, quartz is a compound called silicon dioxide. Many materials can be formed into plates that will resonate. However, quartz is also a piezoelectric material: that is, when a quartz crystal is subject to mechanical stress, such as bending, it accumulates electrical charge across some planes. In a reverse effect, if charges are placed across the crystal plane, quartz crystals will bend. Since quartz can be directly driven (to flex) by an electric signal, no additional speaker or microphone is required to use it in a resonator. Source: Wikipedia.org

Lets have a deeper look at what we meant to say here. On unscrewing a wrist watch, It should look something like this: Armbanduhr_Rueckseite [Image Source: Wikipedia]   The cylindrical silver container(with some script K0S6M) that you see, lies the heart of this clock. Inside this lies a small quartz crystal which is responsible for the resonate motion. Its vibration is then responsible for rotation of the clock hands. However the actual cause of vibration of this quartz is due to the battery which produces electric current connected to it(Can you spot the circular battery with markings GP364). This happens by the phenomenon called reverse Piezoelectric effect. Basically due to the voltage of battery, the quartz is resonates. Well Quite the physics! However still more interesting is the engineering involved in creating the tiny quartz tuning forks. These crystals are made with such a perfect accuracy that they must have frequency 32,768 +/- 0.06 error. That’s like real perfection!

[Tuning fork crystal used in a modern quartz watch.[Source: Wikipedia]] Keep innovating!!!

Intelligent Systems and IBM Watson…

I have always been curious on Artificial Intelligence and its related topics. During my undergraduate studies, I was introduced to Alan and HAL of http://www.a-i.org. Oh they were my best buddies, I must say. You could talk to them for hours. AlanAlan, specially is my favorite. He says he is merely based on pattern matching, However, I find him very intelligent still. Recent chat bot to their addition is Jennifer. JenniferIts interface is quite like a regular chatting session.

Chat bots have always been my favorite. Recently TARS of the hellotars.com started a whatsapp chat service, where they have local experts serving all requests for you. I initially thought it was a AI service. However, they have a human interface at the top layer to answer the queries. Now this makes the whole idea not so fascinating 😦 However, they said they r working on building AI based agents soon.

During my final year of B.Tech. Project, I was working on a similar idea to build an ER diagram though Natural Language processing. Now this was my first attempt to understand how compilers and Intelligent systems work. Since then I had been learning and reading on Pattern recognition, Parsers, etc. Although, I am still a beginner in all these. I am quite optimistic that I will build an Real AI system soon. (Very soon.)

My recent discovery is the IBM Watson! Oh its a charm! It works though cognitive framework and learns by taking the decisions just like we humans learn: Observe, Interpret, Evaluate and Decide. The best part is it can understand any form of raw data. And that makes it even more powerful.

How does it learns? Watch the below video for the same.

Now I am still exploring more on Watson! I will update you on it soon… But you guys keep thinking on innovation!!