Getting Python 3 up and running on Amazon’s cloud

You are moving up in the world and that means you need to move onto the cloud too! Exciting times; Amazon Web Service or AWS is a powerful place to be and used by pretty much everyone. Possibly the best reason to use AWS is that when it goes down, none of your customers or clients blame you because all of their stuff goes down too! My friend Chris put together a fantastic .ipynb that will get you connected to AWS and transferring files back and forth. They offer a free tier that includes 700+ hours a month of their basic EC2 instances (you can ignore the request increase part of Chris’s walkthrough unless you are going to be processing photos or something else super computationally heavy).

But what do you do once you get connected and have little minion computers running out there in the ether? You want to put them to work of course! Different instance setups come with preloaded software and a few with Python, but it can be tricky to get Python 3 working and the proper packages installed. And I know you want to be using Python 3, because you are not some sort of heathen who doesn’t put parentheses around their print statements! This method should allow you to do that on their Amazon Linux images. I used the Amazon Linux AMI 2017.09.1 (HVM), SSD Volume Type in particular as it comes with everything I needed.

Per Chris’s tutorial, you’ll create an instance on the AWS website. Then you’ll install the command line tool:

# Install the AWS command line tool like so:
pip3 install awscli — upgrade — user

And you’ll connect to it via ssh with your own .pem key file and the address to your particular instance.

ssh -i "my-key.pem" ec2-user@ec2-000-000-000-00.compute-1.amazonaws.com

You might have some troubleshooting with pem files and connecting; AWS has some good documentation there if you can’t follow along from Chris’s tutorial. After that, you should be connected to the command prompt of your little computer in the sky. Let’s install Python 3.

# You'll probably want to make sure your environment is updated
sudo yum update -y
# Then install python
sudo yum install python36
# Simple, yea?
# If you want to install a particular version, you can search for it
sudo yum list | grep python3
# This will return a list of available versions on AWS

Alright, that’s no big. But unfortunately you are going to have trouble pip installing all the packages you need and making sure they actually end up connected to Python 3. We can resolve that like so:

# Go get it
curl -O https://bootstrap.pypa.io/get-pip.py
# And set it up for python3
python3 get-pip.py —user

Two comands and you’ve got pip ready to go. Now I still had a lot of issues convincing pip to install what I wanted. You can use sudo some more to try to coerce it, but even with that I was pulling errors and unable to actually import most things. So here was the workaround Chris helped me through:

# Make python do your work for you!
vim build.py
# This opens up a new file in the lovely text editor vim; 
# alternatively you could have created this previously and uploaded
# In the editor type in this:
import pip
# And then a line like these for each package you need
pip.main(['install', 'pandas'])
pip.main(['install', 'tweepy’])
# Once you've finished listing packages, hit 'esc' and type in 
# ':wq' to write and quit out of vim
# Now you run your python file
sudo python3 build.py

Congrats, you’ve built up your python environment. That’s basically how you might create any virtual environment with a requirements document to install all your packages for you.

Before you go, there is one more trick I want to give you. The first time I went through this was to run a script I made to get a whole bunch of information from Twitter’s API. Because of the request rate limits, the script took forever and I kept hitting problems when leaving it running on my machine. When I ran it on my AWS instance, I thought I could just quit out of the instance and leave it running no problem. Unfortunately every time I disconnected, something would cause it to hang and I would log back on later to find it dead. That’s when I discovered the magic of screens.

Screens basically allow you to create what amounts to a new tab in your terminal that you can switch back and forth from and leave things running in the background. You can disconnect from AWS, reconnect, and then check in on whatever screens you left running to find them chugging away. It should already be installed, but if not, that’s simple enough:

sudo yum install screen

Easy. And here are the rest of the commands you are likely to need to get up and running:

# Create a new screen:
screen -S <screen_name>
# To switch back to your main terminal, hit
control + a + d
# To return to a screen:
screen -r <screen_name>
# And if you did something horrid and need to kill it:
screen -X -S <screen_name> quit

Hopefully that is enough to get you started utilizing AWS with Python 3. If you want to take a look at the script I wrote that uses Twitter’s API, you can check it out here. Now go build yourself a botnet bent on world domination!