Posts categorized “python”
The Threat and Vulnerability Analysis team at Rackspace is charged with providing internal vulnerability scanning, penetration testing, and red/purple teaming capabilities to reduce cyber-based threats, risk, and exposure for the company. One of our tasks, as part of meeting certain compliance objectives, is to ensure systems are not exposed from various networking "perspectives" without going through a bastion first.
A few months back, I decided to find a way to lighten the load of ad-hoc vulnerability scanning requests by our system owners. Our most frequent requests used to go something like this: "Can you scan this"..."Ok I fixed it, scan again."
Given the prevalence of Slack and associated bots, I thought it would be a good idea to try to write my own for scanning. Enter the InsightVM Slack Bot!
At our annual rax.io internal technical conference in San Antonio this week, I had a blast hacking on a reporting tool for our new content engine behind developer.rackspace.com and support.rackspace.com.
I have spent the majority of my career as a Java developer. As a result, I learned to be more productive using an IDE instead of an editor like Vi. Even though Vi is still my editor of choice when I’m in a Linux shell, I don’t believe it’s practical when managing large Java projects.
This is the second in a series of posts written by the Repose Ninja on Duty. Special thanks to Jim Baker: author of the Fireside project, Jython core contributor, and integral braniac in the effort to support the WSGI specification through Servlet technologies.
If you have ever had the pleasure of evaluating Repose, you may have noticed that, while it provides an incredibly powerful foundation, it is missing that one all-important feature that you need. While the Repose team does its best to handle all common and reasonable use-cases, there are an infinite number of problems for which Repose is a solution. Therefore, it is impossible to predict and develop features to solve every problem. Luckily, Repose is built on a pluggable architecture that any developer can leverage to solve the problem of the day.
In this post, I will expand upon the previous post in this series by diving deeper into the Repose extensibility model and explaining how Repose plans to make that model more developer friendly in the future.
Wes McKinney started working on Pandas in 2008. Since then, Pandas has become one of the most popular and useful software components for the data scientist. For good reason; using Python, Pandas and iPython/Jupyter notebooks makes it simple and quick to perform analysis on various datasets.
In this post, we perform some basic analysis on the City of Baltimore employee salary data from data.gov, but this technique can be used on a wide variety of data sets very easily.
Pandas and Jupyter notebooks make this work quick. It may be surprising to see where the money goes!
As a PhD student at UC Berkeley, my duties involve some amount of teaching; so, this semester (Spring 2015), as well as last spring, I have been a teaching assistant for a class taught by my advisor, Tom Griffiths. The class, called Computational Models of Cognition (COGSCI 131), aims to introduce students to computational models of human behavior. The problem sets are a mixture of simple programming assignments—usually requiring students to implement pieces of different models—and written answers, in which students report and interpret the results of their code.
In the past, the problem sets were written in MATLAB. This year, however, we decided to make the switch to Python. In particular, we decided that the IPython/Jupyter notebook would be an ideal format for the assignments. The notebook is a cross-platform, browser-based application that seamlessly interleaves code, text, and images. With the notebook, it is possible for us to write instructions in the notebook, include a coding exercise after the instructions, and then ask for their interpretation of the results immediately after that. For an example of what the notebook looks like, you can check out try.jupyter.org for a demo.
Python Tennessee was a wonderfully put together conference with a great variety of speakers.
The IPython/Jupyter notebook is a wonderful environment for computations, prose, plots, and interactive widgets that you can share with collaborators. People use the notebook all over the place across many varied languages. It gets used by data scientists, researchers, analysts, developers, and people in between.
Keystone and many current OpenStack API components run in an Eventlet based http server. Eventlet is designed to perform well in networked environments and handles everything in a single thread.
The developers responsible for the Keystone project have recently recommended using Apache (with the mod_wsgi module) as a front-end rather than the traditional “Keystone” Eventlet-based process.
By using Apache as the front-end for Keystone, one gains better performance due to Apache’s ability to do multithreading. One can also take advantage of the variety of http server modules currently available for Apache. One popular module, Shibboleth, provides the ability to use one set of credentials to authenticate against multiple OpenStack clouds (more info here).
Here is a straight forward guide on how to setup Keystone to utilize Apache in your existing OpenStack deployment.
TLS and SSL are two critical technologies which underly much of the secure communications that occur on the internet. Over the past few years, spurred by increasingly effective attacks and a desire for new functionality, SSL and TLS have seen many new features, as well as practical improvements.
Python is currently in a transitional period between Python 2 and Python 3. For
the past few years, all new feature development has been happening on Python 3,
including new features in Python's
ssl module. This means that Python 3 users
have had acccess to these improvements to TLS, but Python 2 users (still the
majority of Python users) have been falling behind.
One of the most powerful features of Python is the REPL (Run, Evaluate, Print, and Loop) This allows developers to run their code and get quick feedback. Developers are able test out new ideas and try out different things without the cycle of modifiying, compiling, and running the source code.
Another feature of Python's REPL is the introspection capabilities. This allows developers to easily and dynamically explore libraries.
In the Java world there are a number of options to do this. This post explores some of those; and how to leverage groovy and jclouds to achieve the same speed of development.
On 22 January, Apache Libcloud project chair Tomaz Muraus announced the release of Libcloud 0.14, a Python package which abstracts away the many differences among cloud provider APIs, allowing developers to target one interface regardless of the vendor.
This is a guest post written by Michael DeHaan, CTO at AnsibleWorks. AnsibleWorks provides IT orchestration solutions that simplify the way IT manages systems, applications, and infrastructure.
A while back I wrote about Ansible as a way to simply automate IT infrastructure, and showed how to achieve some interesting zero-downtime rolling update capabilities.
If there is anything I love about the Python ecosystem, it's the scientific computing ecosystem. Standing on top of this stack for me is IPython, a robust tool for interactive computing. It has features like a simple navigable history, auto-completion, a brilliant web based notebook with inline plotting, an easy to use parallel computing framework, magic, and a well structured protocol that is being used to extend IPython for interactive computing with other languages including Julia. If you haven't heard of IPython before, I recommend you watch Fernando Perez's keynote talk on IPython from PyData Silicon Valley 2013.
It’s been a while since my last post on Project Meniscus, which is an open-source, Apache 2 Licensed, cloud-scale logging service that collects logging data from cloud servers and services, makes the data easily searchable through ElasticSearch, and dispatches it into numerous other data stores, including MongoDB and Hadoop. Today, I want to update everyone about the current status of the project and our future plans.
In an earlier post I reviewed the OpenStack miniconf that preceded the main PyCon, which was held in Hobart, Tasmania on July 6–7. I had meant to write this shortly after PyCon ended, but the whirlwind of travel back to the US and getting back into the daily grind pushed it off my plate.
The conference was recorded, and all the videos are available on the pyvideo.org website. I encourage you to watch as many of the sessions that interest you as you can – lots of good stuff in them!
The conference actually started for me earlier – the organizer, Chris Neugebauer, had asked for volunteers to help with the conference prep work: badges, swag, all that stuff. This was on the Wednesday before the conference, which happened to be the day I arrived, so it was as good an excuse as any to get out of my hotel and into Hobart. For those of you who have never gone to a PyCon, it is completely run by volunteers. No one gets paid; no one gets free admission; no one gets special perks. This was shocking to me when I moved from the Microsoft conference world a decade ago, where conferences were run as profit centers, and attendees paid for tickets that cost well over $1,000, but who could then relax and treat their time there as a vacation (which many did, at their employers’ expense). But PyCons are the exact opposite, and as a result everyone has a stake in the conference experience. I’ve found that volunteering not only makes you feel like you’re contributing, but it also means that you meet a lot of interesting people who might otherwise remain anonymous faces in the crowd.
Friday was the pre-conference day, with two miniconfs: one for Django, and the other for OpenStack. While I'd love to spend some time digging deeper into Django, I figured that given my background as an OpenStack developer, the OpenStack miniconf was for me.
A well-documented Python library is something to be proud of. Reading through the code, you've got docstrings explaining the various parameters and constraints around your functions, hopefully using standards like Sphinx or Google formatting. Having to open up the code to find this information is less than ideal, but it's easy to turn these docstrings into beautiful, hosted HTML, updated every time you push to GitHub.
We'll be taking advantage of plenty of different technologies to make this happen:
- ReadTheDocs manages the builds, versions, and hosts your docs.
- Sphinx with autodoc renders your docs.
- GitHub or BitBucket hook into ReadTheDocs to trigger builds whenever code is commited.
Before we get started, head over to ReadTheDocs and create an account, if you don't have one already. It's a free, open-source service. You can even spin up your own private instances, although their documentation for doing that isn't the best :)
Dream big: that is our vision on the Rackspace Project Meniscus team. In one of our dreams, we provide a top-tier Logging-as-a-Service (LAAS) solution for the cloud. In another, we are accepted as an incubator project within OpenStack. These are lofty dreams, but we are a focus-driven team, and our dreams are our goals.
Project Meniscus is a better focusing lens for system and application events. It is completely open source (Apache 2 License) and headed by John Hopper, the original author of both the Repose and Atom Hopper projects. Both of these projects have seen great success within Rackspace as well as adoption around the world by many companies. These projects continue to move forward and benefit from their initial startup as open- source projects; they also have been proven to scale to the demands of the Cloud.
I'm a pretty big fan of Python as a programming language. It allows me to program by discovery, that is poke and prod at things until the work. Not having to compile an entire program every time I change something is pretty fantastic, as is the ability to insert a debug statement and be able to break a program at that point, then run arbitrary python code within that context. Pretty indispensable to how I write software.
Another thing I like about Python, which some may not, is the ability to do magic things. Not quite so magic as xkcd would like us to believe, but fun stuff indeed.
Initially, when Hart migrated to Octopress from WordPress he was using Swiftly to push new posts to Cloud Files. Swiftly, a tool written by Cloud Files developer Gregory Holt, provides a Client class and a command line tool for common Swift functions. Since Rackspace Cloud Files is based on Swift, it made a great tool to sync our working Octopress directory to a CDN-enabled Cloud Files container.
The team over at Mailgun just posted a Python tutorial written by Mailgun customer Paul Finn about how to use Python and the Mailgun API to upload large images to Cloud Files. The use case for the post is uploading photos from your phone, but the same process could be used anytime to want to parse an email attachment and store it in the cloud. You can check out the full Mailgun and python tutorial at the Mailgun blog, but we have the highlights here. --Hart