Developer Blog

This Week in Information Security (Week of June 22nd)

Welcome to another installment of This Week in Information Security! This week we have news of a critical vulnerability in millions of Samsung phones, a high-profile compromise of the password management tool LastPass, and a new method for stealing encryption keys via radios, without physically touching the machine in question. We also have tutorials about the basics of cryptography and using the Linux command line, and more!

As always, you can find me on Twitter @ccneill if you have any thoughts on this post.

Read More

Built an App on OpenStack at QCon NY 2015

Last week I went to QCon NY 2015 to be both a student and a teacher in their tutorial track. They follow the standard pattern of having 2 days of tutorials prior to the conference proper. To understand QCon a bit better, here's their mission statement.

"QCon empowers software development by facilitating the spread of knowledge and innovation in the developer community.

A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams."

Read More

Playing with CoreOS, Fleet and Docker

In this post, we will look at Docker, CoreOS & Fleet and demonstrate how one could use all of them in an application scenario.

Read More

This Week in Information Security (Week of June 15th)

Welcome back to This Week in Information Security! Sorry if you missed us last week, but posts should follow the schedule you're used to going forward. This week, we have news of two high-profile compromises, a few hair-raising hardware/firmware vulnerabilities, tools from DARPA for searching the "deep" and "dark" web, an opinion piece about vulnerability embargoes in open source software, and more. Finally, we wrap up the week with a fun, interactive article from Bloomberg about the meaning of code and the people and culture that produce it.

As always, you can find me on Twitter @ccneill if you have any thoughts on this post.

Read More

Goodbye XML!

As a developer working in the cloud, you're probably aware that OpenStack (and the cloud industry in general) have moved away from supporting XML APIs. In fact, OpenStack has just eliminated XML support for the Compute v2 API in its latest "Kilo" release. In our quest at Rackspace to be as consistent as possible with upstream OpenStack, we are planning to follow suit. The last day of XML support in the Rackspace public cloud will be July 20, 2015.

Read More

OpenStack Swift Use Cases in Rackspace Private Cloud

OpenStack Swift is the object storage project within OpenStack. From the OpenStack Swift documentation, "Swift is a highly available, distributed, eventually consistent object/blob store. Organizations can use Swift to store lots of data efficiently, safely, and cheaply."

Rackspace Private Cloud 10 introduced functionality into our Ansible tooling to deploy, manage, and scale Swift. No longer is Swift a second class citizen in our tooling.

Now that Swift can be easily deployed and used, what are some of the use cases for it in Rackspace Private Cloud?

Read More

This Week in Information Security (Week of June 1st)

This week we have some interesting research on the security of many Docker containers in the official Docker Hub repository, showing that a large number of images are vulnerable to severe exploits like ShellShock and Heartbleed. We also look at some research introducing a new attack vector utilizing novel steganography to deliver exploits to the browser using nothing but HTML5 and images. In other news, the NSA's phone metadata snooping program has folded (at least temporarily) following a failure of the Senate to extend the relevant provisions of the USA PATRIOT act that authorized the program.

As always, you can find me on Twitter @ccneill if you have any thoughts on this post!

Read More

Ruled the Stack and Lived to Tell About It

Last week I had the privilege to attend the OpenStack Super Bowl, aka the OpenStack Summit, in Vancouver. It was incredible just to be around so many other folks who also believe strongly in OpenStack.

So in between sessions, I stumbled across a friendly competition sponsored by Intel called, Rule the Stack. It was a competition to see who can build a fully functioning OpenStack cloud the fastest on (6) six physical servers. My coworker had mentioned it to me a week earlier, but, frankly, I forgot about it. I was focused on my two workshops and did not have extra time to plan. Anyone who knows me would know I love a challenge and never turn one down. Yes, of course you know I had to sign up to give it a go.

Before going much further, I wanted to fully disclose that I did not win the main prize in any way :D. I watched the SUSE guys do it in 6 minutes (which is a whole other discussion). Despite knowing I would not 'win' the competition, I went for it anyway. For me personally, it was not about winning but was more about solving this real life puzzle in a real-life repeatable way. The Intel guys appreciated my determined nature and awarded me as the 'Most Determined' participant.

When dealing with OpenStack, one of the challenges is designing an architecture that can scale horizontally and make decisions based on the commodity hardware presented to you. Holding true to the foundation OpenStack was originally built on, open cloud platform can run on any hardware (OEM or commodity or Open Compute). This competition pushes you to make all those decisions.

Again, this struck a cord in my heart because this is what I do for a living and because I believe the approach we take with RPC (Rackspace Private Cloud) makes solving those decisions very easy.

The quick breakdown of the competition is:

  • You are provided with (6) six physical nodes consisting of (3) three different configurations. Two node types had the same processor and memory but had a different number of drives. Then the third node type had a different processor, more memory and TPM module (more details can be found on the Intel site above).
  • Had to build using the Kilo release of OpenStack
  • The process for building out your configuration was open to you to decide. You could connect to the local network where the servers are connected via your own laptop or laptops provided.
  • There was opportunity for bonuses, shaving time from your final clock time, and penalties could be given for unconfigured nodes or nodes that were not optimized for use.

As soon as I saw the node configurations, I knew exactly how I wanted it to be setup. Keep in mind, I was not aiming for the fastest build but, rather the most complete flexible real life design. Despite HA not being a requirement (although I am attempting to have that rule changed for Tokyo (wink wink)), my reference architecture did include a dual server control plane. Also, I decided to include dual Cinder nodes and, of course, dual compute nodes. My complete reference architecture is outlined below.

The next step was to determine how to utilize the (4) four VLAN networks part of the provided specs. RPC asked for three individual network bridges and a management network. Each node had two NICs, and the first NIC was bound to VLAN 11. I sort of went back and forth with this decision for a while but finally settled in on one that worked.

My reference architecture

At this point, I am all ready to go, but there is still one last decision to make. How do I lay down the base OS on these servers? Again, not totally concerned with speed, I wanted to use a way that could be repeatable, flexible and cover the most ground possible without requiring post-install configurations. After a quick poll of my team, two contenders came to the forefront: Cobbler or MaaS. I'm not going to say which one turned out to be the most complete option and how I did it, as it could be my secret weapon for Tokyo. I will say is you would be shocked as to which one turned out to be the best option.

So everything is prepped and the clock starts. Let’s just say the first time around was not pretty at all. Did I give up then? Of course not! I just signed up to try again. Second attempt was a bit better, but I literally ran out of time before the next participants arrived (at that point I was still building at the 2 hour mark). Yes, the third attempt was running perfect, and, yet again, I was stopped because the previous participants had cut into my time slot a bit. My fourth, and final, attempt did the trick. This last attempt would have come in right under 1 hour and 30 minutes, but, unfortunately, I was literally being kicked out by security at the end of the session day on Thursday.

Pro tip: Sign up early and do not wait until the end of the Summit, as you will not be allowed a big enough time slot to finish.

All and all, it was a great experience and one that I plan to repeat in Tokyo. Special thanks go out to the Intel staff on hand in Vancouver - they were the best and very supportive/accommodating. Just a great set of guys! Congrats to the SUSE team who I have to assume were the winners in Vancouver. Good thing mostly about this get another chance to step up and show off your stuff in order to be crowned “Ruler of the Stack”.

Tsuki ni anata o sanshō shite kudasai! (Translation: "See you in November!” per Google :D )

Spoils of war

Read More

Using stackforge/os-ansible-deployment to review OpenStack Patches

As someone who works daily on libraries that power OpenStack, OpenStack itself, and deployment strategies for OpenStack, it's nice to be able to combine all of these roles when possible. As a core reviewer for the OpenStack Image Service (Glance), it's crucial that I not simply review the code in the a given change by eye but also test it to ensure:

  • It doesn't introduce new bugs
  • It fixes the bug, or provides the functionality, it claims to

Lately, I have been using the os-ansible-deployment (a.k.a., OS-A-D) to test changes that I review in OpenStack. This provides me with several benefits.

  1. Working on OS-A-D is my primary responsibility so the more I use it, the more familiar I become with it.

  2. It deploys all of the OpenStack services similar to how Rackspace Private Cloud deploys them but on a single server instead of on hundreds.

  3. It runs all of the services inside containers, so, if OpenStack is (at that point in time) not co-installable, it isn't a problem for my testing since I only care about testing how the patch works with the given service and other affected services, not that the dependencies of different services are incompatible.

  4. It's easy to use Ansible to continuously redeploy a service inside a container (assuming you're already familiar with Ansible). All of that said, it isn't a complete replacement for DevStack. DevStack, used by the Jenkins jobs in OpenStack to ensure a patch will pass the gate, is also necessary. If you're developing a patch for OpenStack, Ansible becomes a bit more cumbersome than DevStack to use, especially since you need to worry about how DevStack interacts with your patch.

That aside, let's look at an example of how you might review a change with OS-A-D with an example. I do all of my development on servers in Rackspace's public cloud. Let's start by creating a new server:

$ nova boot --key-name my-key \
    --flavor performance2-15 \
    --image 8226139f-3804-4ad6-a461-97ee034b2005 \
    --poll osad-glance_store-review

We'll need a few things before we can start the OS-A-D playbooks:

# apt-get update
# apt-get install -y fail2ban tmux vim git

tmux is optional, but I tend to use it heavily in my development environment. After everything finishes installing, I usually start up a tmux session and do the following:

# git clone
# cd /opt/os-ansible-deployment

In here, we have a directory with vars for playbooks and a sub-directory with vars that decide which versions to install of certain OpenStack services and dependencies OS-A-D will install. In the example, I'm going to walk through with you, we will be editing playbooks/vars/repo_packages/openstack_other.yml. We'll want to update it to read:

glancestore_git_install_branch: bug/1263067

In our example, we'll be testing out from glance_store. You'll notice we're pulling from my fork of glance_store. This is largely due to the fact that OS-A-D is meant for deployments, and it's unlikely anyone would deploy from Gerrit. So to test a change, I will happily push it to my fork, since I usually check it out locally to review it anyway. Once we've edited that, all we need to do is run ./scripts/ and wait for it to build an all-in-one version of OS-A-D. This will also run a subset of tempest's tests against the AIO as well as defcore's tests. These should provide a good indication that the AIO is functional.

If you want to check what version of glance_store we have installed, you can do:

# ansible glance_all -m shell -a "pip freeze | grep store"

You can also do

# pip install -d . glance_store

This downloads the wheel that was built from the local package index. Now that we have an functioning cloud, we can test the actual patch. But that's something I'll leave as an exercise for the reader.

When you want to test your patch at scale, the best way to do it is with os-ansible-deployment. You can easily scale your test environment from one machine to tens (or hundreds) of machines and ensure that everything works in a production environment. It manages your dependencies for you, builds them from source, and gives you a way to have a reproducible set of build artifacts by giving you a private package index that you can reuse and clone for future use.

Read More

This Week in Information Security (Week of May 25th)

This week we have information about the much-talked-about LogJam vulnerability in SSL/TLS, and several other issues that received less attention, but are scarier in my opinion (e.g. the buffer overflow in NetUSB and the DoS vulnerability in IPsec). We take a brief look at some recent vulnerabilities involving race conditions, affecting companies like Facebook, Dropbox, and Starbucks. We also look at some new research from Google about the effectiveness (or lack thereof) in using "security questions" like "what was your first pet's name?" or "what's your favorite kind of food?" (Spoiler: they're terrible in practice). Finally, for our random link of the week we have an opinion piece calling for the death of the PowerPoint, which sounds great, right?!

As always, you can find me on Twitter @ccneill if you have any thoughts on this post.

Read More
Racker Powered