This is a thought experiment, not an actual proposal, so hear me out. Let’s take one state (Alabama and Mississippi come to mind as front-runners), and make it white-only. There will be a national relocation service, and within a period of say, two years, all non-white, non-Christian (can’t have Muslims or Jews here!) people will have to relocate to anywhere else in the country they want. Similarly, all white supremacists, including KKK, Nazis, “alt-right”, etc., will have to relocate to this newly-whitened state. For convenience, let’s assume that Alabama is selected, and renamed Alabaster.
Once that is done, it will be illegal to espouse those views outside of the state of Alabaster, but it will be completely legal within. We can follow the Chinese example and create a Great Firewall around Alabaster, so that all social media posts made in this state cannot be seen outside, and posts from non-whites cannot be seen within. Inside Alabaster they will be free to fly Confederate and Nazi flags and erect all the statues to Confederate figures they want. Anyone caught promoting white supremecist messages in the rest of the country will be forced to relocate to Alabaster. And, of course, Fox News will only be able to be seen by residents of Alabaster (and it will be the only news station).
We could also make it more comfortable for these racists by eliminating some of the other things that bug them. Obamacare will not apply to residents of Alabaster, so they can enjoy being excluded for pre-existing conditions and the like. There will also be no federal taxes: income tax, social security, medicare – none. Of course, no federal funds will be given to Alabaster, so they’ll have to maintain their own roads, feed their own poor, care for their own elderly, all out of their own pockets. In other words, Tea Party heaven!
Having set the stage for this thought experiment, I’m wondering what the population of Alabaster would eventually be. In other words, how many people in the current USA would volunteer to move to such a white-only state? And for those who did, I wonder how happy they’d be, now that all of the things that they are currently so angry about have been removed from their lives. I would like to hear your thoughts on this.
Last week was the OpenStack Summit, which was held in Sydney, NSW, Australia. This was my first summit since the split with the PTG, and it felt very different than previous summits. In the past there was a split between the business community part of the summit and the Design Summit, which was where the dev teams met to plan the work for the upcoming cycle. With the shift to the PTG, there is no move developer-centric work at the summit, so I was free to attend sessions instead of being buried in the Nova room the whole time. That also meant that I was free to explore the hallway track more than in the past, and as a result I had many interesting conversations with fellow OpenStackers.
There was also only one keynote session on Monday morning. I found this a welcome change, because despite getting some really great information, there are the inevitable vendor keynotes that bore you to tears. Some vendors get it right: they showed the cool scientific research that their OpenStack cloud was enabling, and knowing that I’m helping to make that happen is always a positive feeling. But other vendors just drone about things like the number of cores they are running, and the tools that they use to get things running and keep them running. Now don’t get me wrong: that’s very useful information, but it’s not keynote material. I’d rather see it written up on their website as a reference document.
On Monday after the keynote we had a lively session for the API-SIG, with a lot of SDK developers participating. One issue was that of keeping up with API changes and deprecating older API versions. In many cases, though, the reason people use an SDK is to be insulated from that sort of minutiae; they just want it to work. Sometimes that comes at a price of not having access to the latest features offered by the API. This is where the SDK developer has to determine what would work best for their target users.
Another discussion was how to best use microversions within an SDK. The consensus was to pin each request to the particular microversion that provides the desired functionality, rather than make all requests at the same version. There was a suggestion to have aliases for the latest microversion for each release; e.g., “OpenStack-API-Version: compute pike” would return the latest behaviors that were available for the Nova Pike release. This idea was rejected, as it dilutes the meaning and utility of what a microversion is.
On the Tuesday I helped with the Nova onboarding session, along with Dan Smith and Melanie Witt. We covered things like the layout of code in the Nova repository, and also some of the “magic” that handles the RPC communication among services within Nova. While the people attending seemed to be interested in this, it was hard to gauge the effectiveness for them, as we got precious few questions, and those we did get really didn’t have much to do with what we covered.
That evening the folks from Aptira hired a fairly large party boat, and invited several people to attend. I was fortunate enough to be invited along with my wife, and we had a wonderful evening cruising around Sydney Harbour, with some delicious food and drink provided. I also got to meet and converse with several other IBMers.
There were other sessions I attended, but mostly out of curiosity about the subject. The only other session with anything worth reporting was with the Ironic team and their concerns about the change to scheduling by resource classes and traits. There was still a significant lack of understanding about how this will work for many in the room, which I interpret to mean that we who are creating the Placement service are not communicating this well enough. I was glad that I was able to clarify several things for those who had concerns, and I think that everyone had a better understanding of both how things are supposed to work, as well as what will be required to move their deployments forward.
One development I was especially interested in was the announcement of OpenLab, which will be especially useful for testing SDKs across multiple clouds. Many people attending the API-SIG session thought that they would want to take advantage of that for their SDK work.
My overall impression of the new Summit format is that, as a developer, it leaves a lot to be desired. Perhaps it was because the PTGs have become the place where all the real development planning happens, and so many of the people who I normally would have a chance to interact with simply didn’t come. The big benefit of in-person conferences is getting to know the new people who have joined the project, and re-establishing ties with those with whom you have worked for a while. If you are an OpenStack developer, the PTGs are essential; the Summits, no so much. It will be interesting to see how this new format evolves in the future.
If you’re interested in more in-depth coverage of what went on at the Summit, be sure to read the summary from Superuser.
The location was far away for me, but Sydney was wonderful! We took a few days afterwards to holiday down in Hobart, Tasmania, which made the long journey that much more worth the effort.
This seems so simple that it should be obvious, but it apparently it’s not, so I guess I have to spell it out. When waiting at a baggage claim carousel, you should stand back a few feet until you see your bag. Not only does it make it easier for everyone to get a view of the bags, but it leaves room for unloading the bags. Don’t crowd the carousel like these people:
One guy has the right idea, but he can’t see the bags that are coming because the others are blocking his view.
On a recent flight I saw my bag coming around , and had to squeeze my way through the people crowding the carousel. When it came, I grabbed it and lifted it off in order to put it on the ground. In the process it struck one of the people crowding the area, and he gave me a dirty look. I gave him one right back. It’s simply rude not to give a fellow traveler enough room to retrieve their luggage, and when you see someone grabbing their bag, it’s up to you to get out of their way.
Are you numb yet? To all of the outrageous, dishonest, and self-serving things that Donald Trump does? Or does your blood still boil every time his name is mentioned? I find myself in the latter camp, but I’m here to remind you of something very important: Trump Is Not The Enemy.
He is an enemy, of course, but by focusing on him, we miss the bigger picture. Thought experiment: imagine that tomorrow morning you wake up to see that Trump has tweeted his resignation. He’s gone. Not only that, but Bob Mueller also announces indictments of Trump Sr., Trump Jr., and Ivanka Trump. They’re all going to prison for a long, long time. Would that mean that things will be all right again?
Hardly. The Republican party controls all 3 branches of government, and they have been more than happy to ride along with the Trump populist train in order to achieve their goals. They have confirmed nominees to cabinet posts who are not only unqualified, but who have expressed views that are 180° opposed to the office they occupy. They have tried to take away health care to millions of people, and are currently planning on redistributing even more of the income of the masses to the extremely wealthy. They have wantonly ignored the truth, and instead parroted Fox News talking points. If Trump were to disappear, we’d have President Mike Pence, who would gladly continue, if not accelerate, the decline of our country.
It is not Trump, but the Republican party that is the true enemy. The only way we can progress as a country is to remove them from power.
How do we do that? My suggestion is to tie everything that Trump does to Republicans. If Trump hints about nuking North Korea or killing gay people, don’t place the blame solely on him. Place it on the enablers. Place it on the Republicans. Hold them accountable.
There are several Republicans in Congress who have expressed privately that they feel that Trump is unstable, but instead of acting on that, they just let it continue. Hold them accountable.
There is an overwhelming track record of violating the emoluments clause, and personally profiting from government use of his properties, but the Republicans turn a blind eye to that. Hold them accountable.
If in the weeks to come, evidence of Trump’s collusion with Russia surfaces, or any of a number of other potentially impeachable offenses are revealed, the power to impeach is 100% in the hands of the Republican majority. If they don’t act as swiftly and as thoroughly as they did investigating Hillary Clinton for Benghazi, they are excusing those acts, and thus complicit. Make that the headline, and hold them accountable.
We need to constantly tie our outrage to the Republicans, and not just to Trump, if we ever hope to move this country back to a positive direction. We need to stop focusing on winning the White House, and focus instead on winning state legislatures so that we can undo the gerrymandering that has allowed them to control the House with a minority of votes. We need to be sure to target every single Republican who is up for re-election in 2018, and tie them inextricably to Donald Trump. Trump is going down in flames, and we need to bring all of his enablers down with him.
Last week was the second-ever OpenStack Project Teams Gathering, or PTG. It’s still an awkward name for a very productive conference.
This time the PTG was held in Denver, Colorado, at a hotel several miles outside of downtown Denver.
It was clear that the organizers from the OpenStack Foundation took the comments from the attendees of the first PTG in Atlanta to heart, as it seemed that none of the annoyances from Atlanta were an issue: there was no loud air conditioning, and the rooms were much less echo-y. The food was also a lot better!
As in Atlanta, Monday and Tuesday were set aside for cross-project sessions, with team sessions on Wednesday–Friday. Most of the first two days was taken up by the API-SIG discussions. There was a lot to talk about, and we managed to cover most of it. One main focus was how to expand our outreach to various groups, now that we have transitioned from a Working Group (WG) to a Special Interest Group (SIG). That may sound like a simple name change, but it represents the shift in direction from being only API developer-focused to reaching out to SDK developers and users.
We discussed several issues that had been identified ahead of time. The first was the format for single resources. The format for multiple resources has not been contentious; it looks like:
In English, a list of the returned resources in a dictionary with the resource type/name as the key. But for a single resource, there are several possibilities:
# Singular resource
# One-element list
# Dictionary keyed by resource name, single value
# Dictionary keyed by resource name, list of one value
None of these stood out as a clear winner, as we could come up with pros and cons for each. When that happens, we make consistency with the rest of OpenStack a priority, so elmiko agreed to survey the code base to get some numbers. If there is a clear preference within OpenStack, we can make that the recommended form.
Next was a very quick discussion of the microversion-parse library, and whether we should recommend it as an “official” tool for projects to use (we did). This would mean that the API-SIG would be undertaking ownership of the library, but as it’s very simple, this was not felt to be a significant burden.
We moved on to the topic of API testing tools. This idea had come up in the past: create a tool that would check how well an API conformed to the guidelines. We agreed once again that that would be a huge effort with very little practical benefit, and that we would not entertain that idea again.
Next up were some people from the Ironic team who had questions about what we would recommend for an API call that was expected to take a long time to complete. Blocking while the call completes could take several minutes, so that was not a good option. The two main options were to use a GET with an “action” as the resource, or POST with the action in the body. Using GET for this doesn’t fit well with RESTful principles, so POST was really the only option, as it is semantically fluid. The response should be a 202 Accepted, and contain the URI that can be called with GET to determine the status of the request. The Ironic team agreed to write up a more detailed description of their use case, which the API-SIG could then use as the base for an example of a guided review discussion.
Another topic that got a lot of discussion was Capabilities. This term is used in many contexts, so we were sure to distinguish among them.
What is this cloud capable of doing?
What actions are possible for this particular resource?
What actions are possible for this particular authenticated user?
We focused on the first type of capability, as it is important for cloud interoperability. There are ways to determine these things, but they might require a dozen API calls to get the information needed. There already is a proposal for creating a static file for clouds, so perhaps this can be expanded to cover all the capabilities that may be of interest to consumers of multiple clouds. This sort of root document would be very static and thus highly cacheable.
For the latter two types of capabilities, it was felt that there was no alternative to making the calls as needed. For example, a user might be able to create an instance of a certain size one minute, but a little later they would not because they’ve exceeded their quota. So for user interfaces such as Horizon, where, say, a button in the UI might be disabled if the user cannot perform that action, there does not seem to be a good way to simplify things.
We spent a good deal of time with a few SDK authors about some of the issues they are having, and how the API-SIG can help. As someone who works on the API creation side of things but who has also created an SDK, these discussions were of particular interest. Since this topic is fairly recent, most of the time was spent getting a feel for the issues that may be of interest. There was some talk of creating SDK guidelines, similar to the API guidelines, but that doesn’t seem like the best way to go. APIs have to be consumed by all sorts of different applications, so consistency is important. SDKs, on the other hand, are consumed by developers for that particular language. The best advice is to make your SDK as idiomatic as possible for the language so that the developers using your SDK will find it as usable as the rest of the language.
After the sessions on Tuesday, there was a pleasant happy hour, with the refreshments sponsored by IBM. It gave everyone a chance to talk to each other, and I had several interesting conversations with people working on different parts of OpenStack.
Starting Wednesday I was in the Nova room for most of the time. The day started off with the Pike retrospective, where we ideally take a look at how things went during the last cycle, and identify the things that we could do better. This should then be used to help make the next cycle go more smoothly. The Nova team can certainly be pretty dysfunctional at times, and in past retrospectives people have tried to address that. But rather than help people understand the effects of their actions better, such comments were typically met by sheer defensiveness, and as a result none of the negative behaviors changed. So this time no one brought up the problems with personal interactions, and we settled on a vague “do shit earlier” motto. What this means is that some people felt that the spec process dragged on for much too long, and that we would be better off if we kept that short and started coding sooner. No process for cutting short the time spent on specs was discussed, though, so it isn’t clear how this will be carried out. The main advantage of coding sooner is that many of these changes will break existing behaviors, and it is better to find that out early in the cycle rather than just before freeze. The downside is that we may start down a particular path early, and due to shortening the spec process, not realize that it isn’t the right (or best) path until we have already written a bunch of code. This will most likely result in a sunk cost fallacy argument in favor of patching the code and taking on more technical debt. Let’s hope that I’m wrong about this.
We moved on to Cells V2. On of the top priorities is listing instances in a multi-cell deployment. One proposed solution was to have Searchlight monitor instance notifications from the cells, and aggregate that information so that the API layer could have access to all cell instance info. That approach was discarded in favor of doing cross-cell DB queries. Another priority was the addition of alternate build candidates being sent to the cell, so that after a request to build an instance is scheduled to a cell, the local cell conductor can retry a failed build without having to go back through the entire scheduling process. I’ve already got some code for doing this, and will be working on it in the coming weeks.
In the afternoon we discussed Placement. One of the problems we uncovered late in the Pike cycle was that the Placement model we created didn’t properly handle migrations, as migrations involve resources from two separate hosts being “in use” at the same time for a single instance. While we got some quick fixes in Pike, we want to implement a better solution early in Queens. The plan is to add a migration UUID, and make that the consumer of the resources on the target provider. This will greatly simplify the accounting necessary to handle resources during migrations.
We moved on to discuss the status of Traits. Traits are the qualitative part of resources, and we have continued to make progress in being able to select resource providers who have particular traits. There is also work being done to have the virt drivers report traits on things such as CPUs.
We moved on to the biggest subject in Placement: nested resource providers. Implementing this will enable us to model resources such as PCI devices that have a number of Physical Functions (PFs), each of which can supply a number of Virtual Functions (VFs). That much is easy enough to understand, but when you start linking particular VCPUs to particular NUMA nodes, it gets messy very quickly. So while we outlined several of these complex relationships during the session, we all agreed that completing all that was not realistic for Queens. We do want to keep those complex cases in mind, though, so that anything we do in Queens won’t have to be un-done in Rocky.
We briefly touched on the question of when we would separate Placement out into its own service. This has been the plan from the beginning, and once again we decided to punt this to a future cycle. That’s too bad, as keeping it as part of Nova is beginning to blur the boundaries of things a bit. But it’s not super-critical, so…
We then moved on to discuss Ironic, and the discussion centered mainly on the changes in how an Ironic node is represented in Placement. To recap, we used to use a hack that pretended that an Ironic node, which must be consumed as a single unit, was a type of VM, so that the existing paradigm of selection based on CPU/RAM/disk would work. So in Ocata we started allowing operators to configure a node’s resource_class attribute; all nodes having the same physical hardware would be the same class, and there would always be an inventory of 1 for each node. Flavors were modified in Pike to accept an Ironic custom resource class or the old VM-ish method of selection, but in Queens, Ironic nodes will only be selected based on this class. This has been a request from operators of large Ironic deployments for some time, and we’re close to realizing this goal. But, of course, not everyone is happy about this. There are some operators who want to be able to select nodes based on “fuzzy” criteria, like they were able to in the “old days”. Their use cases were put forth, but they weren’t considered compelling enough. You can’t just consume 2 GPUs on a 4-GPU node: you must consume them all. There may be ways to accomplish what these operators want using traits, but in order to determine that, they will have to detail their use cases much more completely.
Thursday began with a Nova-Cinder discussion, which I confess I did not pay a lot of attention to, except for the parts about evolving and maintaining the API between the two. The afternoon was focused on Nova-Neutron, with a lot of discussion about improving the interaction between the two services during instance migration. There was some discussion about bandwidth-based scheduling, but as this depends on Placement getting nested resource providers done, it was agreed that we would hold off on that for now.
We wrapped up Thursday with another deep-dive into Placement; this time focusing on Generic Device Management, which has as its goal to be able to model all devices, not just PCI devices, as being attached to instances. This would involve the virt driver being able to report all such devices to the placement service in such as way as to correctly model any sort of nested relationships, and determine the inventory for each such item. Things began to get pretty specific, from the “I need a GPU” to “I need a particular GPU on a particular host”, which, in my opinion, is a cloud anti-pattern. One thing that stuck out for me was the request to be able to ask for multiple things of the same class, but each having a different trait. While this is certainly possible, it wasn’t one of the use cases considered when creating the queries that make placement work, and will require some more thought. There was much more discussed, and I think I wasn’t the only one whose brain was hurting afterwards. If you’re interested, you can read the notes from the session.
Friday was reserved for all the things that didn’t fit into one of the big topics covered on Wednesday or Thursday. You can see the variety of things covered on this etherpad, starting around line 189. We actually managed to get through the majority of those, as most people were able to stay for the last day of PTG. I’m not going to summarize them here, as that would make this post interminably long, but it was satisfying to accomplish as much as we did.
After the conference, my wife joined me, and we spent the weekend out in the nearby Rockies. We visited Rocky Mountain National Park, and to describe the views as breathtaking would be an understatement.
I would certainly say that the week was a success! It took me a few days upon returning to decompress after a week of intense meetings, but I think we laid the groundwork for a productive Queens release!