Personal letter from Solar VPS President & COO – Ross Brouse
Dear Solar VPS – Solar System Customers in New Jersey:
I want to take this opportunity to personally address the issues that many of you have been experiencing with our Solar System product. In order to do so, I want to explain to you how the current situation we are having came to be.
When we designed and architected the Solar System, we made the decision to build it on a shared storage platform. There are providers out there that use both local storage and shared network storage for their solutions. We chose to go the shared storage route as it provided us with the ability to easily migrate virtual machines between host nodes, perform maintenance without customer downtime, and to provide an overall level of portability that a local storage solution could never provide. We wanted to be sure we provided our customers with a truly enterprise class storage solution, which is why we selected NetApp as our storage layer vendor. Equally as important, we selected Juniper as our network layer vendor and Supermicro as our compute layer vendor. Our entire Solar System platform is built on a 10 Gigabit storage and network backplane in order to provide maximum disk performance. Unfortunately, while we chose what we felt were the absolute best vendors and spent well over a year designing and building the platform, there were some things we could never have anticipated.
Our NetApp SAN consists of two head nodes ( the servers responsible for all of the logic that takes place on the SAN ) and multiple JBOD chassis ( Just a Bunch of Disks ). Our entire Solar System NetApp deployment consists of multiple SAS-based JBODs with SSD caching. As our storage usage grew, we would continually add additional JBOD chassis to meet the demand of our ever growing Solar System. Unfortunately, what we did not anticipate was how the storage platform would react to a multi-tenant cloud deployment like the Solar System.
The reason that so many of you are experiencing outages and performance issues is due to the fact that the NetApp SAN in New Jersey has reached a point where its head nodes are unable to handle the overall load that is being caused by the sheer number of virtual machines on the system and the wide variety of applications that are being run within those virtual machines. We have spoken at length with NetApp and have explained to their engineers how we are using the system and what we are experiencing. They have thoroughly looked over our system and informed us that our Solar System application is “bringing our NetApp to its knees”.
I’m sure that many people would be inclined to make the comment, “add more hardware to it” or “move us to another system”. Let me assure you that we have some of the best and brightest engineers working for our company. If there was a quick fix we could deploy, we would certainly do it. We have no interest in causing pain or trouble for our customers. We truly understand that all of you make it possible for us to continue to run our business. That being said, we have been working very diligently to design a way to resolve these issues. Many of you may have received a message that it may take weeks to months to resolve the issues in question. I want you to know that this timeframe addresses what we feel is an overall and final fix to these issues. This does not mean that you do not have options.
Since we have reached a threshold with our NetApp in New Jersey, we have been forced to design a different and alternative solution for the storage component of our Solar System service. We are currently transitioning over to a local SSD-based storage system where each individual hardware node will have its own RAID10 array of multiple SSD’s. This is something we are working night and day on to deploy and I sincerely hope that we will have it operational very soon. I know that “soon” is a very vague, general and non-committal timeframe; however, please understand that giving you a specific timeframe will be a relief in the short-term but more than likely a disappointment in the long-term if we are unable to meet the provided timeframe. That being said, we will do our best to keep all of our affected customers updated on our progress.
Here are some options available to you:
- Work with our Support Department to move your virtual machine to our Miami, Dallas or Los Angeles locations — your IP Addresses will have to change
- Purchase a new virtual machine from one of our other locations
- Purchase a dedicated server from us — your IP Addresses will have to change
I know that these options are in no way optimal and may possibly frustrate many of you. Unfortunately, these are the best and only options at this point. Once we have our local SSD-based disk solution operating, we will begin the process of migrating existing customer Pods off of our NetApp storage system and onto the new storage system.
I understand that many of you would like more information about this and please know that we will work to be sure you receive the relevant information as it becomes available. Please understand that the exact specifics of the new solution and how we will perform the moves have not been 100% decided upon at this point as we must be sure that what we deploy will meet the demands of our customers so that we can avoid this situation from repeating itself.
I want to personally thank all of you for your patience and understanding as we work to resolve these issues in New Jersey. The performance and availability issues you are experiencing are unacceptable. We understand that and are doing our best to remedy the situation — all hands are on deck.
Solar VPS President & COO