State Panel Discussion: Cloud Computing Webinar...Webinar Series Cloud Computing State Panel...

Webinar Series

Cloud Computing State Panel Discussion

September 5, 2018

Presenters: Linda Jewell, Assistant Director, Information Technology, Arizona Department Child Safety

Jay Cline, Infrastructure Architect, Arizona Department Child Safety

Vallimanalan Thirugnanam, Director of Applications, Maryland Department of Human Services

Joe Vastola, CIO, Florida Department of Children and Families

Joyce Rose, Host, ICF

Nick Mozer, Junior Analyst, ICF

Operator: Welcome and thank you for standing by. Participants are in a listen-only mode until the question and answer session of today’s conference. At that time, you may press star then 1 on your touch-tone phone to ask a question. Today’s conference is being recorded if you have any objections you may disconnect now at this time. And now I would like to turn the conference over to your host Ms. Joyce Rose. Thank you, ma’am you may begin.

Joyce Rose: Thank you and welcome to the Child Welfare Information Technology Systems Managers and Staff Webinar Series, brought to you on behalf of the Health and Human Services Administration for Children and Families, Children’s Bureau. Today’s roundtable features a distinguished panel from three states, who will discuss their experiences and challenges relating to cloud computing. I am Joyce Rose, your host and facilitator for today’s discussion. Next slide, please.

I want to thank our panelists for their time and effort in preparing for this roundtable. And I would like to ask them to introduce themselves, Linda.

Linda Jewell: Um, sure. Hello everyone, my name is Linda Jewell. I am the Assistant Director, Chief Information Officer for the Arizona Department of Child Safety.

Joyce Rose: Jay.

Jay Cline: Hi, my name is Jay Cline and I am the Infrastructure Architect at the Arizona Department of Child Safety.

Joyce Rose: Valli.

Vallimanalan

Thirugnanam:

Joyce Rose:

Hello everyone, my name is Vallimanalan Thirugnanam. I go shortly by Valli. I am the Director of Applications, Maryland Department of Human Services. And last but not least, Joe.

Joe Vastola: Good afternoon everyone, I am Joe Vastola. I am the Chief Information Officer for the Florida Department of Children and Families.

Joyce Rose: Thank you, very distinguished panel. So, next slide please.

Attendees are encouraged to participate in the roundtable with discussion, questions, or comments. All of the participant lines are muted now but we will open them for the Q & A session at the end of the panel discussion. Please be aware that you can submit questions at any time, using the GoToWebinar chat feature and those will be queued up and addressed during the Q & A session. Once today’s roundtable has ended, you may submit additional questions to the email address listed or to your federal analyst. Next slide, please.

So, what is our format for today’s roundtable? I will pose a series of questions to prompt our state panelists in a discussion focusing on these topics, which I hope will give rise to an informational, as well as a lively discussion. We are going to set some context. In other words, we are going to take a look at each state’s cloud infrastructure schematic, we are going to discuss and understand the cost of cloud computing, governance structure contract and payment structures, staffing issues, managing vendor changes and upgrades, understanding the build and release cycles and other complexities of cloud computing. And then of course, the time-honored lessons learned. Next slide, please.

So, to establish some context, our state panelists will explain their programmatic and technical cloud-computing environment. Starting with Linda and Jay from Arizona.

Jay Cline: Hi, this is Jay. Basically, what we are looking at here is the main components that we use of a cloud-oriented nature. As a department and an agency, we were in a pretty good spot when we started setting this up as we were separating from another state agency at the time, so we got a very unique ability to set up these services basically from the ground up. We did not have to do a lot of transferring from an on-prem style to cloud style we got to set up almost everything ground up/brand new. Most of our current resources sit in the Azure Government tenant; a lot of VMs, platform services, software system services, most of those things sit up there. We currently heavily use Office 365 for things like email, internal communications on Skype. We are currently utilizing OneDrive for a lot of home directory information. We are leveraging VMWare’s Horizon

Air for a virtual desktop infrastructure, which allows non-DCS entities to connect to a DCS environment, so we do not have to worry about them connecting through VPN on their own equipment. It allows them to connect through our equipment from a hosted infrastructure. We have another vendor that hosts one of our core business applications, they are also hosted in the Azure Government tenant, as well as setting up OnBase enterprise content management systems also on a highland hosted enterprise environment. For everything that we sit on-prem currently the only aspect we have are our core infrastructure, so some Cisco devices and roughly about four physical servers. We are currently sitting at about two racks of equipment inside a hosted data center here in Arizona. Everything else that we have of the roughly 90+ servers is all in a virtual hosted cloud style environment. Next slide, please.

This kind of outlays two of our main business applications that we are using. You can see that there are two groups of people currently listed in red: DCS Providers and DCS Mobile Application users. This is mostly a reference of our current environment versus the one that we are in the process of building and transitioning to. DCS providers are entities and agencies that work with DCS on some of our information such as foster care providers and licensing agencies. They connect to our environment to connect to our core business application which is currently running in another state agency that we connect to which we are currently in the process of migrating off. They connect through those VDI hosted desktop infrastructure that is current sitting inside VMWare. That VDI infrastructure then talks to the identity management and all of our infrastructure services sitting inside the Azure Government cloud, which has an express route link connected back to our physical infrastructure link where the mainframe environment where all the back-end data actually lives. The more current information that we have for our actual users is a mobile application that our field workers can use for tablet-oriented information that allows them to link into a mainframe back end. It provides a much easier use for their information access while they are out in the field without using either their heavier laptops or, heaven forbid, paper and pencil. They actually access that mobile application through another hosted instance that a vendor provides. That Azure Government instance has an express route back to our physical center, which then talks to our identity management and our Azure Government cloud, which then talks back to our physical mainframe. Now both of these systems work. The problem with this is and what we are trying to transition off is a physical mainframe environment. We are currently looking at transitioning to the right side of this slide where all providers, all DCS employees, and even Arizona citizens can access all the data from our core business applications through a common front end. We are going to be

going to more of a hosted CRM. We are planning on using Dynamics and Office 365 to provide these portals and access, which will link very similarly as the previous ones, but provide, like I said, a very common infrastructure and allow us to store all the information necessary for this in Dynamics itself in our Azure Government environment. Which hopefully in the near future will remove our necessity of hosting that mainframe physical infrastructure in the on-prem data structure, which we currently have now. Linda, do you want to add anything?

Linda Jewell: No. So, that was a lot of information for everyone to consume. As you can see, we are 100% cloud except for our core networks and some of our legacy systems. This is a three-year strategy in the making and Jay made it sound very easy. There were some things that we had to put in place, some good contracting to put in place, some good SLAs and vendor management, understanding all of the dependencies, especially when to run the network connections, the dedicated connections. Our state was not really set up for us to be able to leverage any kind of express route or any kind of technology to cloud environments – dedicated connections. We really had to forge that path ourselves. Anybody looking at really building cloud utilization do not underestimate some of the work that is required in the vendor management and contract management in order to make that happen. Your state might not be set up to have all of those agreements in place especially on the network side.

Joyce Rose: Fantastic, thank you very much Linda. We are going to explore some of those concepts that you just talked about later on in our entire panel discussion session. Let us move on to the Florida cloud schematic and Joe Vastola please.

Joe Vastola: Thanks Joyce, this is Joe. Florida is similar but different to what we just heard; back in December 2017, we completed the transition of our Child Welfare system off of the mainframe platform and then transitioned the production, test development, and training environments of our system into the Amazon public cloud. Today we are running all infrastructure-as-a-service components for our child welfare system in the cloud and included in this transition, probably like most states who receive services from a data center. Our state data center, here in Florida, doesn't just provide us with infrastructure as a service, so they are the folks responding to issues, hardware and other work keeping things patched. So, part of this transition took us out of that service subset and included with the move of the system to the cloud. We needed to create a structure and a set of contracts for all managed services in the system. So today, we are sitting with our Child Welfare System running completely in the Amazon Cloud and we have been able to leverage the cloud environment for a couple things. One was an extension of our analytics environment. So,

included with our state child welfare system is a data warehouse that we have used over time for all the reporting, both ad hoc and operational and any analytics we have done.

There has been a goal here at the department in Florida to integrate our client data across all our systems. We have been able to leverage the cloud by creating an environment for big data. So, we have a big data virtual environment that links all of our child welfare clients to all of the client data that is contained in our other key legacy systems, including Public Assistance, and our Substance Abuse and Mental Health systems, which still reside in our state data center. All of that data is hosted in the virtual environment, now that we have got in the cloud, and we use it for integration and analytics and we are starting to build a data science environment set of tools so that we can begin leveraging some of the abilities of predictive analytics and machine learning and then to extend our master client index to integrate and match all these clients across systems into a master data management environment. We have a master data management tool we have installed and hosted that we have linked to all this data. We are in the process of establishing governance and the necessary controls and processes that we need to properly support that.

And then, in addition to that, we have also in the cloud established an environment for security incident and event management. We have got a SIM tool that we are running in the Amazon cloud that is hooked to our public assistance environment and we are extending that over time. It is basically monitoring all logs across the thousand servers that make up that system and the mainframe, which it runs on. We have a set of contracted resources to monitor and manage that and report back to our security team at the department. Our goal is to work towards pulling all of our legacy data, all of our client data getting them all matched and hosted available for analytics and integration in our big data environment and to continue supporting the system for child welfare out of the cloud.

Joyce Rose: Thank you, Joe. The whole notion of programmatic big data I think is a fascinating topic. We might want to add that to a list for a future webinar. Before we move on to Maryland, I have a question. Joe, you mentioned that you are using the Amazon public cloud. Arizona – Linda, Jay is Azure a public cloud or a private cloud?

Jay Cline: Azure is a public cloud, but Azure has two main identities, they have the commercial side for your standard businesses and then they have the Government side, which is reserved specifically for state and federal entities. The difference between those two is a lot of times on the capabilities and the certifications that they provide in terms of level of Fed Ramp certification, the type of data you can host out of the box without

third party options. The commercial side is very close in terms of certifications to the government side, but for those agencies that need to store much more sensitive data, such as HIPAA type data, the government side tends to be better in terms of making sure those certifications exist and are provable when you have an audit. The only problem, I will say, with the government side is that it does lag six months in terms of the features that are available on commercial side. It is a double-edged sword because you may lag six months behind but by the time it gets to you it is very well proven and very unlikely to cause any issues when you decide to use that feature.

Joyce Rose: Great, thank you Jay. Valli, how about Maryland’s cloud architecture please. Next slide, Irene.

Vallimanalan Thirugnanam: Hi, this is Valli Thirugnanam. I just heard both Arizona and Florida. I

think we kind of have a similar type of set up here, but we are a little bit different in terms of what we are hosting here. We are using Amazon public cloud. I heard Arizona talk about using their government cloud, so we are not using government cloud. We are using a public cloud and working on getting the necessary certifications for HIPAA as well. I will start and give you an overview of the programs we are engaging in the state of Maryland. This is called MD Think. It is the Maryland Total Human Services Integrated Network. It is an ambitious project that the Governor’s office started last year. The idea here is to bring together all the government agencies in the state of Maryland and to share the infrastructure and platform as a service. We chose to go with the public Cloud. What we have done in this program since last year is build the platform architecture including the infrastructure. The idea is to bring together the various agency applications and plug them into the platform itself. It is a dedicated platform that we built for the state of Maryland. In terms of how we have architecture solutions, we actually kept it very simple. Obviously, we have all the layers to our security, data protection, access, rights… and everything is managed within Amazon itself using various products. If you start from the left-hand side, we have the state security administration staff that manage all the user accounts. We also created the consumer portal and the caseworker portal; both will be managed by this team. Then we have the customer portal, which will be used by the clients, so we are talking about multiple agencies and child welfare is one of the applications, we are also bringing the federal eligibility programs as well as Medicaid into this platform. We are also bringing the child support, as well, onto this platform. Three major applications are being built simultaneously. Architecture wise we do have mobile application support. In the MD Think platform level that sits on top of the cloud infrastructure hosted by AWS, all the service calls are

managed by API and we focus heavily on micro service architecture. We do have a typical web application and database layer. Each of the agency applications are built with the architecture using the MD Think platform for that. On the core platform services side, we have listed all the key ones here. The enterprise service hub helps to manage real time data exchanges, we do have exchanges with the federal data hub and there are other federal systems to support our child support program. We do have all those real-time data exchanges managed by Enterprise Service Hub. We have Rules Engine, which is managed by our Corticon products. DevOps, which I will touch more on during the Q&A session, but we do have extensive set of platform specific tools for DevOps support. API Management we have [unknown] that we have used for securing all of our API. Identity management is one of the key things we use out of box from two vendors, which are available in AWS Cloud. One of the things I believe Joe, I think Arizona talked about was log management security and monitoring and we have a Splunk product to do that. We have various BI, reporting, analytics products we use in our platform as well. The bottom four platform services: Master Data Management, Enterprise Control management, Enterprise Search, and Data Lake, we have completed building all of these within the last 18 months. What we have done here is create this platform as a service model, which allows any application that can interact directly with full data-related platform service, which can be used directly as a plug and play design. One of the key things I want to highlight is the change data capture (CDC). This is the methodology we are using to keep the Data Lake up to date. From various application databases, we are capturing the data and pushing it to the Data Lake. The Data Lake will be consumed by our data scientists and as needed we may open it up for public consumption after we identify any of the sensitive information. On the right-hand side, we have highlighted the different system exchanges. The current Cloud infrastructure and MD THINK platform is in production, we have two pilot applications that we have put into production already. As of last month, we released 200 user types of applications for one of the family investment agencies. That is going very well at this point. Some of the key points I heard from Arizona, as well as Florida, are managing the vendors and keeping the infrastructure in place. It is not just the Cloud vendor in place, it is more about making sure the workers and the consumers are able to access the application when they need to without any delay. The infrastructure outside of the cloud is very important and that is one of the key aspects that the state of Maryland focused on including building and extending bandwidth for a lot of our agency offices. We will talk more about some of the highlights of how we manage the government and the vendors and what are the various things that we learned so far in the Q&A session.

Joyce Rose: Okay thank you Valli. I think we can all agree that it takes a significant amount of planning and there is a heck of a lot of detail in designing and implementing a cloud architecture – It is certainly not for the faint of heart. Let us begin our panel discussion. Next slide, please.

I am going to plant the seed here and get our panelists into a discussion. We are going to start with Joe. I am going to ask, which governance model has Florida employed within the cloud?

Joe Vastola: Hi Joyce, this is Joe. I think a couple areas of governance that we have had to address. Since our whole system is there, quite frankly, all of the processes associated with change management, approval, all of the governance-application governance processes had to be modified. Mostly that has been because we are dealing with dollars a bit differently. In our state data center, we had a state budget that was capped, it was an appropriated budget that was defined by the state legislature and we did not have control of the changes or the up and down changes of the costs associated with some of the various projects and enhancements that we would make. That all changes with the cloud. Now with the cloud, we are dealing with real money that changes the minute you make a change. First of all, our processes for overall management of the financial systems had to change and we had to establish a whole set of processes to be able to review and understand the cost of changes and estimate those costs for Cloud infrastructure, for managed services, for network capacity changes, for storage; basically to identify all of those costs and the impact that they would have immediately on change and establish a process for review and a budget. The management infrastructure for the cloud for us is probably governance-wise the key area that we had to spend a lot of time and we learned a lot about. That has extended also, in some cases, to the way we charge back our customers. In our state data center, it was relatively easy for us to determine funding by customer, which we have to do, as everybody would know to charge back the right state and federal programs because our state agency, our state technology that was running that infrastructure would determine the costs and give it to us separated by our primary application. So now, we have one system running in the cloud and other multiple programs that feed information to and from the cloud and that require those services. We have mixed networking that traverses the path from the department to the cloud. We started to look at some of the challenges around how we make sure that the billing and the allocation of cost for resources that we put up into the cloud are accurate and properly reflected. In some cases, it is easy to track – in the cloud, I know exactly what a server and storage and all of the elements that make up a particular environment cost, but I do not necessarily have that same breakdown for network cost and traffic that flow over our network. Governance for our perspective was primarily around overall application

and operational management that has included the way we estimate and communicate with our state and federal partners. All of that billing and all of the estimates we need to make a cost benefit analysis we need to take that into account when before it was really handled by our state agency and the data center staff that were there.

Joyce Rose: Thank you, Joe. Linda, Jay, or Valli do you have any additional information regarding governance models that you have employed?

Linda Jewell: Hi, this is Linda from Arizona. I will weigh in here. I look at - I don’t really see that our governance model really changed, we just looked at the cloud as just in another location that our applications were hosted as well as some of our other infrastructure, although Jay may disagree on that. As far as really understanding like what Joe said before, was really understanding your change process, but the cost that is where you really need to understand and touch in on vendor management, which we will cover in a little bit, but as far as the governance I didn’t really see anything changing, but understanding how the cost operates the throughput you need in your network to maintain good connectivity to your cloud services. Also, how to onboard constantly changing cloud services, so that at least on the Azure side there are new services being spun up that you can utilize in an Azure environment a lot. So, you need to know how to onboard and accept those changes in your environment.

Joyce Rose: And yes, we are going to get into a deeper discussion about payment structures and other costs. Valli, do you have comments regarding the Maryland Governance model?

Vallimanalan Thirugnanam: Yes, I have a couple of comments. I think I agree with both Joe and Linda

that we actually have most of the time getting into agreements with the agencies on how the cost sharing is going to work, that is one of the key things. The good thing is that most of the agencies are already on boarded with the idea of sharing a platform, so they signed a MOU (Memorandum of Understanding) from the beginning so that helped us to work out the details rather than how to get the agencies to come on board. Cost is one factor, but the governance on how the licensing of various products that each agency may be using and the sharing of that was one of the key things. We always struggled working with product vendors; bringing their licenses into a cloud environment and having them share with an entire state was always a challenge. We have a very good leader team that takes care of that working towards procurement and we were able to navigate to most of the challenges with the end user license agreements, the SLAs with the product vendors. That is one of the key challenges we faced, but

we were able to move beyond that. The key point here is getting back early enough in the governance model to resolve all those issues.

Joyce Rose: That is a great segue into our next question. What are the changes needed in contract language or licensing or payment structures with cloud vendors?

Jay Cline: Hi, this is Jay. Contract language is an interesting thing because that’s something I do not really think you’re going to need to change that much contract language is difficult to deal with no matter what, there are obviously certain key things you will need to look at when dealing with a cloud vendor, and every vendor is going to be slightly different in terms of how they deal with these things. So depending upon how you use the service, everything will be different. So with SLA, service level agreements, when there is an outage, how does the vendor deal with it and how do you deal with it, these need to be clearly outlined. The interesting thing with SLAs and these types of contracts is that they may outline a generally 99.9% up time, but that 99.9% up time usually has an asterisk next to it, stating that only some types of services may be available for that up time, so you really need to dig into the details of what you are looking at there. Plus, I have seen contracts before, and we have dealt with them specifically here at the state, where that SLA and that downtime only comes into effect when you have actually reported it to the vendor, so for the sake of argument, if a service goes down for the weekend and you don’t report that to the vendor, that 48 hours of downtime technically doesn’t count against their SLA it only counts on Monday morning when somebody complains that the service is down. How they deal and count that SLA time makes a big difference across those vendors. With SLAs you also want to have good visibility into their service health. One problem that cloud vendors have is providing a level of transparency that looks at the actual base level of service and service health that you are running off them. They will have basic ones that say that the core services are functioning, but figuring out what is actually impacted on these high-level dashboards that they provide is not easy to do. You have to get into a lot of vendor management to even get into the level of figuring out what is happening and actually impacted and get those details and move across from there. The other aspect is data. Data is a huge one when dealing with these contracts because it all comes down to what you are storing in the cloud and obviously dealing with the government side of things you are dealing with very sensitive data most of the time. From our aspect we are dealing with a lot of PII, the HIPAA style data, and making sure that whatever vendor you are utilizing has the ability of meeting the criteria to host that data is one thing. Again, there is always an asterisk when it comes to these things and that is specifically stating that they may be able to host that in a secure manner, but it may not be done out of the box there

may be extra configurations that you need to look at and you need to do to make sure that data actually meets something like HIPAA compliance. I will give a basic example of Google. Google Drive is very easily available to make HIPAA compliant, but out of the box if you put information inside Google Drive it is not HIPAA compliant, they will give you the steps to actually make it HIPAA compliant, but you need to go through an extra set of requirements to make that happen. When going back to data, it is about retention. If you decide to leave a vendor what happens to that data, is it destroyed, is it given back to you, do you have verification that the data no longer lives inside that vendor’s environment. A lot of these agencies, ours included, lives and breathes on the data we are able to provide, so data management is one of those key aspects of looking at this and one of the highest on the list, in my personal opinion.

Linda Jewell: Thanks, Jay. This is Linda. I want to tag onto that a little bit too. That is within the cloud environment. It is also really important to understand how you need to get data to the cloud environment. If you saw our beginning slide, where we showed all of our different types of network connections that we have there, those were also considerations that we required to make our cloud healthy, did we understand the throughput required? If we have 2800 employees and close to 8000 connecting vendors coming up through our cloud, do we have the right throughput? Getting those contracts, and negotiations, and even hardware in place to be able to handle those dedicated network connections.

Joyce Rose: Joe or Valli?

Joe Vastola: This is Joe, just to add to that and again setting the context here our picture is one of running our child welfare system up there. So, we have had a number of interesting contract dynamics we have had to deal with here. First of all, chances are when you are going to the cloud, I can’t speak for Microsoft, but you are going to be buying cloud services from a re-seller, you won’t be buying directly from Amazon; I think Google also uses re-sellers.. I think with Microsoft you can buy it either way. When you are looking at SLAs its good to look at the what the Cloud providers SLAs are because the real ones you are going to live and die by are from your managed service provider. There are interesting and new elements that go into contracts when you are setting up these environments and some of it relates to the model that you plan to put in place. And by that as example, specifically things like are you and your staff going to be responsible for changing and modifying the environment; increasing and adding resources to it, even to push that to the managed services provider. You have to decide those things because you will have to contractually deal with the keys to the account, make those changes, and how the penalties relate to those kinds of changes.

I think as an additional consideration, there is also a decision that you have to make about who owns the security of your Cloud environment. That is one thing we worked really hard to understand and make sure we had locked up appropriately, like we should. But coming out of our state data center, the state data center owned that security environment. By getting a cloud service provider – a managed service provider that does not necessarily set up the responsibilities for who needs to secure that environment. There are additional services to consider, if you are going to have the managed service provider have SLAs or penalties around security. Those are the kinds of things that we found to be different around contracting in terms of elements, even though we had SLAs with our state data center that we did not have to deal with, that they had to be put into contracts as we went through this process.

Joyce Rose: Thank you, Joe.

Vallimanalan Thirugnanam: Yes, that is a great point, Joe. We thought about that the same way and we

obviously had to use the re-seller, but one thing we did different was we wanted to go with a reseller that had some reputation and that could take care of issues if something went wrong. We took some time to narrow down to setup managed providers who can be a reseller. It took, obviously, months for us to work it out with them, but it was one of the key considerations. I am talking about for any state, if you are going to go with a cloud provider; you have to think about all of these key aspects. Those are the key areas to consider since states typically have control when they have their own data center, but you are probably giving the control away when you are going to the cloud. But there are several other mechanisms that we have in place. From the state government side, we have a security team that manages that part. For onboarding users, we actually have control of all of that. That takes the risk away from the vendor perspective, but we control who gets and who has access and we also have the ability to monitor what is getting out of the cloud as well.

Joyce Rose: Thank you that is very, very interesting. I am sure our attendees’ ears are burning right now. Before we leave the discussion about cost and payment structures, I am going to ask a question about egress cost and understanding that uploading data to the internet, which goes out of the local network, is the definition of egress, what should states be aware of regarding those costs? Valli would you like to start us off, please?

Vallimanalan Thirugnanam: Yes, that is one of the - this is Valli - is one of the key considerations, we

just started this entire program a little over a year ago. We had some estimation initially working at the current usage of the agency application,

and the expecting a similar or more than the usage of what was trending. The good news is on what we have been able to observe, we were able to control the cost of the egress, but we can only control to a certain extent because every agency has the right to get the data out of the platform for their own use. We are still trying to figure out a method on how to actually keep this in an optimized way. We are still figuring it out. One of the ways we may find out, maybe a couple of years from now once we have identified trending we should be able to put some control around this and find a way to upload some of the data, using a snowball option. We still have some of the data centers in the state where we could offload some of the data without using the bandwidth. That is something that we are still considering. That is one of the aspects when we negotiated with the reseller on how we are actually going to bulk download data without impacting our cost. That is one of the key things that we considered.

Joyce Rose: Linda, Jay, Joe?

Jay Cline: This is Jay. Egress costs are an interesting thing because depending on the actual cloud vendor, egress costs may be more or less. I will use, obviously state of Arizona as an example, you can look at egress costs from the state data center to the cloud, but you also have to look at egress costs from the cloud to external data centers back to your state agencies, people who are consuming that data out of that cloud environment. When you start shifting an infrastructure from an on-prem style environment to a cloud environment, one of the obvious things that changes is that you have less structure on-prem since you are shifting to the cloud, but you have to bolster that usually with some network infrastructure on the back end, so you can support now all the connectivity required to support all that in and out data now out of your physical clients who are sitting technically on-prem and your cloud based resources. The other aspect you have is, using Azure for an example, is that egress costs out of Azure are actual costs that you have to deal with on a monthly basis, but it depends on the type of traffic to whether or not you are actually charged. Looking at the resources you are dealing with and making sure that you are grouping proper resources together so that you are not doing anything like splitting a resource that does a lot of data consumption, so part of it is on-prem and some of it is in the cloud to try to reduce that amount of traffic and the obvious amount of cost. Going back to Azure as an example, they do allow some types of egress traffic to be 100% free, they want you to use certain services, so they give this to you as a benefit, that example mainly being as backup and restore. I can back up all my physical infrastructure on-prem to a storage infrastructure sitting inside Azure and if one of those resources dies and I need to restore, that egress traffic is 100% free and none is charged to me because they want me to consume that service. So, it really depends what you are using these things for.

Joyce Rose: Joe, Linda?

Joe Vastola: This is Joe. Just one additional point to add to all of that good information, when you are considering your data movement costs it is not just egress from the cloud to you, it may also be communication based on the structure of your cloud provider and the solution you have up there. What I mean by this is, for example, with Amazon they have different availability zones, which are data centers throughout the country and depending on how you structure your application you take advantage of availability zones and you run instances of your application across them for immediate to near immediate disaster recovery. In Amazon when you are spamming availability zones, you are also paying for the network traffic that goes between them. So, it’s not only coming out of the cloud, depending upon your cloud provider and the structure of your solution, you may also be paying for data that moves from within the cloud and its structure.

Joyce Rose: Very interesting. Linda do you have additional comments?

Linda Jewell: No, I do not have anything to add on egress I think that everyone covered the topic quite well actually.

Joyce Rose: Cool, okay! Let us move on to another topic, and that is personnel. Valli I am going to ask you to start. What considerations should states give to the type and number of IT staff to support the cloud environment?

Vallimanalan Thirugnanam: Sure, yeah. This is Valli. This is one of the interesting topics, since this is

something that I manage for the state of Maryland. When we take applications from within the cloud, we try to use as many products as possible, doing any kind of customization. What this means is that we have to have specialized resources with experience dealing with specific products. When we were dealing with the mainframe base and the client server, architecture base, we might be able to use resources who work with the multiple technology. Unfortunately, with the cloud computing, there are resources who are specialized in specific products that are the only thing they actually do. We want isolation in terms of separation of duty, and to optimize resources available to get things done efficiently and quickly. So, this posed a lot of challenges for us. When we tried to find a way to get these resources, mostly on the vendor side, as well as on the state side, we had to find resources with specific skills. Then there were a lot of areas where we had a lot of overlap, so we had to consider all those factors before we can get the resources with the specific skill-set hired. One of the challenges we currently still face is when we tried to hire resources through our vendors, they also tried to find these specialized resources and 6 out of 10 times they are struggling to find the right

resource for us. But, we are taking our own time and doing some research in the market to find the right resources, mostly we are using our IDIQ type of contracts, we are not really using one vendor to do all of our work-related infrastructure or application development. We actually are kind of putting the structure in place in identifying the resources and it is one of the challenges on the state side as well. With the cloud computing a lot of folks who are familiar with the mainframe architecture they have to be retrained to learn about the new technology. That is another area that we continue to work on to include their insight as well.

Joyce Rose: Linda, how did Arizona approach considerations regarding IT staff supporting your environment?

Linda Jewell: It is an interesting dynamic and we are a little bit different than what Maryland had to overcome. As Jay had mentioned earlier, we were separating from another agency so we built our IT group from the ground up which means we went cloud first if you want to call that a strategy. Everything that we wanted to do we built in the cloud, likewise when we went to hire we went to hire resources we looked at it specifically for building the cloud infrastructure. I can tell you that we have 90 servers up in the cloud; we have all different kinds of connections going up. By the end of this, we will probably have between 120 and 150 servers when we are done with our CCWIS implementation. With that, I have one infrastructure architect, we have three server admins. We have been really hiring for attitude and then we train on the aptitude for managing in the cloud. There are not skillsets out there required to manage the kind of infrastructures that we are building, at least not at the salaries that we pay at the state, so we are looking at ways that we can train, use our premier hours to have some trainings for the staff. But, I can tell you that we are lean, we have really good standard operating procedures. We understand our security; we do not have managed service providers that do a lot of those services for us. We do not deal with resellers; we deal directly with Microsoft itself. So I think that has helped us cut and that has helped us better manage our capacity as far as staffing goes and the functions we provide. The other thing to that, and we kind of touched on it before… when it came to the governance, you have to have a really structured way so that while you may have ways for people to connect, it is a different skillset, there is more to managing cloud infrastructure and understanding those reports that Jay said were a little bit nebulous and knowing how to read them and determining the health of your environment. Understanding change and release management, understanding the impacts, having good incident management process that you know when a service is down and how it is impacting. Those are the soft skills that you need to build up, not just being able to manage cloud services.

Joyce Rose: Great, you mentioned the ability to be nimble to changes and upgrades from cloud vendors. Joe, I am going to ask you, what is it that you have done that allowed you to be nimble to those changes and upgrades?

Joe Vastola: I think that the best thing we did to help us through this process was that we picked a group of people and we educated them on what they needed to do, and we supplemented with external help. So, there are really two kinds of skillsets that you need to move and operate systems and solutions in the cloud. One of our great wrench turners that we all have is how to create an architect and develop a cloud environment. Those are tough resources to train and to find, and retain, because there is a lot of demand for them. In our case, we have a small group of those folks who work with our managed service provider to keep things in check. And then we rely on the agility in scaling our contracted services with managed service providers and application providers.

The second area, the second skillset that helps us to move - to keep things moving smoothly is what I will call cloud management. I think Linda touched a little on that. You have different skills you need if you are managing an account in the cloud, you are monitoring that cloud. We had examples where even though we are paying all these service providers, we find an instance where a storage bucket is not connected to something that is showing up on our bill. For whatever reason we needed it during testing and we do not need it anymore, but it has not been removed so you have a whole monitoring process you have to create to keep that environment efficient and clean. We had differences in the way that folks showed up to audit our environment, so often we are educating those people. I would sum it up as talking about those two focus points around the architecture and the work you do to create, manage, monitor, and enhance the cloud environment and then the work you do to manage the peripheral elements that keep it running smoothly.

Joyce Rose: Thank you. Related to bill and release cycles, what complexity and time issues were introduced from utilizing the cloud? I will ask Linda and Jay to start and Valli to contribute.

Linda Jewell: One of the really important things to understand is how you actually move and set up your environment to be efficient. There are two parts of this. In an on prem situation, you can have dedicated build, test, and train environments and you really need to rethink that in a cloud environment as well. Do you really need all the infrastructure up all of the time to manage throughout your build? The other thing is really making sure you understand and leverage the scripting and the buildup of these environments as you move through your build and test cycles. Understanding how you move and leverage tools like VFTF to move your

configurations through. I think those are important considerations on what having dedicated environments will cost you in money and storage, which you might not have considered before. It is important to understand in your build and release cycle to understand exactly when you need those instances and how to shut them off.

Joyce Rose: Valli?

Vallimanalan Thirugnanam: That is one of the key aspects of setting the environment and making sure

that release is happening without any issue. Typically, when you have an on-premise environment, you have a dedicated team, but with the cloud environment and multiple application teams working it really becomes a bottleneck when you have a team that manages the release cycle, setting up the environment. We rely on automation, we are close to about 60-70% of automating infrastructure building and close to 90% for the build and release cycle management using the various scripting tools. One of the key things is when we build these environments, in a typical architecture when you build with high availability, we talked about the disaster zones and other things, we tend to build the architecture to mimic what it will look like. After our initial testing and everything is confirmed, we do not have to maintain that kind of infrastructure. That is one of the lessons learned from our operations, so far; we do not have to maintain a similar infrastructure in a non-production environment. What that requires is a little bit of change in the way we architect the solutions and help the architecture to dynamically scale rather than to have your entire architecture rely on the concentration that you are defining in the beginning. There is a little bit of change in the architecture and scripting that help to achieve that. The pipeline for our CADC, automation for build and release, we rely mostly on the development team and the testing team to automate all of this. The DevOps team job is to help the initial set up and from that point to monitor and look at the log and try to identify any anomalies. That is how the setup we have in play works. Our goal is to have 100% end to end DevOps release cycles including automation of testing of our scripts. We are building an automation test library with various scenarios for different applications. We have a dedicated team that works closely with our development team to create those scripts. This is an interesting topic, and I would say the ever-changing nature of various products and tools available in the market, the more you think about the change that is happening, it will be very hard to keep up with the changes. What we have decided to do is that we do not have to use all the available services and products that are claiming, it is going to improve efficiency. We will look at this and if required we will adapt to this. That is our strategy so far.

Joyce Rose: Great. So that we save time for our audience Q&A, we are going to move on to lessons learned. I am going to ask Joe to start. Let us spend about a couple minutes each. Let us talk about the most significant lessons that you have learned. Joe?

Joe Vastola: Okay, this is Joe. The lessons span a number of areas. I think first, I would say, early on we spent a lot of time working over SLAs and what we thought would be the right SLAs. Ultimately that ended up not being a wise use of time because, quite frankly, you are going to do a deal with a cloud vendor and they are going to give you their SLAs. Amazon is not going to negotiate with the Department of Children and Families here in Florida and change their whole national structure for SLAs because we put one system up in the cloud. I tell people that on the one page of important things when I started this is different now. We learned a lot about the reality of software licenses. You are hearing a lot of good information about how to spin up new environments and take advantage of the cloud and expand that environment as you need it and contract it as you do not. That is true with the cloud but not necessarily true with software licenses. You can only span to the level of license that you have. And depending on the solution of the vendor products that you have got there, if they are not all subscription based, then you are probably limited in what you can expand to. Making sure that the license approach that you take when you start is appropriate and spending time on that is well worth it. We went through a lot with the number of vendors on licenses, down to where vendors even told us that our on premise license that we have in our data center did not transition to an in-cloud license. In some cases, you need to expect your vendors to not have answers for you because they might not have all of these things worked out and you have to work through it with them.

I think we really underestimated the amount of change that we would have to go through just in managing the cloud. You are not just changing an application or an infrastructure in a move to a cloud; you are putting contracts in place you are dealing with all these things you have heard discussed on the webinar today. There is an amount of work there that is not a part of your normal project that you have to make sure you are staffed and ready to do. I know from our perspective that we had a lot of interest from our legislative staff in what we were doing and trying to do with the funds they had provided to us so as you start out on this course I think there is time well spent in marketing and communicating where you are going and what you are doing.

In addition to those areas, the procurement process, the more you can leverage something that is in place. What we had planned to do was an invitation to negotiate in Florida, which boils down to the longest and

hardest procurement process that you ever want to apply. When we looked, we found a national contract that we were able to have approved for use here in Florida that turned our procurement into a RFQ based procurement and that took six months off of our project just on the front end. Look at the contracts that are available and make sure you are leveraging opportunity, however your state handles procurement and whatever your requirements are. Consider penalties, we worked with a number of managed service providers while we were procuring, and penalties were something they were surprisingly ill prepared to discuss. I believe when we are talking about moving systems like a child welfare system or public assistance system, business systems with overnight batch cycles and all of these nuances necessary to complete a cycle of financials and everything. And most of the managed service providers we talked to were used to solutions that, for example our managed service provider runs the online site for the NFL, so they are used to on game day getting so much traffic that they expand that infrastructure so that they can handle that traffic and condensing it again on non-game days, but they were not ready for systems with overnight cycles. There may be some curveballs you throw at them with the type of systems and the requirements you have, state contracts, some of the liability elements. I think it was a surprise to me that they were a little unprepared for those discussions. Ultimately whomever you select, depending upon your model you will have to help them through it. It is important to consider early on your approach, you are either moving workload or database function or infrastructure, you are lifting and shifting an application or you are re-engineering it to take advantage of your cloud provider and infrastructure enables. I would think about all of the benefits that there are, because there are benefits for each one of those and you make sure you are maximizing those benefits. Once you implement it, I really think it comes to staffing and the elements of how you operate and manage a system that is running in the cloud and the application that you have to consider. Things like ongoing agreements, when we create new instances in the cloud, for a while we were going through a process where our managed services provider had to submit a new statement of work and they didn’t have an a la carte menu of costs for substantiating and architecting and operationalizing an instance. So, we worked through those activities to be as flexible to operate, as agile as we can, to move forward. In some cases, we got to the point where it was taking almost as long to gen up a resource in the cloud with all these dynamics and statements of work that might as well have not been in the cloud. So, we kind of challenged ourselves to say we are in the cloud and we have these on demand services, how do we align our processes and all the things we do to make it efficient.

Joyce Rose: Joe, for the sake of time, we have several questions in the queue, but I want to give Linda and Jay and Valli a chance to add what they believe was their most significant lesson learned. Please go ahead.

Jay Cline: Hi, this is Jay. One of the biggest things we had to deal with and rethink about was vendor relationship management, both on the boots on the ground side that being our server team all the way up to infrastructure level, contract level, CIO level, indeed. We have an extremely tight relationship now with Microsoft, we have Microsoft on premises at least once a week to discuss what the new services they are offering in the cloud, what is your roadmap coming down the pipeline, so we can create these plans and know and adjust these services if we need to directly. And that we are using the current services to the best of their abilities. Microsoft is constantly adjusting things like costing models and availability solutions, being able to go through and readdress our current services to new costing models and new options that they release on a continual basis allows us to be always on the edge to make sure we are spending money correctly, spending money wisely, and utilizing the services to their fullest extent. And going back on some of the IT personnel issues, our server operation people, and we all know how much they like to interact with other people…, and the big thing is that they need to interact with vendors now. If we are majority cloud environment, my server level people, if there is something wrong with the hosted service itself they now need to interact with other people since they can’t solve the problem on their own, it is not just talking to another guy in an office or a cube, it is getting someone on the phone and it is creating these channels, both direct channels - creating tickets inside a system - as well as back channels to people that you have dealt with at these vendors constantly to ask them if there is an issue and what is going on, can I get a heads up on something. Creating those really tight relationships with vendors is going to make sure that you understand everything you need to in terms of the current roadmap and future roadmap, current licensing issues, making sure that you are using these services correctly, and making sure that your staff has those open channels to communicate what they need to and to get the answers that they need as efficiently as possible.

Joyce Rose: Thank you, Jay. We are going to move now into the attendee Q&A, we will give Linda and Valli the first chance to answer these questions given that they did not provide their lessons learned. So, that concludes the discussion portion, I would like to invite my colleague Nick to run the Q&A session. Nick.

Nick Mozer: Hello there everyone, this is Nick Mozer, Junior Analyst for the Division of State Systems. I am going to read our first question and it actually is directly for Florida, so sorry for the mix up there. But, Joe did you have

any difficulty integrating the cloud CCWIS with on-prem medical and benefit systems? The follow up question is; do you use MMIS to validate and pull client information into the CCWIS?

Joe Vastola: So, I have to confess that our CCWIS is our SACWIS re-named, so we are at the point where we are trying to determine how to get data in and out of the system of the final architecture for where it will be. Ultimately, we are not seeing any issues moving data because we do move data from other systems. So, we are moving benefits information in and out from the system it is connected to the public assistance system. We have a robust interface there for Medicaid eligibility determination and all of the data that has to flow. Basically, the cloud is sitting as a component on our network and really, once we resolved all of the network connections, it is an interesting process. It really is like an application in our network with the right capacity given the size of the pipeline that we have to move data back and forth. Our challenge, too, is that we receive data from external providers, our child welfare system has an outsource component to it on the service side so all of that data still flows in from our state network through to our application to the network in the cloud and then to our CCWIS.

Nick Mozer: Thank you, very much. Operator, do we have any questions over the phone and if so can you remind participants about how they can ask questions, please.

Operator: Thank you, so much. As a reminder, if you have a question over the phone line please press star then 1. Unmute your phone and record your name clearly, so I may introduce your question. Presently I have no questions in queue but again, please press star then 1 at any time.

Nick Mozer: Thank you very much. And just as a note, this has been such a fascinating presentation/webinar and we want to be respectful of your time, so if you do have questions and if we are unable to get them answered within the timeframe we have allotted. With your permission, we would love to follow up with you and with the panelists afterwards, to get you the answers to those questions and ideally, we can post those online as well. It may be the case that a question you ask helps a fellow colleague that you have out there, perhaps in a different state. Thank you very much everybody and Operator do we have any questions in the queue in the interim.

Operator: No, we do not. Thank you.

Nick Mozer: Thank you. I will now read our next question. The next question is for Maryland. What is the vehicle for your real-time data exchanges, i.e. what tool, enterprise service bus, etc.

Vallimanalan Thirugnanam: For our real-time data exchanges, we are using the Apache products. One

of the things we decided to do from the beginning was to use for all of our products to be open source. We have used the Apache product to build the enterprise service bus model; it is a very simple system that connects in a simple way with the external or internal real-time data exchanges. We use XML and JSON based exchanges. One of the things we decided to do, as I said in the very beginning was to keep it very simple, that was one of the lessons learned; keep it simple solution in a repeatable way to leverage the same code as much as possible for all of our data exchanges, for real time, as well as for validated exchanges as well.

Nick Mozer: Thank you, very much. Operator, do we have any questions over the phone?

Operator: No, we do not, please press star then 1 to unmute your phone and record your name if you do have a question. Again, I have nothing at this time

Nick Mozer: Thank you. Joyce, I am going to do a quick time check, at what time do you want to transition to concluding this webinar, I want to inquire if we have room for another question.

Joyce Rose: You have room for one question and maybe two.

Nick Mozer: Thank you. I have seen a couple different questions related to cost, so I am going to combine a few. So please bear with me.

I am curious if your state IT organizations led or hindered your organizations transition to the cloud, how well your state IT organization does in cloud management and governance, and whether is in any redundancy between roles in what is offered and automated on the cloud. In regarding rates, do state IT charges negate the financial benefits of moving to the cloud? An additional follow up from another participant – do any of you believe that this transition to cloud services has saved you money compared to the previous system?

Vallimanalan Thirugnanam: This is Valli. I just wanted to quickly point out that our analysis is that we

will be saving several millions of dollars. We can slice and dice this in many ways, but we realized that consolidating many data centers that we use within the state of Maryland, many agencies have their own IT for hosting alone, bringing everybody into one platform, and using the infrastructure as a service and platform as a service is expected to save us several millions of dollars. Cost saving is definitely there no question about it. In terms of how we are going to achieve it, we are going to share the various resources, whether IT or licenses, those are the things that

require additional discussion. Once the agencies agree, the cloud model will benefit the entire state and individual agencies in the long run.

Joe Vastola: This is Joe. Just to add to that, here in Florida, I would say that our state IT organization helped, they had to approve our procurement mechanism for alternative IT services. There was not a lot they could help with in terms of selecting a cloud provider because in many cases it was in advance of them doing it themselves. There is certainly redundancy in basically all of it, we have infrastructure as a service in the cloud that is provided by the state in the data center and we have managed services that is also provided. As far as savings go, when I look at our numbers we are running at 50% of the cost to the state that it cost them to run the application in our state data center and the application is actually performing at a higher level, so it services more users. And then, I think the last piece was something around IT state charges. Once we moved in December to the cloud, our budget for the system continued to be applied to the state data center for the year. There was a whole set of dynamics that we worked with our state legislative staff and our data center around given the reality for how our state data center is funded.

Jay Cline: Hi, this is Jay. There are a lot of different perspectives to look at when you are dealing with potential cost savings and actual cost savings. Being able to properly leverage the resources if you do it correctly, at a minimum you are going to be able to reap a lot of benefits that you potentially would never be able to buy beforehand. I would argue that there are very few if any states in the nation that would be able to afford a fully redundant infrastructure that a cloud provider could give you if you properly leverage those services such as having those resources replicated in multiple physical data centers, having data replicated between multiple racks within that data center. These are things that are just very expensive components to buy and maintain on an ongoing basis. Also taking use of the consume – and basically buy and consume at the same time, one of the easiest things to look at to save money is a backup and restore structure, being able to build that on-prem you have to buy the majority of the storage that you would use right off the bat, whether you are going to use it or not you are going to have to forecast what you are going to use and that is a very large cost outlay. Using something on a consume basis, it allows us to use that infrastructure for only what we need but expand at an infinite level and use all of what we need at any point in time without having that huge initial cost outlay and then of course on-prem management after that. In terms of dealing with the state charges one of the big things about that is making sure that you are justifying these costs and understanding the difference between owning the IT infrastructure, versus essentially renting the IT infrastructure at this point in time. Being able to properly explain that to them so that they understand these things

and the reason why in some cases it is better to go to a cloud infrastructure than hosting it on-prem then we find those hurdles are fairly easy to get over. That can be a fairly large hurdle to properly explain correctly.

Nick Mozer: Thank you, very much. This is Nick Mozer, again. We have run out of time for the question and answer session. If anyone has additional questions that they would like answered, feel free to enter them into the chat box and we will follow up with you afterwards. Panelists, thank you for everything you contributed today, this has been fascinating. I am stealing Joyce Rose's thunder though, so with that, I will transition to Joyce. Thank you, very much.

Joyce Rose: Thanks, Nick. I think we could go for another hour in discussing this particular topic, very interesting and fascinating. On behalf of the Children Bureau I would like to thank our panelists Linda, Jay, Valli, and Joe. If you would like to contact them directly, their contact info is listed. This webinar has been recorded and will be made available online. When it is complete and posted, a message will be sent announcing availability on the Children’s Bureau website. In terms of upcoming events, those topics are under discussion right now, and the schedule will be posted starting in October. Thank you all for attending and that ends the roundtable discussion. Thank you and goodbye.

Operator Conference has now concluded, thank you for attending. Please go ahead and disconnect.

END

State Panel Discussion: Cloud Computing Webinar...Webinar Series Cloud Computing State Panel...

Documents

Transcript of State Panel Discussion: Cloud Computing Webinar...Webinar Series Cloud Computing State Panel...