5. Mirroring Databases for High Availability (Bob Binstock)

Speaker 0 00:00:01 Welcome to data points, a podcast by InterSystems learning services. Make sure to subscribe to the podcast on your favorite podcast app, such as Spotify, Apple podcasts, Google play, or Stitcher. You can do this by searching for data points and hitting that subscribe button. My name is Derek Robinson and on today's episode, I'll chat with Bob Benstock, a technical writer here at InterSystems about mirroring databases for high availability in InterSystems. Iris, Speaker 0 00:00:39 Welcome to episode five of data points by InterSystems learning services. As always. We hope you're enjoying the first month or so of data points. The homepage for the podcast is now data points.intersystems.com. So you have an easy landing page to go subscribe and to see all of the latest episodes today, I'm chatting with Bob Benstock about high availability and mirroring databases. Bob has a pretty wide purview of expertise being a technical writer here at inner systems. And one of the areas he's worked quite a bit on is mirroring in our discussion. We'll talk about some of the reasons that applications today really require ultra high availability and then how mirroring and InterSystems Iris and other products help to address that issue. Speaker 0 00:01:22 Welcome to the podcast, Bob Benstock, one of the technical writers here at InterSystems. Uh, Bob, how's it going? Good. How are you? Good, thanks. Um, so today we're going to be talking about mirroring and high availability, which is a, um, important topic for people that have applications running on InterSystems products or really any products I think, uh, in the world today and with the way that software development goes and application development goes. So, uh, before getting into the specifics of that technology and those features, let's talk about the concept of high availability in the first place. Um, what is high availability and why is it important for application development and for really any applications running today? Speaker 2 00:01:57 Well, what you just said is actually very accurate because more and more everybody needs, uh, their applications to be online 24 seven, three 65. We used to talk about mission critical applications that couldn't afford any downtime. But the fact is that organizations in healthcare, government, financial services, manufacturing, all sorts of areas live on the network now. So essentially everything they do is mission critical. I mean, it may not be, seems so critical to me that I can buy something on Amazon right now, but it is to them. Their business is in trouble if they have interruptions. So high availability basically means zero downtime or as little as you can manage. And, uh, that's what InterSystems Iris mirroring is for. Speaker 0 00:02:49 Right, right. So, uh, yeah, that's a great, the Amazon, one's a great example because, um, it, it matters to their business and it matters to them the way it is today. A lot of customers too, like you, it's amazing how you'd get mad as a customer now because you expect it to be so available and ready all the time. Speaker 2 00:03:04 Sure. And, and think about, um, healthcare, nobody wants healthcare to go back to the days when people were looking through stacks of binders to find information about a patient or about a medication. But if you think about it, healthcare happens around the clock and, um, clinicians and others need to be entering sharing and analyzing data around the clock. So as healthcare, more and more goes online, it has to be up all the time. And that's what high availability is about. Speaker 0 00:03:34 Yeah, exactly. So, uh, taking that into the, kind of how that is put into practice in the inner system stack of technology, particularly InterSystems Iris and other products, we'll get to, um, what is mirroring starting with Iris. And then you can kind of expand that to the other products that it's related to. Speaker 2 00:03:48 Sure. Uh, mirroring is based on the fail-over pair, which is a common approach in which a synchronized partner, uh, or standby system takes over. If the primary system fails or becomes unavailable. For example, there might be a network outage. The problem with most fail over base technologies is that they involve a single point of failure, both fail over nodes using the same storage or the system relies on physical copying from one storage device to another, which makes you vulnerable to a storage failure or possibly data corruption getting introduced while the copies made. So mirroring instead uses logical data replication, not physical. All of the needed information is passed over the network. So the synchronized backup copy of a database can be kept on a separate storage device or even in a separate data center. This eliminates that single point of failure. And as well as the possibility of data corruption. Speaker 2 00:04:45 So mirroring is, is based on journaling, which is one of the things that InterSystems Iris uses to recover from crashes. So every update to a database is saved in its journal files. If your instance goes down or the, you have any kind of problem, the first thing you will do is restore the most recent backup of that database, but you still have to take care of the updates that happen after that backup was made. Well, what InterSystems Iris does after the backup is read the journal files and apply every update made since the backup was taken to the databases, which means you're back to the state they were in when the crash happened. Uh, except of course that open transactions are rolled back, cause there's no way to close them. Now mirroring synchronizes the databases on the backup, fail over member with those on the primary by continually sending the updates that are in those journal files, from the primary to the backup where they're applied to the database copies just as they would be after a crash. Speaker 2 00:05:45 So basically you insert a row, uh, on the production database that insert gets passed to the backup and the same thing as done there, very, very the primary and the backup communicate synchronously with the backup, acknowledging each set of general updates that receives. So the mirror always knows if the backup has caught up that that is fully synchronized with the primary, right? So if there's an outage of the primary, when that is the case, when the databases are synchronized, which they typically will be, the mirror automatically fails over redirects applications to the back application, connections to the backup with zero data loss. And it usually is so quick that users don't even notice, uh, the backup is now the primary. And when the former primary comes back online, it automatically becomes the backup. Uh, this can also be triggered deliberately when you need to bring the primary down for maintenance or upgrades. Speaker 2 00:06:43 Uh, when you're done, you just trigger a fail over back to the original and primary. So sometimes one of them or databases may not be quite caught up. Uh, so in those cases, the backup usually can retrieve journal information from the primaries host. You know, it knows what, what journal information it's missing. The mirror knows the backup didn't have that latest update. And you can usually get that information from the primary host that might extend, fail over to just a few seconds, not long enough to cause any problem if databases aren't synchronized and the whole primary's host goes down, you might have to get into manual procedures, but the point is, uh, you can always quickly get back to where you were. Speaker 0 00:07:28 Right. Right. Yeah. So, so you, you talked about the, basically the mirror that that is your backup kind of knowing, uh, whether it's up to date with the primary. Right. Um, so let's say, I want to go back to one of the points you made about the journal file, kind of being able to fill in the gaps between those, if it's not up to date. Um, so let's say my primary goes down and my backup knows that it is not up to date because it basically has as, is not quite there to what the primary was at is the journal file accessible. Despite the fact that the primary went down, is that kind of, where do we, where do we get that journal file to make sure we can access it? Speaker 2 00:08:01 That's what I was saying about the diff it's a difference between when the InterSystems Iris instance is not an operation or when the host is down, or let's say unavailable, the journal files are just files on the host file system. So, uh, the, each, each mirror member has, what's called a mirror agent called the ISC agent, which does things like retrieving journal information from the primary host. So if the host is available, you can get that stuff. If it's not, you might have to make a decision between waiting until you can restore the host, uh, the primary host to get that information, or going ahead with fail over anyway, knowing that you've lost a little bit of data, but you'll know exactly what that is. You will be able to get that information available eventually. So you could ultimately, you can catch things up. Right? Exactly. So, Speaker 0 00:08:56 Uh, taking some of that, that's, I think a good explanation of the technology of mirroring and how it works and InterSystems Iris. Now my understanding is that for pretty much all of InterSystems products, this concept really works the same way as far as how mirroring functionally works. But what are some of the differences you can talk about when it comes to health share and at a high level, why there's more that you need to kind of know without just being able to do it as simply as you maybe can with InterSystems Iris, Speaker 2 00:09:21 That that's an excellent question. Um, one of the important things to remember about mirroring is that it synchronizes only the mirror databases. So information that's stored on the primary, but not on those databases, such as for instance, user accounts and security configurations isn't synchronized and your software settings and configurations aren't synchronized. So if you want any of that stuff to be the same on the backup, when the backup takes over, you've got to do that yourself, or you have to arrange for it to be done, which can it's are almost always can be automated pretty easily. It's not a big deal, but with house share, which is a complex application, usually involving multiple nodes, there's a lot of that kind of thing that you need to do that is required before you can start the mirror going. In other words, with a typical Iris data server, it's not the end of the world. Speaker 2 00:10:19 If you haven't, um, synchronized some information outside the mirror databases, uh, that maybe you should have, you can usually deal with that, how sharing the information is too complex and too distributed. So there's a lot of things that you must do when setting up a house share mirror that would be considered optional or, you know, things to consider when setting up a simple data server mirror. Uh, so it's really important for anyone using mirroring and house share to, you know, go through all that documentation and make sure all the I's are dotted and the T's are crossed. Speaker 0 00:10:55 Right? And you can thank Bob afterwards for, uh, for having that documentation be very helpful for you. Uh, once you go through this process, so kind of moving into some of the, why does this matter? We talked about it a little bit at the top about some of the importance you, you mentioned Amazon as an example, which is a really good one. If you have a couple others, like what are some common real-world use cases for mirroring databases that, that people might not think of, or that kind of is a good way to put it in perspective of why this is important? Speaker 2 00:11:20 So the first thing to do is I mentioned the fact that basically almost every business or organization lives online. Now, the other thing to remember is that the amounts of data are truly staggering. So number one, you need your applications to keep running. Number two, you need to keep up with the data. If you miss some, it can be catastrophic. Let's say you are a manufacturer and you've equipped your factory with the latest internet of things, technology to monitor all your, you can look at your, either returns from your, uh, machinery and know exactly what's going on. Well, you can't afford gaps in that data. If you try to analyze it, your analysis will be wrong. Cause you're missing data. You might not get the alerts you need saying, you know, such and such a equipment is about to break down unless certain maintenance is done, et cetera. Speaker 2 00:12:14 And when you're talking about even one factory in IOT stuff, you're talking about just a constant, huge stream of data. Now, if you think about a financial services business, a big one, a worldwide one. So around the clock, they have customers just to take one thing that happens with these companies, which is people's trade stocks, right? When you order a stock trade, you want it. Now, you don't want it 20 minutes from now when they managed to bring the system back online, because things will have changed, right? People demand instant action. And again, if you think about the millions of users that accompany may have, and in addition to user orders or requests, you've got constant querying going on. So you have, you have transactions, which is one kind of query and database query, and you have, uh, analysis, which is another. And meanwhile, you have the third kind, which is incoming data is streaming in from all sorts of sources. You've got to organize it and put it in the right places. So any of these get interrupted, uh, you've got a big problem. Exactly. So high availability is just, I don't think anybody in a modern enterprise will consider any technology that doesn't address that. Speaker 0 00:13:34 Yeah. It seems like a linchpin for really all modern apps that we really have today. Um, so, so kind of bringing it back a little bit, two more questions. Um, one is just, uh, you know, this, I think we kind of talked about this when you were talking about it, using it in Iris, but um, just kind of to reiterate it for the Iris users out there, or, you know, you can have a footnote about other technologies that use it as well. We talked about healthcare, but how easy is it, uh, in contrast to do this in InterSystems Iris, Speaker 2 00:14:00 It's, it's very easy in Iris. Um, although, you know, so much depends on it. It might be intimidating to think of what happens if you make a mistake, but really we've got it set up so that it pretty much, won't let you, if you're creating the databases, you're doing a new mirror and you're creating the databases then, uh, it's very simple. You configure the two instances as a mirror. You might have to, if they don't already have SSL, TLS configurations for data security, you have to create those that doesn't take a long time. And typically systems will in fact have that in place. Uh, you configure them as a mirror and then you add the databases on the primary, which is, you know, it takes a minute or two to add a database and then a systems Iris. And as soon as you do that, the mayor makes copies the backup and start synchronizing them. Speaker 2 00:14:52 So that is very easy. If you're adding existing databases to a mirror, it takes a bit longer because the journal files for a mirror database are different from those for a non-mirror database. So you can't just stick the database in there. Once you add it, it starts creating mirror journal files. So at that point, you've got to pause activity backup, or copy the database on the primary. Then restore that to the backup and add it to the mirror there. In other words, you're, you're copying, you're backing up or copying a mirror database and putting it on the backup and you add it to the mayor under the same name as it has on the primary. And again, we'll immediately start synchronizing, get caught up very quickly. Now, obviously there are related setup tasks you may need to do, and that's, that's true for any mirror, not just healthcare. Speaker 2 00:15:48 For instance, there are network considerations. We have a good amount of documentation about getting your network and your subnets set up the right way. There are different ways of redirecting those application connections after fail over. Uh, some of them are highly automatic, but of course the more automatic something is, you know, the more it requires getting set up in the first place is something called a virtual IP address, which basically means that the, that the IP address represents whichever of the fail-over members is the primary at the time that takes some configuration, but it's none of it's none of it's, um, you know, longterm or complex. Uh, another concern in any fail over technology is the risk that the backup will incorrectly take over while the primary is still operating, which would lead to two nodes, both trying to act as a primary, which of course is a disaster for the data mirroring. Speaker 2 00:16:43 However, uses a node called the arbiter to make sure this doesn't happen. The arbiter stays in constant contact with both the primary and the backup, and essentially gives the backup a go no-go decision when it wants to take over at the same time, if the primary loses touch with both the backup and the arbiter, it goes into automatically goes into a trouble state. The arbiter therefore knows that it can tell the backup to take over at that point without the risk of the primary coming back online. Uh, so the Alberta is, is pretty much takes care of that concern. Speaker 0 00:17:21 Right? Right. Cool. So lots of good stuff there on mirroring. And, uh, of course you can check out more details on this with the documentation that I'm sure Bob has written a lot of@docsdotinnersystemsdotcomaswellasouronlinelearningcontentatlearningdotintersystems.com. So, uh, Bob Binstock thank you so much for joining us and we'll see you next time. Thank you Derek. So thanks to Bob for breaking down everything about mirroring. There, there's a lot of information about mirroring available in the documentation, including a first look, exercise on data resiliency and mirroring. You can head over to docs.intersystems.com to search through that content and to learn more. That'll do it for episode five. Don't forget to rate and subscribe on whatever podcast app you use. See you all next time on data points. Speaker 3 00:18:04 <inaudible>.