Data Virtualizer

UPDATE: recorded videos now available

If you can’t make it in person to Oaktable World, check it out via live streaming !

here is the agenda

khailey Uncategorized

Want a free version of Delphix Express installed on your computer with the assistance of Oracle and Delphix experts? We will be holding #CloneAttack again at Oracle Open World on Oct 27 from 1:30-4:30.

CloneAttack is a hands on labs to install a fully operation Delphix Express environment on your laptop.

The labs will be hosted at the Creativity Museum, which is at the corner of Howard St and 4th St above the carrousel,  on Tuesday Oct 27  1:30pm to 4:30pm.  Dell  and DBvisit will also beholding #ToadAttack and #RepAttack in the same location with us.

Prerequisites

Standing at the corner of 4th and Howard, Oaktable World is in the Creativity Museum right behind the carousel and just down the sidewalk from the entrance to Oracle  Open World.

khailey Uncategorized

The following is an example of rolling a virtual database (VDB) back in time with Delphix Express, the free version of Delphix. Delphix by default keeps a 2 week window of changes for virtual databases (configurable to 2 months, 2 years etc). Thus we can go back in time to any point in that time window.

Why would we want to go back in time?

Maybe something went wrong. For example VDBs are typically used by developers who aren’t as versed in databases as a DBA.  What happens when a developer accidentally drops a table yet there was other work in the database that they want to keep. In the industry, half the time, developer databases are not backed up because they are “just development databases.” With Delphix those developer databases are always backed up and we can go back in time in a few minutes, so if a developer accidentally drops a table or updates the wrong column we can recover in minutes.

A common example of going back in time is QA who runs destructive tests and wants to rollback to the state of data before the destructive tests. For example say QA copy of production has to be first masked, then has to add “corner case” data then the destructive tests are run and after running destructive tests, we want to get back to just after the “corner case” data was added. Delphix will allow you to do this in minutes with just a few clicks of the mouse.

Here is the video

In the video, the first thing we see is adding corner case data in the web UI for the QA person. The web UI for the QA person is accessed in the web browser via the target machines IP with “:1080″ added on to the end to signify the port, as in http://172.16.160.160:1080

It looks like

where we can added new employees in the bottom left. QA will add corner case data like “-1″, null, “deleted” etc. After adding the corner case data we can snapshot the data state. To take the snapshot, click the camera on the left hand side (underlined in red)

Now QA can run the destructive tests.

When QA is finished, then click on the “Rewind VDB” button in the bottom middle:

You’ll get a dialogue box asking you to confirm the rollback:

And that’s it. In a couple of minutes the VDB will be refreshed to the last bookmark.

Summary

It’s as easy as as clicking a camera icon to snapshot a point in time. You don’t even need to take the snapshot to go back in time, but without the snapshot you need to remember what point in time you want to go. With the snapshot in place all it takes is hitting “rewind VDB” to go back to the most recent snapshot.  This is the admin interface. It’s even easier in the self service user interface  Jet Stream.

khailey Uncategorized

What if I’m a developer or QA person using a copy of production database to do my work. What if my copy of production is now out of date and I want to refresh my data with the data as it is on production.

Below is a video of how to refresh a virtual database to the freshest data from the source database:

This example uses the free version of Delphix called “Delphix Express” and  “Landshark” demo environment consisting of two VMs a source VM with a source database and a target VM with no database originally where we provision a virtual database.

The first thing to do in this setup of Express with Landshark is turn on changed data collection by Delphix from the source. The demo environment turns this off by default (though a normal install of Delphix this would be on) , so the first thing to do is turn it on.

To access the data collection settings, click on the data source. The data source in this case is “Employee XE DB”. Click on it:

Then we see

click on the arrow underlined in red above. This opens up a detailed card on the data source:

in the bottom left of the card is a button that says “disable”. Slide it to enabled. Now Delphix will start gathering data changes from the source. We are going to make data changes on the source and then refresh a virtual database on the target so it see the new data.

Steps

Enter some new data on the source. We can enter data using the web application. The web application is accessible through IP address of the source VM  followed by “:1080″ which designates the port the web application is running on as in http://172.16.160.160:1080

We can enter new employees in the bottom left. Say we enter “Larry Ellison”. So now the source data has data that is not in the currently running virtual database on the target machine.

Delphix will automatically capture these changes but is on the order of a minute or two behind the source changes. If you want to test this out quickly, as in the video above, you can take a snapshot of the data on the source which ensures that Delphix captures the data. To do this click the camera icon on the data source:

Now to refresh the virtual database on the target machine called “devdb”, click on the devdb line on the left of the Delphix console

Click on the up arrow (underlined in red above) which will pop out the detailed card on the virtual database:

In the bottom right of the card are a number of functions:

The functions are

1. Migrate virtual database to a different machine
2. Upgrade the Oracle version
3. Rollback
4. Refresh
5. Snapshot
6. Stop

We are going to use 4. Refresh which is the disk icon with V in the middle. When we click on this we will get a list of snapshots of source we can refresh to:

By default the most recent snapshot is active. You can click on other snapshots to “refresh” to older points in time. You can also click the slider on top of the snapshot cards to access a timeline where you can slide a pointer to the exact time you want to refresh to.  In our case we want the most recent data, which is selected by default, and then click “Refresh” in the bottom right.

Now if we go into the target we application we will see “Larry” show up:

The above web application is going through the target which is on 172.16.160.161 where as the web application and database were we added the data as one 172.16.160.160.

Summary

All the above steps were taken in the administration UI for Delphix Admins but users can refresh through the self service interface. See the last blog post before this for an example of a users refreshing an environment. In normal cases we don’t go through most of the above steps. Delphix by default automatically collects changes from the source and all we have to do is click on the virtual database of interest, and click on the “refresh” icon for the database and then click the “provision” button and then we are refreshed with data on the source database.

khailey Uncategorized

photo by Chase Elliott Clark

Here a video on using Jetstream which I like to call “DevOps Jetpack for Developers” (Remember you can get Delphix Express and Jet Stream for free.) See previous blog post on how to set up Jet Stream.

With Jet Stream, users can control their data. Users can refresh their data from the source, users can bookmark data to rollback to, users can bookmark their data and share those bookmarks with other users so that those uses can access the same data.

In the video

Part I

1. QA person adds edge case data to a copy of production data
2. QA person then bookmarks the edge case data and production data
3. QA person then runs destructive testing
4. QA person then restores to the bookmark of data just after adding edge case data so they are ready to run destructive tests again

Part II

1. QA person finds some data that causes a problem and wants to log a bug on it
2. QA person bookmarks data state with problem data
3. QA person shares bookmark with developer so the developer can reproduce it
4. Developer selects bookmark and activates that data set to investigate bug

Part I

To access the demo application provided with Delphix Express (Delphix Express optionally comes with a demo application is on the source VM and we provision a thin clone to the target VM)  go to http://IP_of_target:1080 after provisioning the application and database onto the target machine.

The application looks like:

On the bottom left we can add employees

On the bottom right we can remove employees

For edge case data, the QA person adds some employees like “-1″, “deleted” etc

After adding the edge case data, then we can snapshot the data in Jet Stream:

To snapshot the data state, click the point on the farthest right, then click the bookmark on the bottom left. Give the bookmark at name like “after the edge case data”.

Now the QA person can run destructive testing like removing all the Employees.

After the testing is over, we can go back to the bookmark by clicking on the bookmark, then clicking on “restore”

After the “restore” finishes, then QA is back to the original data, along with edgecase data and can run destructive tests again

Part II

Now imagine the QA person has added some edge case data that causes bugs. For example the QA person could add an employee called ”” (4 single quotes) which comes out ” (2 single quotes) in the list of employees. Now if I try to remove an employee with 2 single quotes, it doesn’t work. So I want to log a bug on this and bookmark not only the application but also the data.

All the QA user does after they have found the problem data is to create a bookmark in the Jet Stream interface and then click on “Share bookmark.”

Then the developer can go into the Jet Stream interface and make this bookmark data active for the developer.  The developer’s copy of the bookmark data is private to the developer meaning that the developer can modify the data without affecting the QA analyst and vice versa. This works because all duplicate data blocks are shared but anytime a data block is modified, that modified block is stored separately and is visible only to the copy that made the change.

Step 1

QA bookmarks the data and shares that bookmark.

The QA users clicks on the star on the far right, then on “Bookmark” on the bottom menu. After the bookmark is created, then the QA clicks on the bookmark and clicks “Share” on the bottom menu. Now other users will be able to see the bookmark.

1. Now the Developer can come in and select bookmarks in the bottom of the screen.
2. Then select the “Bug XXXX data” bookmark
3. Then click the branch icon on the bottom right of the screen

After the branch has taken place the “bug XXXX data” will be active for the developer.

What this means is that Delphix has shut down the database and application for the “master” branch that the developer was working on and now the data branch with data from “bug XXXX data” has been started and is active.

By default, Delphix only allows user to have one instance of each virtual database(s) and virtual data source(s) active in a container, thus when a user branches a container, Delphix shuts down the running virtual database(s) and virtual data sources and starts up the database(s) and/or virtual data sources(s).

If the developer clicks on “Branches” on the left side of the screen they will see their original “master” branch and the new “bug XXXX data” branch and the developer can switch between the two

If the developer clicks on the “default” branch, then on “Activate” they will be back to their regular data (the master branch or default branch  in this case)

Summary

In summary, Jetstream allows developers and QA analyst to take actions themselves and have results in minutes that would have otherwise required teams of peoples and days of work (DBAs, system admins, storage admins, storage, data copying, recover etc).

We gave  two examples here.

Example 1 was a QA person being able to save a data state and run destructive testing and being able to restore to the baseline data state in minutes.

Example 2 was a QA person being able to bookmark a data set the caused a bug and a developer who was able to spin up that application stack and data set in minutes so they could reproduce and investigate the bug.

Jet stream has many more applications.

As a performance person I like to do A/B testing of database. Jet Stream is awesome for that. Jet Stream allows me to create indexes in one copy, drop indexes in another copy, change init.ora parameters in another copy and to be able to switch between these data branches and run comparative performance tests in minutes.

Other examples are a developer who could have branches for

•  Latest full data set from production
•  Sub-setted data set from production, used for functional testing of new code
•  Data set for upcoming release, new schema supports new features
•   Data set imported from a QA users bookmark, used to fix an important bug

or a software architect who could have data branches for

•  Database for product A, release scheduled for next month
•  Database for product A, release scheduled 6 months from now
•  Database for product A, release scheduled for 1 yr from now
•  Database for product B, successor to product A
•  Empty database, brainstorming area for blue sky ideas
•  Hackathon project

or a business analyst who has data branches for

• Latest financial transaction data
• Financial transaction data from last week
• Financial transaction data from last quarter

Data as a Service functionality  brought to you by Jet Stream and Delphix is changing the data dynamics of companies and accelerating application development as well as financial data analysis.

khailey Uncategorized

Delphix has a developer self service interface called “Jetstream.”  I like to call it the “DevOps Jetpack” for developers.

Jetstream is available both in the Delphix enterprise engine and also in the free Delphix Express engine.

See following blog post on users using Jet Stream.

Here is a video on how to setup Jetstream for two users, a developer (Joe Dev)  and a QA analyst (Jane QA). In a blog post to follow this one, we will see how Joe Dev and Jane QA can work together thanks to Jetstream which allows them to share versions of data (similar to how people can share versions of code with source control).

steps

1. create users
2. create a template (defining what virtual data will be in a container)
3. create virtual data for the containers
4. create containers with virtual data following the template definition

by step 4 the users will be able to log in and have self service to refresh, rollback, bookmark, branch etc for all the data in the container. If there are multiple data sources defined in the container then when refreshing or booking marking, everything will be refreshed or bookmarked in sync.

Create Users

The first step is to create a couple users accounts for the developers and QA people using Jetstream.

To create users, go to the top left menus and choose “Manage” then choose “Users”

After choosing “Manage -> Users” you see the following screen. On the following screen click the “+ Add User” button in top left, then on the right fill out users name, email address, password and be sure to click the “JS Only User” box.

For these examples I added two users “JaneQA” and “JoeDev”.

Setting up Jetstream

Jetstream organizes data sources work into a “Template”. A Template defines which data sources will be used together in a container.  A container contains virtual data copies and belongs to a developer. All data copies in the container can be refreshed, bookmarked or otherwise operated on together and in sync.

Once a template had been defined, one can  instantiate containers for users.

Create a Template

A template defines what data sources will be allocated in containers given to users. User containers are  based on templates.

To create a template, go into the Jet Stream interface. To go into the Jet Stream interface, click on the “delphix_admin” in the top right, then choose “Jet Stream”

Now you are in the Jet Stream interface.

To create a template, click on “Add Template +” in the middle right.

Now name the template. I used “Dev_QA_Template”

Then click “Add Data Source +” in the bottom right.

For the first data source, choose the Employee XE database

The second data source to add is the Employee Web Application. Click “Add Data Source” and add the Employee Web Application.

Finally click “Create” and now you have a template for containers and can start creating containers for users.

Create Virtual Application Stack

Each user JaneQA and JoeDev will need virtual data copies of the XE database and the web app to put into the containers based on the above template.

We can provision a virtual web application and  virtual XE database for each user. There is a trick in this lab situation. We are using XE databases so there can only be one XE instance per host, thus we have to create a second host if we want a second copy of the database for the  second user.  To get a second host for this lab, you can just re-import the target OVA. See

• Setting up second target

For the developer we set up “devdb” and “devapp” on one target and for the QA person we set up “qadb” and “qaapp” on the other host.

Above is what the admin screen looks like after create the development app stack copy and the development database copy. We also need to create another copy of the app and db for Jane QA.

Create Containers

Now that we have the template and virtual data copies to supply, we can make containers for uses

When you come in to Jet Stream, you will see the template we just created. Click on the name of the template.

After clicking on our template, you can create containers for that template clicking “Add Container” on the top right.

To create the container, give it a name and make sure that the owner is the correct user, in our case it will be JoeDev (then we will create one for JaneQA)

Choose the virtualized data sources to correspond to the definition of  the template.

Users

When users log in, for example Joedev, the screen looks like

The middle of the screen shows that the data used by this user has been branched off from the data source at 10/12/2015 11:12:22am.

The red star is the current state of the the virtual data.  We can click on any point in the time line to bookmarket or restore to.

• Activate – start the database instance(s) and any application instances in the container
• Bookmark – mark this spot to go back to or share with another uer
• Branch – make a new read/write instantiation while still being able to go back to the old version
• Share – make a bookmark shareable with other users
• Refresh – refresh data from the current data sources state
• Restore – restore data to a previous point in time
• Reset – reset data to most recent bookmark
• Stop – stop the database instance(s) and any application instances in the container

Some cool things:

If I was a performance person I could create multiple branches of the data perhaps with different index configurations and run the same performance tests across the different branches and compare. To go from branch to branch requires choosing the branch and activating it, which will deactivate  any current activated branch.

If I was a QA person I could bookmark not only the application version with a bug but also the data state that has the bug because bugs are going to need example data to reproduce the bug and sometimes bugs are directly related to specific data data states.

We will look at how to use these features in future blog posts.

khailey Uncategorized

What kinds of things keep a IT ops person up at night?

What kinds of things make IT Ops people have to work over the weekends?

What keeps me up:

• outages
• performance problems
• running out of disk space
• corruption requiring recovery

These things are stressful.

One the most stressful things is recovering from corruption. Usually when there is a corruption, some important data and/or system is not accessible and won’t be until I recover the data. Now data recovery is not something I typically do everyday, so when I have to so it under pressure and using backups that probably haven’t even been tested for viability, then it’s a hugely stressful situation.

How can I make my life easier? How can I address all these bullet points to reduce the stress level and make life easier?

What I can do is use Delphix. Now you may say “what does Delphix, the development acceleration guys, have to do with IT operations?”

Delphix is known for being able to spin up development environments along with full copies of large production databases in minutes for almost no extra storage per copy and on top of that mask the data for usage in development.

What’s this technology have to do with making IT life easier?

First and one of the greatest stress relievers is using Delphix for backup and recovery. For example backup and recovery of Oracle can be expensive and time consuming and complicated. Backing up Oracle is complex enough that it requires it’s own tool, RMAN. RMAN syntax can be challenging and complex. Usually the back up itself is automated and put into motion to run on it’s own. Problem with this is that we often don’t know if the backups actually work and when it comes to recovering, we have to remember where all the files are and all the correct syntax for recovery. Recovery often requires copying the data from a backup location  to a location where it can actually be used. The backup strategy proposed by Oracle is to take a full backup on the weekends and incremental backups during the week, which means if the database has a problem on Friday, 5 days after the last full backup, then it will require taking the full backup and replying multiple incremental to that full backup as well as any archive required and online redo. If anything goes wrong it might be hours before we find out. If we do anything wrong it might mean hours of work to redo everything.

This is why I love Delphix.

With Delphix, all I have to do click a button in a web UI, and Delphix will start up a copy of the recover database without having to move any data around. Delphix uses the backup data in place for the recovery. Delphix takes care of all the recovery commands behind the scenes, starting up the database down to the second or SCN that I request from a graphical slider. It only take a few minutes. It’s so easy. It just eliminates the stress. I don’t have to think about how to do something, I just have to think about what I want. I’d rather think about the business goals than about RMAN commands. 

And using Delphix radically reduces the amount of disk space I have to use for backup. Let’s look at an example of a 9 TB database with 1TB of changes per day and compare the standard Oracle recommended backup strategy of full back on the weekend and incremental during the week vs using Delphix:

The Oracle strategy has to save 9 TBs every weekend and 1 TB a day.

The Delphix solution links to the 9 TB once and only once and using compression saves this in about 3TB. Then and forever Delphix never takes another full backup. Delphix just collects the changes and the 1 TB of changes a day is stored in about 300 GB, so the Delphix storage foot print never even reaches the size of the original database for almost a month, despite the fact that we can recover the database down to the second in minutes for almost no extra storage anywhere in that 30 days.

Delphix also has the option of automatically recovering the database, say every 5 minutes to prove that the backups are correct. With this automation we have caught Oracle redo log bugs that would have otherwise gone unnoticed until it was too late.

If you have a 25GB or less Oracle database (or MySQL, Postgres, Sybase,  or SQL Server)  you can set up this backup architecture with Delphix for free!

Delphix makes recovery effortless, bullet proof,  fast and  inexpensive.

khailey Uncategorized

I have previously announced that Delphix now has a free version available called Delphix Express as well as how to install Delphix Express.

In this blog will look at an example of virtualizing your first database and not only the database but virtualizing the application as well.

What does virtualizing mean? Like in the world of virtual machines where we make multiple virtual machines on one set of hardware, in the world of virtual data we make many read/write copies of data with one actual physical copy of the data. We do that by sharing unmodified data and storing each version of a data block that gets modified and keeping that modified version of the block only visible to the clone that made the modification.

Here is the demo video:

Delphix does 3 things

1. collect data from source (usually a “production” database)
2. manage virtual data time flow on storage allocated to Delphix
3. provision virtual copies of data to target machines, typically developer and QA environments. To provision data to target machines, Delphix mounts thin clone copy of the database files onto the target machines using NFS and then Delphix starts up and recovers the database to the point in time requested.

Here is what the Delphix Express web UI screen looks like after it has automatically linked with the source VM and the target VM from the “landshark” demo environment.

On the top left we have a set of folders created automatically by the landshark demo configuration. The folders are used for organizational purposes as well as a way to limit user visibility to what environments are on Delphix.

In the “Sources” folder there are two data sources linked into Delphix. One is a web application “Employee Web Application”, the other which has been selected is the “Employee XE DB”.

In bottom right is a provision button. When we click this button we can provision the selected source onto a target machine.

We select the target machine on the first dialogue popup window:

Notice that the “LINUXSOURCE” is selected on the left hand side of the dialogue. We get an error because of this because we’ve disallowed provisioning back to the source. The source is where the data comes from, and Delphix is capable of provisioning a thin clone back to the  source host, but generally that is not what we want so we have chosen an option for the LINUXSOURCE not to allow provisioning.

We click on “LINUXTARGET” instead.

We also want to change the “Database Unique Name” to “devdb” because we are also going to provision a thin clone of the web application and in the web application we are going to specify the database to connect to as “devdb”.

In the next dialogue I’m going to chose the “Dev Copies” folder to put the virtual database into.

The next dialogue is to add any pre or post provisioning scripts and I’ll just skip this and hit “Next”

The final popup window is a review of the choices we’ve made, and to finish we just click “Finish”

In the “Dev Copies” folder on the left we will see the new virtual database appear and a status as to how complete the job is:

While the virtual database is provisioning we can provision a copy of the web application as well. I’ll click on the web application source on the left and then click “Provision” on the bottom right:

The two important steps here are to note the “Webserver  Port” and to make sure the “Database Service Name” is the same as the virtual database we provisioned:

Like the virtual database, I’ll put the virtual web application in the “Dev Copies” folder

and I’ll just hit “Next” on the pre and post provisioning page:

Finally, I’ll see the review of choices and click “Finished”

Now it takes a couple of minutes for the database and web application to provision to the target.
If we try to access the web application, but typing the IP address of the target machine colon the web application port number, and we do this before the web application finishes provisioning, then we will get an error:

If we access the web application after it finishes provisioning but before the database finishes provisioning (the database will take longer than the web app) we will get an error from the web application trying to connect to the database:

When the virtual database provisioning finishes then we see the working web application:

The Delphix Express UI with the new virtual database and virtual web application looks like:

In the following blog posts we will see

• Refreshing a virtual database from the source
• Rolling a virtual database back to a save point
• Branching a virtual database to recover lost data
• Installing Oracle 11g EE on Express
• Setting up second target
• Setting up jetstream
• Using jetstream

khailey Uncategorized