Customer Reviews

Refactoring Databases: Evolutionary Database Design

5 star:		(15)
4 star:		(4)
3 star:		(3)
2 star:		(0)
1 star:		(1)

 

 

This is an excellent book that, in my opinion, serves two purposes. First, it is a compendium of well thought-out ways to evolve a database design. Each refactoring includes descriptions of why you might make this change, tradeoffs to consider before making it, how to update the schema, how to migrate the data, and how applications that access the data will need to change. Some of the refactorings are simple ones that even the most change-resistant DBAs will have used in the past ("Add index"). Most others (such as "Merge tables" or "Replace LOB with Table") are ones many conventional thinking DBAs avoid, even to the detriment of the applications their databases support.

This brings me to the second purpose of this book. Many DBAs view their jobs as protectors of the data. While that is admirable, they sometimes forget that they are part of a software development team whose job is to provide value to the organization through the development of new (and enhancement of existing) applications. One of the best DBAs I ever worked with viewed himself as a "Data Valet." He said his job was to make sure the data was presented to applications when and where they wanted and to protect the doors from getting dinged while under his care. Through its first five chapters and then the refactorings that follow, this book will help DBAs expand their view of their role in the organization from one of simply protecting data to one of enhancing the value of data to the organization.

This book is one that you'll keep on your reference shelf for many years to come. Highly recommended.

In software development a 'refactoring' is a change that improves code quality without changing functionality. Refactoring helps keep an application maintainable over its life-cycle as requirements evolve, and is particularly of interest to those adopting modern 'agile' methodologies. This book comprises five general chapters on database refactoring - about 70 pages - followed by a 200 page catalog of various refactorings. The refactorings are classified as 'structural', 'data quality', 'referential integrity', 'architectural' and 'methods'. An additional chapter catalogs 'transformations', more on which in a moment. Each catalog entry uses a template including 'Motivation', 'Tradeoffs', 'Schema Update Mechanics', 'Data-Migration Mechanics' and 'Access Program Update Mechanics'. The 'Mechanics' sections include example code fragments for Oracle, JDBC and Hibernate.

Several of the structural refactorings are just simple database schema changes: rename/drop column/table/view. Adding is not really a refactoring so add column/table/view were cataloged as 'transformations' - changes that do affect the application, a distinction that appears to me a little clumsy. Some structural refactorings are more interesting: merge/split columns/tables, move column, introduce/remove surrogate key, introduce calculated column, introduce associative table.

The data quality refactorings include introduce/drop default values, null or check constraints, standardize codes, formats and data types, use consistent keys and lookup tables. Most of these are common best practices, seeing them cataloged as refactorings didn't yield me any new insights. Only replacing type codes with flags was of special interest.

Referential integrity refactorings include the obvious add/drop foreign keys with optional cascading delete, but also using triggers to create a change history and hard vs. soft deletes. Using before and after delete triggers to implement soft deletes is probably the best example in the book.

Architectural refactorings include using CRUD methods (ie. stored procedures or functions to select/insert/update/delete records), query functions that return cursor refs, interchanging methods and views, implementing methods in stored procedures, using materialized views and using local mirror tables vs. remote 'official' data sources. All these are common design techniques and the discussion of motivation and tradeoffs is particularly relevant.

The final section on method refactorings is more abbreviated and covers typical code refactorings. These qualify for inclusion only because databases include stored procedures, but they have nothing to do with schema evolution.

An important aspect of this book is that the catalog of refactorings is presented in the context of evolutionary database development described in the first five chapters: this approach emphasises an iterative approach, automated regression testing, configuration control of schema objects and easy availability of personalized application database environments for developers. Refactorings and transformations are intended to be applied one by one, and an automated regression test suite used to maintain confidence that a change does not introduce an application defect. Change control and a change tracking mechanism are essential to manage the application of schema changes to integration, QA and production environments.

What do I like about this book? The catalog of refactorings is thorough (some might say pedantic) which makes it a good learning tool for new database developers and DBAs, and as a shared reference for communicating on larger projects and in larger organizations. Experienced DBAs working on smaller projects are less likely to find it useful.

What don't I like? Relatively little is provided about the tools required to make regular refactoring practical, the authors simply state that these are being worked on. utPLSQL is not mentioned at all. The discussion on tracking changes is thin (but check out the LiquiBase project on Sourceforge). No guidance is provided on how you might use Ant to build and maintain developer database environments. Little is covered on the tough topic of building and maintaining test data sets. A final pet peeve: no discussion of refactoring across multiple schemas shared by an application suite.

In summary this book sketches out some important ideas but much work remains to be done. The catalog takes a number of established techniques and best practices and places them in a new framework which at least provides value to some for now.

This book is a much needed exploration on the subject. It tries to categorize those operations that developers and DBAs do on a database, who, for various reasons, must address a specific problem, need. I only gave it three stars because it is somewhat insufficient. It also doesn't make much of a distinction between a database in development and one in production. In the latter case, it is really difficult to make changes when there is already a data structure in place with data, being updated constantly by users, and plans to migrate to a different data model while a system is in production is really not for the faint of heart. What I am really loooking for is a thick book of bad designs that, for various reasons (unclear or evolving requirements, political), a database model presents itself with a bunch of problems, real problems and not just theoretical, and the ways a DBA and developers came about after a big pow-wow on how to solve it.

It's been almost 7 years since Fowler's Refactoring book, and now the database community has finally caught up with the rest of us. This book shows how to refactor a relational database schema, working you through the detailed process steps for doing so and providing the source code for implementing more database refactorings than I would have thought existed.

The first five chapters describe how to go about database refactoring. Chapter 1 overviews the idea that you can evolve your database schema in small steps, a radical departure for many traditional DBAs. It also overviews the need for supporting techniques such as agile data modeling, database regression testing, and version control of your data models and scripts. I would have liked to see more coverage of these topics, but at least the modeling material is covered in Ambler's Agile Modeling book and there are some great SCM books out there.

Chapters 2 and 3 walk through the process of implementing a database refactoring, first through the simple situation where there is only a handful of applications accessing the database. I guess this sort of thing happens in smaller companies, but most of the time you really have to worry about scores of applications accessing your database which is a much harder situation. This is actually the focus of Chapter 3 and of the presented solutions in Chapters 6 through 11 which provide reference implementations for all of the database refactorings. This approach belies the true strength of the book: it reflects actual experience in large organizations, not just the theoretical pie in the sky stuff you see from other authors.

Chapter 4 focuses on deploying database refactorings in production, providing detailed instructions for how to roll refactorings between various sandboxes. It importantly describes how to merge the refactorings of several teams together. If you have 100 applications accessing a shared database, then potentially you need to manage the refactorings coming from 100 different development teams. Of course it would never be that bad, but even merging refactorings from 10 teams would be tough. This might be where the technique falls apart because many companies likely don't have data managers who are skilled enough to do this sort of thing efficiently enough to keep up with agile developers. We need new tools, so hopefully companies like Oracle will build something for us.

Chapter 5 describes a collection of best practices and challenges pertaining to refactoring databases. The authors share their experiences as well as identify potential issues, such as a lack of tooling and agile expertise within the data community, that could slow adoption of this technique. My guess is that the smarter teams within companies will start doing this stuff right away, for the most part it's pretty easy technically, but that bigger companies will struggle to change as they always do.

Chapters 6+ are reference descriptions for the individual refactorings. Each one is described using a UML data model, which is a little strange at first although once you get used to it you can see how it's a much better notation than Crow's feet, a detailed text description and source code. The source code examples are detailed, I guess the authors want to be thorough and provide a complete solution so that there's no question how to implement each refactoring. The application examples are written in Java or Hibernate, but they're simple enough that you could see how to implement them in C#, C++, Ruby, or even VB. The database code is Oracle, once again it's pretty straightforward so you can easily see how it would work in other DBs like Sybase or MySQL.

All in all, if you're a DBA or agile programmer you need to seriously think about buying this book.

I've done a lot of refactoring and restructuring programs and always had to deal with the problems involved with changing the layout of persistent storage. This book describes solutions for those issues I encountered. It explores some tough questions, like how to refactor a database used by multiple applications, only one of which is currently undergoing modification. I recommend it to anyone involved with the database side of systems, whether an administrator or a programmer who uses one.

Scott Ambler and Pramod Sadalage wrote Refactoring Databases, they say, "to share their experiences and techniques at evolving database schemas via refactoring". The book, particularly in the thorough list of refactorings detailed in later chapters, reveals them to be experienced users of, and writers about, agile development approaches. Their core premise is that data and databases must evolve in the same way as code does - that is incrementally.

They argue persuasively that a big-bang, up-front approach to database design is unacceptable in a modern environment. There is simply too much change and too much uncertainty in the business world for this to be realistic. The basic techniques for evolutionary database design include refactoring (the topic of the book), evolving the data model, database regression testing and configuration management and developer sandboxes. This more evolutionary approach is going to be a big shock for many data professionals, something the authors note, but I think the need for effective evolution and ongoing development of applications and thus their databases is compelling. "Change time", the period after an application (or database) is first deployed is bar far the majority of the life of an application. Techniques that help you cope with this, like database refactoring, are a good thing. Database refactoring as described in the book, is part of an evolutionary approach and with development teams taking on SCRUM, XP and other agile methods it is more important than ever for database teams to do likewise. Many data professionals will likely have the same knee-jerk reaction I did when first approaching this - Why not just get it right up front? But if you believe that agile model-driven development is here to stay for code then you have to accept the need for the same approach to database design.

Martin Fowler's original book Refactoring: Improving the Design of Existing Code made the point that a refactoring must retain the behavioral semantics of the code and this is just as true in databases. The authors take great pains to explain refactoring in enough detail that it you can apply it constantly to keep the database as clean and easy to modify as possible. They emphasize repeatedly the value of test-driven or test first development - even in database design and deployment. The authors stress the importance of testing, especially regression testing, of all the components that access a database when refactoring it. They advise making refactoring efforts small as well as test-driven. They point out that refactoring should be done as a series of small steps and that database develops must not succumb to the temptation to combine refactorings into larger, more complex efforts. The need to treat database designs, and even data, as artifacts subject to change control comes through loud and clear.

The concept of a potentially very long transition period in which both the old and new schemas are supported is a particularly good one. I worry about the organizational dynamics of having the old schema removed by a new team that does not remember the original refactoring but nothing else seems rational in a typical environment where many applications run against the same database. I also liked the paranoia of the authors, for instance in their suggestion to always run regression tests BEFORE refactoring to make sure the database is actually as you think it is! While the book focused on refactoring, many of the hints and suggestions would be good for implementing real changes in business policy.

The book is a surprisingly easy read for such a potentially dense subject. The book starts by describing the fundamentals of evolutionary database development and the basics of refactoring. A process overview, deployment notes and some best practices follow. These initial chapters, designed to be read in sequence, introduce and explain the topic well and have a nice little "What you have learned section" at the end. There were many worthwhile asides in the book as it covers these topics and after these introductory chapters, the book then goes (somewhat abruptly) into a series of chapters on various kinds of refactoring - structural, data quality, referential integrity, architectural, method and transformations. These chapters take a series of very specific refactorings. The potential motivation, tradeoffs and implementation mechanics are defined for each. The refactorings are self-contained and, while this makes reading them as a set somewhat repetitive, it should make the book a much better reference guide for actual users.

The book did not really touch on how you should consider data and database designs used in analytic models or the potential value of using business rules, but these are minor quibbles. The book is well written, full of great examples and gives lots of good advice.

Much more beautiful code is being written as a result of Fowler's Refactoring; the Gang of Four's Patterns; Beck's Test Driven Development; etc. Unfortunately the database development community either hasn't applied these principles & practices or hasn't bothered to read about them.

So Scott and Pramod have brought the mountain to Mohammed with Refactoring Databases. Most database developers I interact with are loathe to change a database design. It's scary and risky! As a result most database applications continue to incur increasing technical debt, entropy, and ugliness. In my work as an Agile Coach and Agile Data Warehouse developer I hardly ever get to see a truly beautiful data model in production. This book, when applied, will go a long way toward helping database developers create beauty and avoid the scariness of changing data models.

This book is a must read for database developers, and I reserve "must read" status for very few books. Next thing we need is a database patterns book.

Wow! In a word....Wow!

Ambler and Sadalage have brought to the database world a truly important and sorely needed concept. Yes, data is critical to all applications. An application with process only is an application that does a lot, and accomplishes nothing.

The world of software development is finally starting getting smart with code refactoring. It is about time that we are starting a similar discussion regarding data refactoring as well. To think that our data models would start and stay perfect, while our code and our models would change over time is wrong.

Ambler and Sadalage recognize that and offer practical refactorings and transformations (including SQL!), to help any team, large or small, get back on track. This seminal book will be used for years to come in our industry. It is concise, complete, and undoubtedly field tested. Isn't this all we could hope for in our own systems under development....? These men are not spouting ideas from the tops of mountains, ideas conceived in ivory towers. They have "been there...done that" and we need to listen....now.

I truly look forward to a future book from Scott Ambler. I know that he will certainly have a look at Process Refactoring before long. Have any of us worked on a project where the initial process defined was the final one? Didn't think so ;-)

Support for database refactoring has been needed for some time. All projects that use a database, whether Agile or not, wind up with the need to improve the structure of the database. This book describes about 70 refactorings, telling you how to do it, what the pros and cons are, and even how to transition the changes over time, to give your applications time to convert. Good stuff!

I've put a longer review on the XProgramming web site, but the bottom line is that if your database is getting crufty, this book can help.

This is in response to the review concerned about the minimal coverage of database testing. When Pramod and I wrote the book we decided to focus just on database refactoring itself, which I hope you'll agree that we did. In Chapter 1 we do in fact point out that database regression testing is an important enabler of database refactoring, as is configuration management, agile data modeling, and developer sandboxes. We didn't invest much time on those subjects because we wanted to remain on topic.

We have gone into a fair bit of detail with the individual refactorings. In particular, we provide the source code required to implement each one and show Java examples of how application source would potentially change as the result of the database being refactored. Each refactoring is described as a stand-alone reference, the end result being that if you were to read the book from cover to cover you would see several common themes, such as migrating data and deprecating the original schema, repeated throughout. But, who is going to read reference material cover to cover?

As you know, there are several good books written about configuration management and therefore covering that in detail didn't make much sense to us. The topic of developer sandboxes, although important, likely only rates an article or two. Nobody has written a book specifically about Agile Data Modeling, although with a little bit of searching here on Amazon I have no doubt that you can find a good book or two about Agile Modeling ;-). I definitely agree that a book is needed about database regression testing and I'm thinking seriously about writing one myself.

If you are interested in those topics, I highly suggest that you visit the Agile Data site where I've written extensively about them already. Otherwise, I hope that people find Refactoring Databases to be a comprehensive discussion of this new, leading edge technique.