Index of Related Posts
This is part 2 in a series: Part I, Part II, Part III, Part IV, Part V.
Update: Oracle Acknowledges Software Defect
During the development of this study, Oracle’s Product Manager in charge of the In-Memory feature has cited Bug #19308780 as it relates to my findings. I need to point out, however, that it wasn’t until this blog installment that the defective functionality was acknowledged as a bug. Further, the bug being cited is not visible to customers so there is no closure. How can one have closure without knowing what, specifically, is acknowledged as defective? Also read more at The Register: Click here.
Preface
In this post you’ll see that I provide an scenario of accidental paid-feature “use.” The key elements of the scenario are: 1) I enabled the feature (by “accident”) but 2) I didn’t actually use the feature because I neither created nor altered any tables.
In Part I of this series I aimed to bring to people’s attention what I see as a significant variation from the norm when it comes to Oracle licensed-option usage triggers and how to prevent them from being triggered. Oracle Database Enterprise Edition supports several separately licensed options such as Real Application Clusters, Partitioning, and so on. A feature like Real Application Clusters is very expensive but if “accidental usage” of this feature is a worry on an administrator’s mind there is a simple remedy: unlink it. If the bits aren’t in the executable you’re safe. Is that a convoluted procedure? No. An administrator simply executes make -d ins_rdbms.mk rac_off and then relinks the Oracle executable. Done.
What about other separately licensed options like Partitioning? As I learned from Paul Bullen, once can use the Oracle-supplied chopt command to remove any chance of using Partitioning if, in fact, one does not want to use Partitioning. I thought chopt might be the solution to the issue of possible, accidental usage of the In-Memory Column Store feature/option. However, I found that chopt, as of this point, does not offer the ability to neutralize the feature as per the following screenshot.
Trivial Pursuit of the Ignoramus or Mountainous Mole Hill?
There is yet no way I know of to prevent accidental use of the In-Memory Column Store feature/option. Am I just making a mountain out of a mole hill? I’ll let you be the judge. And if you side with folks that do feel this is a mountainous-mole hill you’d be in really good company.
Lest folks think that we Oaktable Network Members are a blind, mutual admiration society, allow me to share the rather sizzling feedback I got for raising awareness to this aspect of Oracle Database 12c:
Geez!
No, I didn’t just want to dismiss this feedback. Instead I pushed the belt-sander off of my face and read the words a couple of times. The author of this email asserted I’m conveying misinformation ( aka “BS”) and to fortify that position it was pointed out that one must:
- Set a database (instance initialization) parameter.
- Bounce the instance.
- Alter any object to use the feature. I’ll interpret that as a DDL action (e.g., ALTER TABLE, CREATE TABLE).
Even before I read this email I knew these assertions were false. We all make mistakes–this I know! I should point out that unlike every release of Oracle from 5.1.17 to 11gR2 I was not invited to participate in the Beta for this feature. I think a lot of Oaktable Network members were in the program–perhaps even the author of the above email snippet–but I don’t know that for certain. Had I encountered this during a Beta test I would have raised it to the Beta manager as an issue and maybe, just maybe, the feature behavior might have changed before first customer ship. Why am I blabbering on about the Beta program? Well, given the fact that even Oaktable Network members with pre-release experience with this feature evidently do not know what I’m about to show in the remainder of this post.
What Is An Accident?
Better yet, what is an accident and how full of “BS” must one be to fall prey? Maybe the remainder of the post will answer that rhetorical question. Whether or not it does, in fact, answer the question I’ll be done with this blog series and move on to the exciting work of performance characterization of this new, incredibly important feature.
Anatomy of a “Stupid Accident.”
Consider a scenario. Let’s say a DBA likes to use the CREATE DATABASE statement to create a database. Imagine that! Let’s pretend for a moment that DBAs can be very busy and operate in chaotic conditions. In the fog of this chaos, a DBA could, conceivably, pull the wrong database instance initialization file (e.g., init.ora or SPFILE) and use it when creating a database. Let’s pretend for a moment I was that busy, overworked DBA and I’ll show you what happens in the following:
- I executed sqlplus from the bash command prompt.
- I directed sqlplus to execute a SQL script called cr_db.sql. Many will recognize this as the simple little create script I supply with SLOB.
- The cr_db.sql script uses a local initialization parameter file called create.ora
- sqlplus finished creating the database. NOTE: this procedure does not create even a single user table.
- After the database was created I connected to the instance and forced the feature usage tracking views to be updated (thanks to Morgan’s Library for that know-how as well…remember, I’m a database platform engineer not a DBA so I learn all the time in that space).
- I executed a SQL script to report feature usage of only those features that match a predicate such as “In-%’
This screen shot shows that the list of three asserted must-happen steps (provided me by a fellow Oaktable Network member) were not, in fact, the required recipe of doom. The output of the features.sql script proves that I didn’t need to create even a single a user table to trigger the feature.
The following screen shot shows what the cr_db.sql script does:
The following screenshot shows the scripts I used to update the feature usage tracking views and to report against same:
The “Solution” To The “Puzzle”
Stepping on a landmine doesn’t just happen. You have to sort of be on your feet and walking around for that to happen. In the same vein, triggering usage of the separately licensed Oracle Database 12c Release 12.1.0.2 In-Memory Column Store feature/option required me to be “on my feet and walking around” the landmine–as it were. Did I have to jump through hoops and be a raging, bumbling idiot to accidentally trigger usage of this feature? No. Or, indeed, did I issue a single CREATE TABLE or ALTER TABLE DDL statement? No. What was my transgression? I simply grabbed the wrong database initialization parameter file from my repository–in the age old I’m-only-human sort of way these things can happen.
To err to such a degree would certainly not be human, would it?
The following screenshot shows the parameter file I used to prove:
- You do not need to alter parameters and bounce an instance to trigger this feature usage in spite of BS-asserting feedback from experts.
- You don’t even have to create a single application table to trigger this feature usage.
Summary
This blog thread has made me a feel a little like David Litchfield must have surely felt for challenging the Oracle9i-era claims of how Oracle Database was impenetrable by database security breaches. We all know how erroneous those claims where. Unbreakable, can’t break it, can’t break in?
Folks, I know we all have our different reasons to be fans of Oracle technology–and, indeed, I am a fan. However, I’m not convinced that unconditional love of a supposed omnipotent and omniscient god-like idol are all that healthy for the IT ecosystem. So, for that reason alone I have presented these findings. I hope it makes at least a couple of DBAs aware of how this licensed feature differs from other high-dollar features like Real Application Clusters in exactly what it takes to “use” the feature–and, moreover, how to prevent stepping on a landmine as it were.
…and now, I’m done with this series.
Kevin Closson's Blog: Platforms, Databases and Storage 
Superb; this part of the series speaks for itself.
So, in summary, the new parameters were set and the database was started at that point (in being created it was started.) I have not tested whether dynamically setting an in- parameter would go into effect without a fresh instance start. Based on the source of that claim I’d suspect it to be true.
So everyone agrees the feature arrives enabled. It seems a memory parameter that did not exist before this patch does need to be set and *probably* an instance start needs to occur before it goes into effect. But once that happens apparently actual creation of a table using the feature is not needed.
While I’ve always been of the opinion that priced options should default to off, except in the case of an Oracle vended application product where during its evolution Oracle has chosen to use a feature (in which case the feature would be on, but only require a license if the customer make a custom use of the feature aside from the Oracle Application), it does appear that you need to set a parameter and start the instance for it to go into effect.
Two good friends have each appeared to have made a small error in this thread (I hope I got the summary right.) The BS call was, however, made in a private forum. With 20-20 hindsight I’d suggest that resolving that should also have been private.
Nevertheless, it appears far too easy to trigger a license event even though a particular parameter does need to be set at the start time of the instance. (Presumably prior to the patch this was an invalid parameter that was ignored or a legal parameter that had no effect.) For a running database that was patched in a way not requiring a bounce itself, that would probably would require a bounce, since the instance would not have honored the memory allocation requested prior to the patch.
I’m still a bit confused about why a parameter file with new parameter values was laying around in a repository. I hope I’m correct that
inmemory_size=1G
was not inserted into an existing parameter file by Oracle’s patch. That would really be out of bounds.
mwf
inmemory_size=1G was set in the parameter file to make the scenario: I accidentally grabbed the wrong init.ora, created a database, did not create a single table and yet I “used” the feature–a feature specifically related to using tables, no?
yes. New parameter required to trigger needing to license the feature, actual creation of a table not. This indeed seems like too easy an invocation – but without the new parameter allocating the memory the feature being “on” (meaning available if you also allocate memory for it) it does not cause a license event. Still, this is nasty, because you can’t even allocate the memory to access remaining machine capacity (avoiding actual use of the feature by NOT creating an inmemory object) without triggering the license event. I’m still thinking that the arrival of the patch, even with a repository of init files, requires the addition of that parameter to one of the init files in the repository. So while I agree in general that you should have to set “I want to use feature X” in an init file to have to license it, both of these parameters would appear in an init file intended to actually use the feature, so just grabbing the wrong init would entail building such an init. Already operational DBAs are pretty damn careful about what init files they use, but your warning should productively add extra caution to their behavior. Seems like a loaded gun left around carelessly to me. It seems hard to imagine that Oracle actually wants to charge for features that are not used, because it will be more trouble than it is worth. Like adding a second tenant to a container database, this one seems just a little too easy for maintaining good business relations. The whole feature usage business should keep at least a daily history, anyway. You should be able to list the options you need a license for, which you have a license for, and whether they are recorded as used. There should be a parameter “disallow unlicensed feature use” that checks the license list and tosses appropriate errors. In the current case, if someone used in-on and in-memory allocated, then on startup that should cause an invalid parameter if “disallow unlicensed feature use” is on and no license record exists for in-memory. A few of the IT directors of MOSES and VLDB railed against this in the early 1990’s, so I’m saying nothing new.
It didn’t take creating a table to show the usage and it should have.
It did require an instance start with particular parameters set.
I don’t think I missed the point, I think I mostly agree with you.
It’s not quite a trip wire, but it is a loaded gun once you have those two inits in a file in a repository.
Maybe these folks can push some of these issues: http://www.clearlicensing.org/
Kevin,
Thanks for the tweeted invites to hop on the comment thread. Great blog entries. I learned from both your posts as well as the comments. Team members alerted me to your shout-out. In my mind, the issue is significant enough for me to make a timely post from the middle of a two week vacation.
In Q1 2004 DB 10g arrived with AWR and other new related, attractive engineering. A new 26 page canned document appeared: “Oracle Database Licensing Information 10g Release 1 (10.1) Part No. B13552-01.” 12 pages of the doc were content. The doc was interesting in what it asserted.
” ■ Some Diagnostics Pack features are accessed by way of database server APIs and
command-line interfaces:
– The DBMS_WORKLOAD_REPOSITORY package is part of this pack.
– The DBMS_ADVISOR package is part of this pack if you specify ADDM as the value of the advisor_name parameter, or if you specify for the value of the task_name parameter any value starting with the ADDM prefix.
– The V$ACTIVE_SESSION_HISTORY dynamic performance view is part of this pack.
– All data dictionary views beginning with the prefix DBA_HIST_ are part of this pack, along with their underlying tables.
– All data dictionary views with the prefix DBA_ADVISOR_ are part of this pack if queries to these views return rows with the value ’ADDM’ in the ADVISOR_ NAME column or a value of ’ADDM*’ in the TASK_NAME column or the corresponding TASK_ID.
– The following reports found in the /rdbms/admin/ directory of the Oracle home directory are part of this pack: awrrpt.sql, awrrpti.sql, addmrtp.sql, addmrpti.sql, awrrpt.sql, awrrpti.sql, addmrpt.sql, addmrpti.sql.
(p. 2-5)
A commenter on your second post wrote: “It seems hard to imagine that Oracle actually wants to charge for features that are not used, because it will be more trouble than it is worth. Like adding a second tenant to a container database, this one seems just a little too easy for maintaining good business relations.” Unfortunately in our experience, Oracle would and indeed has been known to leverage contractual compliance issues to allow a customer out of the liability for pennies on the dollar of Exa credits. Similarly Oracle has been known to pressure customers with alleged compliance issues that were indeed non-contractual by offering a 90% discount to settle the matter.
In my mind, the moment Oracle puts either of those offers on the table, Oracle is conceding it has lost all moral ground. Yet there are too many organizations that contract on either of those those terms rather than immediately push back from the table and terminate the compliance meeting. Any is too many. And I believe there are too many.
There are those that believe it would be ok to knowingly use extra license features enabled by default without paying the license. Although I am not similarly inclined, I understand and respect the position.
Anybody recall Larry Ellison’s assertion the 10g Database would make the DBA obsolete? It’s sure a good thing that happened because it would appear we need to reallocate those payroll and benefits resources to licensing staff, and related procurement and legal personnel. Today’s “Oracle Database Licensing Information 12c Release 1 (12.1) Part No. E49208-03” weighs in at a hefty 130 pages–approximately 118 of those being actual content. That’s almost a 1,000% increase over that doc’s 2004 debut. That’s just DB licensing, folks. Are shops daunted or overwhelmed by that? Maybe some didn’t even know the guide existed? No worries. Oracle sales will be happy to interpret it for you.
Let’s give Oracle the benefit of the doubt and assume it was in a rush to get 10.1 out the door in 2004. So maybe Oracle’s documentation folks didn’t get adequately in sync with product management. The result of the hypothetic scenario: STATISTICS_LEVEL defaulting to TYPICAL (licensing obligation land mine planted) got left out of the doc. That Oracle didn’t correct the “oversight” by minimally including how to disable such an extra license feature in the next rev of the licensing doc (let alone for years after until 11.1) either explicitly or by way of external documentation reference, becomes an integrity issue in my view.
What about those that unknowingly use extra licensed features without having paid for them? In my observation, the largest RDBMS licensor in the world is remarkably non-litigious in such issues. Aside from Oracle’s heavily discounted Exa credits and 90% off tactics, I am unaware of a single legal action Oracle has brought for such unknowing use. I am dearly hoping that changes. But I don’t have my hopes up because I see Oracle having huge disincentives to file.
Let’s move into the hypothetic court room. Counsel for the defense (licensee) includes these elements in opening comment:
* The Oracle DBA’s primary responsibilities are:
* (Whether or not my peers would argue I got that list right is beside the point; understanding Oracle licensing is not high on the list.)
* Oracle shipped extra license features enabled by default.
* The licensing guide declared even command line query access constitutes invocation of an extra license liability.
* The licensing guide content has expanded ~1,000% since its debut a decade ago.
* The licensing guide has been consistently silent on disabling extra license features for years until the introduction of the parameter CONTROL_MANAGEMENT_PACK_ACCESS in 11.1. Of course, that parameter defaults to both Diagnostic and Tuning Packs enabled but one must look externally to the licensing guide to make that discovery.
* A licensee’s DBA deliberately accessed an extra licensed feature without being aware it was an extra licensed feature, or unknowingly accessed an extra licensed feature.
I can see the judge turning to the Oracle bench with an initial inquiry, “Really?” I smell a quick case, if the trial isn’t pre-empted by the judge’s granting defense counsel’s motion for summary judgement.
Far from seeing accidental use licensees at risk, I’m salivating over the establishment of case law wherein Oracle is required to tolerate the masses’ bundled use of formerly-claimed extra license features until the release is deprecated. No doubt Oracle would immediately retool new releases so as to correct the “oversight.”
My current feeling is the radical expansion of the licensing guide and any expansion of enabled-on-install or easily enabled extra license features strengthens the legal standing minimally of organizations with unintentional extra license feature use.
Well reasoned. Coming from the negotiating school of Monroe Milstein and Michael Prince, it would not occur to me that people back down on these issues when they are in the right. I’d heard of Oracle making offers to get in compliance cheaply, but I (apparently foolishly) thought that was targeted at customers who were really using unlicensed features in the name of both keeping the customer and gaining future maintenance dollars. I have no basis to doubt your experience in the matter. Oracle has never treated ME that way. I encourage anyone who steps in one of these traps who is not abusing license rights to be firm. Perhaps the IOUG can help push back on this practice.
I disagree this is a departure from Oracles standard practice up till now. Try the following:
Install Oracle taking all the default options.
Create a database using dbca, take all the default options.
You are now using partitioning and spatial.
Install enterprise manager, using the default options, and allow it to run during the maintenance window. Accept the recomendataions.
You are now using tuning and diagnostic packs, and advanced compression.
If you were sent on an Oracle University DBA course, this is what you would have been advised to do. The trainers will not tell you about extra cost options, so unless there is someone on the course who already knows about the existence of cost options, there is no way of finding out about these things (Apart from reading the entire manual library)
Oracle have always tried to trick people into using cost options, and is why ther compliance team get so much revenue.