I’m going to tell you something nobody else knows. You’ve heard it here first. Ready? Here’s the deal, no more than 800 MB/s can pass through two 4 Gb Fibre Channel HBAs into any host system memory. It’s that simple. If you want more than 800 MB/s available for your CPUs, you have to either add more 4 Gb HBAs or go with 8 Gb Fibre, or drop FCP all together and go with something that can deliver at that level, but this isn’t a plug for the Manly Man Series on Fibre Channel Technology, I’m blogging about Data Warehouse Appliance technology, specifically DATAllegro.

Exit Conventional Wisdom, and Electronics!

Here is a graphic of the V3 DATAllegro building block. It’s two Dell 2950s (a.k.a., Compute Nodes) each plumbed with two 4 Gb Fibre Channel HBAs to a small EMC CX3 array. According to this piece on DATAllegro’s website, they are the only people on the planet to push more than is electronically possible through two 4 Gb HBAs, I quote:

Data for each compute node is partitioned into six files on dedicated disks with a shared storage node. Multi-core allows each of these six partitions to be read in parallel. Data is streamed off these partitions using DATAllegro Direct Data Streaming^TM (DDS) technology that maximizes sequential reads from each disk in the array. DDS ensures the appliance architecture is not I/O bound and therefore pegged by the rate of improvement of storage technology. As a result, read rates of over 1.2 GBps per compute node are possible.

That’s right. I wasn’t going to point out that each compute node is fed by six disks, because if I did I’d also have to tell you they are 7200 RPM SATA drives, mirrored. Supposedly we are to believe that the pixy dust known as Direct Data Streaming^TM can, uh, pull data at what rate per spindle? Yes, that’s right, they say 200 MB/s per drive! Folks, I’ve got 7200 LFF SATA drives all over the place and you can’t get more than 80 MB/s per drive from these things (and that is actually fairly tough to do). Even EMC’s own specification sheet for the CX3 spells out the limit as 31-64 MB/s. I’ll attest that if your code stays out on the outer, say, 10% of the drive you can stream as much as 75-80 MB/s from these things. So with the DATAllegro system, and using my best numbers (not EMC’s published numbers), you’d only expect to get some 480 MB/s from 6 7200 RPM SATA drives (6×80). Wow, that Direct Data Streaming^TM technology must be really cool, albeit totally cloak and dagger. Let’s not stop there.

What about this 1.2 GB/s per compute node claim? How do you pump that through 2 x 4 Gb FC HBAs? You don’t. Not even DATAllegro with all those Cool Sounding^TM technologies. What’s really being said in that DATAllegro overview piece is that their effective ingestion rate is some 1.2 GB/s, I quote:

Compression expands throughput: Within each node, two of the multi-core processors are reserved for software compression. This increases I/O throughput from 800MBps from the shared storage node to over 1.2 GBps for each compute node.

They could just come out and say it, but they expect you to believe in magic. I’ll quote Stuart Frost (CEO, DATAllegro) on more of this magic, secret sauce:

Another very important aspect of performance is ensuring sequential reads under a complex workload. Traditional databases do not do a good job in this area - even though some of the management tools might tell you that they are! What we typically see is that the combination of RAID arrays and intervening storage infrastructure conspires to break even large reads by the database into very small reads against each disk.

I have news for you. Traditional databases are only victims of what storage arrays do with the I/O requests by way of slicing and dicing. It is not a characteristic of a traditional database system. Even a Totally Rad Non-Traditional RDBMS^TM like the one DATAllegro embeds in their compute nodes (spoiler: it’s Ingres, nothing new) will fall prey to what the array controller does with large I/O requests. In fact, even the CX3 specification sheet stipulates support for only 4, 16, 64, 128, or 256 sector stripe widths. I didn’t have to quote that because everyone who knows Fibre Channed storage arrays well enough to hate them is more than aware that they all chop up large I/Os in the array head. While DATAllegro doesn’t appear to use the CX3 striping, they do rely on the CX3 for mirroring. This type of RAID also chops up I/O requests. Unless DATAllegro has custom firmware for the CX3, their large (e.g., 1 MB, 4 MB, or who knows) sequential I/O requests are getting chopped up and are hitting several drives for each compute node I/O request.

If I’m wrong, I expect DATAllegro to educate us, with proof, not more implied Awesomely Fabulicious CoolFlips Technology ^TM. In the end, however, no matter whether they managed to get custom firmware for the CX3 to achieve larger transfer sizes than anyone else or not, I’ll bet dollars to donuts they can’t push more than 800 MB/s through dual 4 Gb FCP HBAs, and certainly not from 6 7200 RPM SATA drives.

I’ve been watching all these come-lately DW/BI technologies for a while now-especially the ever-so-highly-revered “appliances.” I’m also interested in columnar orientation as my past posts on columnar technology (e.g., columnar technology I, columnar technology II) will attest.

Rows and Columns, or Columns and Rows?

I don’t know, because in that famed Unfrozen Caveman Lawyer style, these things confuse me. However, Stuart Frost, CEO of DATAllegro, puts it this way in his fledgling blog:

At the end of the day, column orientation is just one approach to limiting the amount of data read for a given query. In effect, it’s an extreme form of vertical partitioning of the data. In modern row-oriented systems such as DATAllegro, we use sophisticated horizontal partitioning to limit the number of rows read for each query.

Clue’isms are Truisms

Huh? “Sophisticated horizontal partitioning?” Now that is a novel approach. And if all I want to scan is a column or two with Oracle, I’ll create an index. Is it really that much more complicated than that? An index is columnar representation after all. Heck, I could even partition that “colmnar representation” with a sophisticated horizontal partitioning technology (that has been in Oracle sing the early 1990s) to further reduce the data ingestion cost.

Indexes == Anathema

Oops, I should wash my mouth out with soap. After all, the “appliances” shall save you from the torment of creating a few indexes, right? Well, maybe not. The term of the day is “Index-Light Appliance.”

So I have to ask, what if I were to implement an Oracle-based data warehouse that used, say, 5 indexes. Would that be an Index-Light approach?

Oracle is taking steps to make the configuration of hardware for a DW/BI deployment a bit simpler. If you haven’t yet seen it, the Optimized Warehouse Initiative is worth investigating.

Learning “New and Exciting” Things About Really Old Stuff

…that’s what Max Kanat-Alexander seems to be doing based upon his recent Oracle-bashing rant. Now, I’m not calling Max to the mat because I’ve learned that the Web grants virtual get out of jail free cards to people earning a living developing free stuff, and I have in the recent past been a user of bugzilla (Max is the primary developer), albeit not by choice. I never could get too excited about a bug tracking system that wasn’t integrated with customer support, contracts, field logistics and other general CRM. There certainly is no shortage of bug tracking software, but I’m not blogging about that.

Hey, Old Dogs: Time For New Tricks

The bit I don’t like about Max’s rant is the absurd assertion that Oracle professionals must certainly have never used any other database. I quote:

Most Oracle DBAs, it seems, have never used any other database system. Or they have, but it was in Ancient Times before there was a SQL Standard or something. (By the way, that would have to have been before 1992, when SQL-92 was made. Hi, welcome to the 90’s!)

The nineties? Please! The first SQL ANSI standard was 1986–7 years after Oracle made the first SQL-based commercial RDBMS with lessons taken from the System/R and other playbooks. Yes, 1986, which according to Max’s profile coincided with his days in elementary school. That coincided with the time period in which I was developing and maintaining Informix ACE/ALL applications that fronted IBM 370 mainframes. No, Max, Oracle professionals are not, by and large, rdbms-xenophobes. In fact, the opposite is true. Most shops that deploy Oracle also deploy other products because they have real data centers. By the way, using databases before there was a SQL standard (1986, not 1992) wouldn’t have much to do with SQL because it was, uh, quite scarce.

Max quickly throws us a bone:

I don’t think Oracle is a totally worthless product.

But seemingly recants with the following red-herring:

I know that my Oracle install stopped working once just because I had added, oh, a fifth database to it. Apparently you have to explicitly tell Oracle (with a very cryptic command that’s specific to just your system, because it involves filesystem paths) that you want to have more than about five databases.

I’m not even going to validate that assertion by discussing it. Wait, I changed my mind. No, I’m not going to do it-the assertion is absurd. Just because one tries to base more than 5 databases from a single ORACLE_HOME and had a filesystem-related problem certainly doesn’t mean it can’t be done or isn’t supported. It’s all about configuration resources. How about 80 databases from a single shared ORACLE_HOME in a cluster?

No, Max, we don’t like Oracle because we are ignorant. We use it to solve problems. Can some of those problems be solved by free stuff? I suppose, but I don’t care.

Max continues:

Okay, so I’m biased and I have an unusual viewpoint…[text deleted]…Most people aren’t porting a shipping ANSI SQL application to many different databases. But I am, which means I’ve learned a lot about all the databases.

Huh? Applications supported on all of Oracle, DB2, SQL Server, etc, etc? Avante Garde!

Finally, Max lays it all out there in true protest style:

In every other database out there, an empty string and NULL are not the same thing. The Oracle SQL Reference tells you not to treat an empty string like a NULL (because they might change that behavior in the future), but they don’t actually give you any way to not treat it like a NULL!

You can’t SELECT a CLOB (that’s a TEXT field to the rest of the world) if there’s a GROUP BY clause. What?

Subtracting one month from March 29, 2007 gives you…February 29, 2007, a day that never existed. In fact, because it never existed, Oracle throws an error if you do that. Other databases just give you February 28 (or March 1 if you’re adding, I think).

Oracle doesn’t support the ANSI SQL “LIMIT” clause, it uses something weird in the WHERE clause instead.

Oracle has a hard limit on IN clauses of 1000 items. But it doesn’t complain if you OR together multiple IN clauses with 1000 items each…

Oracle doesn’t allow identifiers to be longer than 32 characters (index names, column names, etc.).

Oracle doesn’t support ON UPDATE CASCADE for foreign keys. Even MySQL supports that, nowadays.

And each of these 7 points have not been churned over and over about a million times on the Web?

Readers, have anything to say about this?

While doing an install of 11g x86_64 11.1.0.6 Clusterware today I hit a problem I’ve seen before but had to think for a moment about what was the cause of the error. I figured if I had to scratch my head on this one, someone, someday would likely be out googling for the answer. Here is the error text as it appears in the log:

The location /data/ocr.dat, entered for the Oracle
Cluster Registry (OCR) is not shared across all the nodes in the cluster.
Specify a shared raw partition or cluster file system file that is visible by
the same name on all nodes of the cluster.

In the following screenshot you can see that in the shell at the bottom I first did a chown ora.dba of the nfs directory where I want to locate the OCR file. Clear evidence of cheating. See, when I did that I was able to proceed to the next screen (voting disk). I instead hit the back button on the voting disk screen and changed the ownership of the /data directory just to raise the error and make this quick blog entry. Nice, aren’t I? Anyway, the deal is that this is an error message without an error number erroring erroneously-afterall, the /data directory was in fact shared between the nodes. The problem was that the install precedure uses a write in that directory as evidence of whether it is shared or not. If it isn’t writable it thinks it isn’t shared.

For sentimental reasons I’ve taken interest in Dataupia. See, their offices are in One Alewife Center, Cambridge, Mass and due to my background in NUMA technology I harbor sentimental feelings for the MIT Alewife System, which, along with DASH were truly the front-front runners in early non-commercial implementations of NUMA technology. However, beyond that cursory connection between Dataupia’s operations location and an obscure non-commercial NUMA system, I quickly find myself confounded by Dataupia. But confounded on the basis of the technology? “Not so much”, I say, in that classic Borat style. Oh, no, I’m much too petty for that.

What confounds me is how to pronounce the name of this outfit. Yes, you heard right. Let me get this straight, I’m told that some pronounce it day-tah-toe-pee-yah. Ugh, I only see on letter t. I’ve heard folks say day-tah-yoo-toe-pee-ah. I say it is impossible to get more than 5 syllables out of Dataupia and I still only see one letter t and only certainly no letter o.

That leaves me with the only phonetically correct possibility: day-tah-you-pee-ah. All I can say is they were smart not to brand their company with those characters split into two words.

Oh that was weak, but hey, I flew back from headquarters yesterday and had a few minutes with the laptop and no WiFi…so you get what you get

How Much Data (Love) Do You Need?

I hate poor marketing…with a passion. Even more so when it smells so similar to a Leo Sayer tune. Check out the following screen shot. What does “as much data as an organization needs” mean? But honestly, repeating the very same lyrics in the next stanza-just for emphasis sake, or just in case we had forgotten so quickly. And while I’m being so petty, could someone tell me what in the heck “persistent access” is supposed to mean? Even after reading that pair of words twice within 20 words in the same paragraph I still don’t understand it. The only thing that comes to mind when I think of persistent access is VNC or Sun Ray. Anyway, here is the screen shot:

Proofless Benchmarks

Does anyone see any proof in this link to a supposed “benchmark?”

In the transcript of yesterday’s earnings call, Larry Ellison said:

We are not going to sit on our laurels; we have a major database innovation that we will announce in September of this year. It is going to be a very big and important announcement for us so we are not standing still in database.

I know, and that is why I can’t blog about any of the work I’m doing at Oracle…at least until September.

I’m not going to hide my true feelings any longer. I hate Fibre Channel SANs.

Yes, I know, I haven’t exactly hidden my position on that topic, considering all the SAN related postings I’ve made. Look, Fibre Channel SAN was great technology for connecting large numbers of disks to a single, large SMP. I just don’t think the technology has a rightful place in Grid computing. I know, broken record.

I’m glad to see that one year after HP bought my former company (PolyServe), the technology is starting to show up in interesting packaging as this piece in Enterprise Storage Forum points out. That’s right, HP is using PolyServe for extremely large scale clustered NAS-a good fit. But that is not why I’m blogging the point.

A few things about the HP StorageWorks 9100 Extreme Data Storage System stand out to me. First, HP has pumped PolyServe’s stomach. I’ve sifted through the, er, um, egestion and found a Fibre Channel SAN suspended in the colloid. Ah, what a relief. That’s right, the StorageWorks 9100 is not a Fibre Channel SAN gateway. It’s all SAS.

The other thing that I see percolating to the top of HP’s messaging on this solution is the fact that if you so desire you can execute applications in the NAS heads. So if you want to have local-disk access speed for data you’ve ingested via NFS, you can do that. Consider, for instance, the ability to log into a NAS head and perform compression without any network overhead. But then, that’s nothing new since this particular product has always supported that sort of mix.

I also read somewhere that folks are critical of HP’s goal to bother offering such extreme capacity. Some folks (rightfully) argue that Petabyte storage “needs” are usually a sign of postponing the difficult work of determining what to archive and when. ILM is tough business to get right, I know. However, the main application of this particular HP offering is for Web 2.0, and face it, when someone wants to view a photo they don’t want to see a browser hour glass while the application goes off to near-line storage for a photo that hasn’t been viewed in the last 90 days.

Finally, I see in the materials that they tout 200MB/s NAS bandwidth per NAS head. I think they are low-balling for safety sake. This system consists of 3.5″ SAS drives and 12 of them offer substantially more than 200MB/s. Trust me. More like 900-1000 MB/s actually. But then, that depends on the workload. I suppose randomly plucking out photo/video content mixed with writes at the rate of 200MB/s per NAS head sounds pretty good.

What does this have to do with Oracle?

Well, I’m an Oracle-minded guy who hates Fibre Channel SANs and Oracle on NFS is fully supported. And, oh yeah, I do like SAS. Finally, I’m former PolyServe and it is my blog, so, I blogged it

…have been greatly exaggerated. However, I certainly have not been blogging as frequently as I’d like. The fact is that I am heads down doing performance work on a product that I cannot speak about until Oracle OpenWorld. Larry Ellison is happy to launch the product at the show and I shan’t steal thunder!

I did notice that my seemingly dead blog was propped by Jeff Hardison who was the gentleman that recommended WordPress as a blogging platform to me. He also gave me sage advice like, “blog everyday, even if it is just a short piece”, which I seem to have ignored completely. I can’t bring myself to blog about nothing. About all I could blog about these days is not having time to blog.

Communicator’s Conference Highlights

Since I’m on site at a Beta customer (testing the product I work on at Oracle), this will be a quick blog entry. I’ve been meaning to direct folks to Gear6 for quite some time now. I have no stake in Gear6, so this is not a shameless plug. I think they solve interesting problems so if you are a large NFS shop, I’d recommend checking them out. They offer a plug-in NFS read-through cache and while I haven’t had first hand experience with their product, I know folks that have and they had good things to say about Gear6.

If any of you are confused about what NFS has to do with Oracle, I recommend this list of Oracle on NFS related posts.

Quite some time back I made a blog entry about deploying Oracle Clusterware for non-RAC purposes. As I pointed out in that entry, there were license ramifications. That was then, this is now.

In this press release about Oracle Clusterware, Oracle is announcing that Oracle Enterprise Linux (with Unbreakable Linux Support) can deploy Oracle Clusterware to provide high availability services for any purpose they so desire.

Now that, is interesting.

Some additional, related links:

Making Applications Highly Available Using Oracle Clusterware

Oracle Clusterware API

Considerations for “Stretch Clusters” with Oracle Real Application Clusters

No worries, we won’t have to lower the Cone of Silence. True, you will see use of an “underbar” init.ora parameter in this post, but its use is not the central theme. No, no Silver Bullets here. This is another post in the Little Things Doth Crabby Make series.

I routinely brag about the sophistication level of my blog readers, so, folks, don’t let me down. Let’s start a thread about why the contents of the following session output would make my Little Things list. OK, come on…

SQL> set timing on
SQL>
SQL> alter session set "_parallel_broadcast_enabled" = FALSE
  2
SQL> select count(*) from ap_ae_lines_all where AE_LINE_ID > 1397437860 ;

  COUNT(*)
----------
         0

Elapsed: 00:01:21.70
SQL>
SQL> alter session set "_parallel_broadcast_enabled" = FALSE;

Session altered.

Elapsed: 00:00:00.00
SQL>
SQL> select count(*) from ap_ae_lines_all where AE_LINE_ID > 1397437860 ;

  COUNT(*)
----------
         0

Elapsed: 00:01:30.46

But my, oh my, how I’ve tried. OK, I guess my new name is Fan Boy. I know for a fact that I’ve been pretty relentless on this particular server for over 100 days of its current 215-day life.

-sh-3.00$ cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 3)

-sh-3.00$ uptime
 14:41:17 up 215 days, 14:32, 15 users,  load average: 37.85, 37.48, 25.89

And, top(1):

  top - 14:40:44 up 215 days, 14:31, 15 users,  load average: 40.91, 38.05, 25.62
Tasks: 309 total,  30 running, 278 sleeping,   0 stopped,   1 zombie
Cpu0  : 92.8% us,  7.2% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu1  : 90.1% us,  9.9% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu2  : 89.3% us,  9.8% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.9% hi,  0.0% si
Cpu3  : 90.1% us,  9.9% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu4  : 89.2% us,  9.9% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.9% hi,  0.0% si
Cpu5  : 89.1% us, 10.9% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu6  : 92.8% us,  7.2% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Cpu7  : 93.7% us,  6.3% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:  10393736k total,  9347616k used,  1046120k free,     1892k buffers
Swap: 10288440k total,   838236k used,  9450204k free,  6264396k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
14919 kclosson  15   0  120m  84m 7076 S 30.3  0.8   0:17.67 sqlldr
14942 kclosson  15   0  119m  84m 7068 S 29.4  0.8   0:17.75 sqlldr
14940 kclosson  15   0  120m  84m 7068 S 28.6  0.8   0:16.21 sqlldr
15008 kclosson  16   0  668m  35m  29m R 28.6  0.3   0:16.48 oracle
14924 kclosson  15   0  119m  84m 7076 R 26.8  0.8   0:16.39 sqlldr
14932 kclosson  16   0  120m  84m 7068 R 26.8  0.8   0:17.07 sqlldr
14959 kclosson  15   0  668m  34m  29m S 25.9  0.3   0:15.96 oracle
14961 kclosson  16   0  668m  34m  29m R 25.9  0.3   0:14.90 oracle
14945 kclosson  15   0  119m  84m 7076 S 25.0  0.8   0:16.07 sqlldr
14980 kclosson  15   0  668m  34m  29m S 25.0  0.3   0:15.09 oracle
14935 kclosson  16   0  119m  84m 7068 S 24.1  0.8   0:15.05 sqlldr
14947 kclosson  16   0  119m  84m 7072 R 24.1  0.8   0:15.90 sqlldr
14943 kclosson  15   0  119m  84m 7076 R 23.2  0.8   0:14.75 sqlldr
14938 kclosson  16   0  120m  84m 7068 S 22.3  0.8   0:14.35 sqlldr
14941 kclosson  15   0  119m  84m 7076 R 22.3  0.8   0:15.96 sqlldr
14951 kclosson  15   0  120m  84m 7068 S 22.3  0.8   0:16.96 sqlldr
14921 kclosson  16   0  120m  84m 7068 R 21.4  0.8   0:17.84 sqlldr
14934 kclosson  15   0  120m  84m 7076 S 21.4  0.8   0:16.13 sqlldr
14929 kclosson  15   0  119m  84m 7076 R 20.5  0.8   0:17.70 sqlldr
14950 kclosson  16   0  119m  84m 7068 R 20.5  0.8   0:13.63 sqlldr
14922 kclosson  15   0  120m  84m 7068 S 19.6  0.8   0:17.40 sqlldr
14977 kclosson  15   0  668m  34m  29m R 18.7  0.3   0:16.38 oracle
15002 kclosson  16   0  668m  34m  29m R 18.7  0.3   0:15.00 oracle
14920 kclosson  16   0  119m  84m 7076 R 17.8  0.8   0:17.97 sqlldr
14923 kclosson  16   0  119m  84m 7068 R 17.0  0.8   0:13.44 sqlldr
14925 kclosson  16   0  120m  84m 7068 S 17.0  0.8   0:13.06 sqlldr
14927 kclosson  16   0  119m  84m 7076 R 17.0  0.8   0:15.05 sqlldr
14931 kclosson  16   0  119m  84m 7076 R 17.0  0.8   0:15.18 sqlldr
14957 kclosson  15   0  668m  34m  28m S 17.0  0.3   0:14.16 oracle
14930 kclosson  16   0  120m  84m 7068 R 16.1  0.8   0:15.31 sqlldr
14986 kclosson  15   0  668m  34m  29m R 16.1  0.3   0:14.37 oracle
14936 kclosson  15   0  119m  84m 7068 S 15.2  0.8   0:15.58 sqlldr
14964 kclosson  15   0  668m  34m  29m S 15.2  0.3   0:17.10 oracle
15014 kclosson  15   0  668m  34m  28m S 12.5  0.3   0:12.83 oracle
14949 kclosson  16   0  120m  84m 7076 S  7.1  0.8   0:15.70 sqlldr
14955 kclosson  16   0  666m  35m  31m R  4.5  0.4   0:03.11 oracle
14966 kclosson  16   0  666m  35m  31m R  4.5  0.3   0:02.80 oracle
14998 kclosson  15   0  666m  35m  31m S  4.5  0.3   0:02.68 oracle

There are 120 spindles downwind of this….drumroll please….

$ asmcmd
ASMCMD> ls
SDATA1/
SDATA2_1/
SDATA2_2/
SDATA2_3/
ASMCMD> rm -fr *
ASMCMD>

Eek!

In other news, I just learned that Greg Rahn is about to get his hands on a set of HP DL580 G5 (Tigerton goodies). I’d sure like to get Silly Little Benchmark numbers from that sweetheart of a system…wish in one hand, you-know-what in the other I suppose.

Oracle’s acquisition spree has wound up producing odd bedfellows.

I’ve often wondered why companies really want to own their computers. HP has voiced the same rhetorical question by launching AIaaS.

Why Do They Put Those Data Centers There?
But Atlanta? Is power really that cheap in Atlanta? It must be all that cheap hydroelectric power down there.

What’s This Have To Do With Oracle?
This is an Oracle-related blog, but I can’t answer that question. I’m watching just like you all are.

The Irony
I find it very ironic that HP announced this hosting initiative at their Technology@work Conference in Barcelona Spain considering the other hot news about the HP DL785 which is an 8-socket “Barcelona” Socket-F based Server. I am happy to see Proliant 8-socket Servers.

More on the DL785

More on the “Barcelona” Quad-Core.

Network Appliance has renamed their company a nickname. That just seems strange to me. Different name, same stuff.

And, yes, NFS works nicely for Oracle Database.

The newly-branded NetApp

Maybe Oracle should be renamed/rebranded to “scott/tiger?”

Some rambling thoughts…

Oracle Database Administrator or Data Administrator?
Oracle continues to add features/functionality that makes the dutiful Database Administrator look a lot more like a Data Administrator. Some would argue it started with ASM since volume management was placed into the hands of the DBA. I’d argue it started with RMAN since that feature was one of the most significant, early features that took functionality away from the OS (and 3^rd Party software) and put control in the hands of the DBA. After all, Oracle backups before RMAN had very little to do with Oracle (version 7 and earlier). I’d argue further that features like SecureFiles also make the Database Administrator more of a Data Administrator-or at least it seems it should make a DBA start to think like a “Data Administrator.” I think that trend will continue. I think that increases the value proposition of the Oracle DBA in the datacenter. I think it is a good thing.

DBAs might even find themselves performing lower-level storage operations such as creating OS-level logical drives. One such example would be HP Smart Array logical drives and the associated CLI is the hpacucli command on Linux.

I’ve been Meaning to Blog This One…
Quite some time ago I was fiddling with hpacucli. Yes, it was late at night. Yes, I was tired and crabby. Why would what’s happened in the following screen output make me crabby?

# hpacucli
Array Configuration Utility CLI 7.15.19.0
Detecting Controllers...Done.
Type "help" for a list of supported commands.
Type "exit" to close the console.
=>  ctrl all show status
Error: Expecting “controller” in target. Type “help” for more information.
=> ctrl all show status
Smart Array P400 in Slot 1
   Controller Status: OK
   Cache Status: Not Configured
   Battery Status: OK

… the CLI doesn’t parse out leading whitespaces.

Maybe somebody, someday will be battling the hpacucli CLI and google their way to this post.

P.S., For goodness sake, please don’t go running to your local HP rep to tell him I’m bashing HP. Yes, HP Smart Array is great technology. I just don’t think whitespaces are as significant as hpacucli does

Yes, I admit, sometimes little things make me crabby. Or is it that being crabby makes me easily irritated at small things? I think it’s the former rather than the latter.

Angry Email Makes Me Crabby
I am surprised at how many emails I’ve received from angry Storage Administrators about my position against Fibre Channel. I generally just ask them to not take it so personally. After all, there is more to storage than whatever storage networking protocol any particular server is plumbed with. Anyone who reads my Manly Man series about Fibre Channel will see that I’m just not a big fan of the current misapplication of Fibre Channel technology. As I point out in installment number two in the Manly Man series, Fibre Channel was originally intended to solve the difficulties associated with connecting hundreds (or thousands) of low capacity drives to a small number of large servers. These days it is being used to provide storage connectivity for hundreds of servers to a couple of huge FC storage arrays. And these hundreds of servers have both Ethernet and FC connectivity-overkill, plain and simple.

A lot is changing in the storage world where Oracle is concerned. I hope to come out of the foxhole to blog more about it over the next few months. Well, maybe I should say a few months from now. In the meantime I’ll be making some datacenter visits. I always learn something interesting when I get a chance to do that.

Little Changes Make Me Crabby
Like, for instance, the following example of the change to iostat -x output from RHEL4 to RHEL5. Notice how the rkB/s and wkB/s columns are missing? Yes, it is simple math to divide [rw]sec/s by 2 to get the same data, but there must be at least 42 of you folks that are feeding iostat -x output through some text processing, right? Time to adjust your scripts. Or is it? I wonder how many folks are really diving into OEL5/RHEL5? It seems a conservative IT shop with a functional RHEL4 configuration would most likely stay put. Any readers care to voice their experience with OEL5/RHEL5 adoption? Is it happening? Are folks waiting for their 11g adoption with plans to team that up with a move to OEL5/RHEL5?

# uname -r
2.6.18-53.ELsmp
# iostat -x 1 1
Linux 2.6.18-53.el5 (host1)        03/07/2008

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.08    0.00    0.04    0.03    0.00   99.85

Device:         rrqm/s   wrqm/s   r/s   w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d0        0.16    10.38  0.33  0.95    10.55    90.64    79.47     0.02   14.12   0.86   0.11
sda               0.00     0.00  0.00  0.00     0.01     0.00    21.05     0.00    3.18   2.87   0.00
dm-0              0.00     0.00  0.45 11.33    10.47    90.64     8.58     0.39   32.67   0.09   0.11
dm-1              0.00     0.00  0.00  0.00     0.01     0.00     8.00     0.00    5.05   0.51   0.00
cciss/c0d1        0.00     0.00  0.00  0.00     0.01     0.00    10.12     0.00    4.56   2.63   0.00
cciss/c0d2        0.00     0.00  0.00  0.00     0.01     0.00    10.38     0.00    4.30   2.52   0.00
cciss/c0d3        0.00     0.00  0.00  0.00     0.01     0.00    10.38     0.00    6.17   3.35   0.00
cciss/c0d4        0.00     0.00  0.00  0.00     0.01     0.00    10.38     0.00    5.50   3.22   0.00
cciss/c0d5        0.00     0.00  0.00  0.00     0.01     0.00    10.64     0.00    3.38   2.59   0.00
cciss/c0d6        0.00     0.00  0.00  0.00     0.01     0.00    10.64     0.00    4.05   3.31   0.00
cciss/c0d7        0.00     0.00  0.00  0.00     0.01     0.00    10.12     0.00    4.44   2.29   0.00

$ uname -r
2.6.9-34.ELsmp
$ iostat -x 1 1
Linux 2.6.9-34.ELsmp (host2)    03/07/2008

avg-cpu:  %user   %nice    %sys %iowait   %idle
           1.79    0.00    1.28    0.30   96.63

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
cciss/c0d0   0.06  11.83  0.14  3.21   23.07  120.39    11.53    60.19    42.77     0.68  201.33   4.60   1.54

Jonah H Harris has provided a good list of external data unloading tools that people use with Oracle. It is worth a gander at:

Oracle Data Unloading Tools

I first heard about Jonah’s blog via OakTable Network email from Tanel Poder. I’ve been reading Jonah’s blog and it looks like good stuff so far.

…just a quick blog entry to plug Daniel Morgan’s (of Puget Sound Oracle User Group) very useful collection of scripts and other goodies:

Morgan’s Library

I’ve been reading a lot of blogosphere content about Data Warehousing these days. I’ve taken a lot of interest in such technology as Netezza, GreenPlum, DATAllegro and others and blog reading proves to be an interesting way to augment one’s knowledge. Who’d have thought I’d learn so much about OLTP through this reading.

Memory is Faster than Disk, So Let’s Do a Complete Rewrite
Why, just today I found out that it is time for a total rewrite of commercial RDBMS products. Uh huh. More interestingly, though, I learned:

• Memory is faster than disk. Really, truely, it is!
• A dual-core (2.8GHz) server with 4GB memory and 4 250GB SATA drives can perform 51,000 TpmC
• Disabling transaction logging entirely in a commercial RDBMS will increase throughput (TpmC) about three-fold

I found these pearls of wisdom while reading a Stonebraker paper referred to on this blog post. Yes, I know that blog is basically a store-front for Vertica, but I like to learn about different things that are going on in database technology. Unfortunately this time I was wasting my time. The URL in that blog post points to the VLDB front page, but a little sleuthing found the paper posted here: The End of an Architectural Era (It’s Time for a Complete Rewrite).

Recite after me:

If you get two orders of magnitude performance gain, you are either not doing it or you’ve moved it closer to the processor.

Dang, and I ain’t even got no too pretty good pedigree. Pshaw, I dasn’t fidget ‘mungst the quality!

Central versus De-centralized versus Shared-Nothing
No, it isn’t time for a re-write, especially one that requires a complete shared-nothing database approach. Now don’t get me wrong, I’m all for de-coupling and grid architecture-most particularly where storage is concerned. If I hear of another poor production site that is head-saturated on a $500,000 storage array when driving a measly 15 or so 15K RPM drives, I’ll BAARF. Please see the following post for what I’m talking about:

Hard Drives Are Arcane Technology. So Why Can’t I Realize Their Full Bandwidth Potential?

In my post called Building a Stretch Real Application Clusters Configuration? Get The CRS Voting Disk Setup Right!, I linked to a paper Oracle maintains that explains how to use an NFS export from a small Unix/Linux server as storage for a third voting disk in a stretch RAC cluster. I pointed out that the paper instructs on how to use the noac mount option for Linux RAC clusters in spite of the many resources that suggest actimeo=0 will do. The authors of the document are standing fast that if you are building a Linux-based RAC stretch cluster and are using an NFS mount as a third voting device you do indeed need to mount that particular NFS filesystem with noac. That nugget of truth contradicts so many different documents that I don’t care to list. Instead, I’ll list a resource from Metalink that helps clarify the issue. In fact, I would say that no matter what the sundry Installation Guides or Release Notes say, refer to Metalink 359515.1 when the topic of Oracle Database 10g on NFS filesystems comes up.

Datafiles or CRS Files? The Mount Options Differ.
Metalink 359515.1 is a really helpful note. It spells out the RAC-related mount options for 10gR2 on Solaris, AIX, HP-UX and Linux. Most importantly, it spells out the options for the datafiles and the CRS files in two separate columns. Lo and behold, Metalink 359515.1 clearly spells out that noac is needed for CRS files, but not for datafiles.

In the comment section of Building a Stretch Real Application Clusters Configuration? Get The CRS Voting Disk Setup Right!, a reader points out that the Third Party Vote on NFS paper has a dead end URL (http://www.oracle.com/technology/deploy/availability/htdocs/vendors_nfs.html) in the section that aimed to point out the fact that you cannot just use some Unix/Linux server NFS exports for any other purpose than this unique third voting disk setup in a stretch cluster scenario. He is right, that URL is a dead end, but I’d rather point folks to Linux RTCM (RAC Technology Certification Matrix) or the Unix RTCM, both of which clearly spell out a list of supported NFS File servers. Missing from the list is, of course, some plain old Linux or Unix server dishing out NFS exports-because the only supported application of simple, Unix/Linux NFS exports is the third voting disk scenario in a stretch cluster.

The reader also added this comment:

Many people say Oracle doesn’t support NFS, we need to verify. Searching oracle.com, “We did not find any search results for: vendors_nfs.html” and the references from google all seem to point at that one mysteriously missing doc.

Gee whiz. Where to start? Yes, for the eleventeenth time, NFS filesystems are supported for Oracle Database (including RAC). Let’s not get so easily confused; NFS is a protocol and the storage is NAS. Let’s all enter the following formula in our decoder rings before reading Oracle documents:

(Some Stupid Little Linux/Unix Server Exporting Filesystems via NFS) != NAS

The only supported application of non-NAS NFS is described in the following paper: Using NFS for a Third CRS Voting Device

Now, for some light reading about Oracle on NFS with 11g, I submit:

Oracle Database 11g on NFS filesystems:

Using Network Attached Storage or NFS File Systems Installation Guide for Microsoft Windows

Using Network Attached Storage or NFS File Systems Installation Guide for HP-UX

Using Network Attached Storage or NFS File Systems Installation Guide for AIX 5L Based Systems (64-Bit)

Using Network Attached Storage or NFS File Systems Installation Guide for Solaris Operating System

Using Network Attached Storage or NFS File Systems Installation Guide for Linux

And, of course: Configuring Direct NFS Storage for Datafiles

But, let’s not forget:

• Manly Men Only Deploy Oracle with Fibre Channel Storage - Part I. Oracle Over NFS is Weird.
• Manly Men Only Deploy Oracle with Fibre Channel - Part II. What’s So Simple and Inexpensive About NFS for Oracle?
• Manly Men Only Deploy Oracle with Fibre Channel - Part III. Did I Hear EMC Say NAS?
• Manly Men Only Deploy Oracle with Fibre Channel - Part IV. SANs are Simple, RAC is Difficult!
• Manly Men Only Deploy Oracle with Fibre Channel - Part V. What About Oracle9i on RHAS 2.1? Yippie!
• Manly Men Only Deploy Oracle with Fibre Channel - Part VI. Introducing Oracle11g Direct NFS!
• Manly Men Only Deploy Oracle with Fibre Channel - Part VII. A Very Helpful Step-by-step Install Guide for RAC on NFS.
• Manly Men Only Deploy Oracle With Fibre Channel - Part VIII. After All, Oracle Doesn’t Support Asynchronous I/O On NFS!

The topic of “stretch clusters” has been interesting to a lot of folks for quite some time. A stretch cluster is one where one or more cluster nodes, one of more portions of the SAN or both are geographically remote. Geographically remote could be within eye-sight (1-2km) or a long distance away. YottaYotta (Robin Harris of StorageMojo.com will notice that name) reached out to me (with hardware to offer) several years ago to set up a 3500km stretch cluster with three 10gR2 RAC nodes. Two of the RAC nodes were co-located and the third was put at 3500km distance using communications hardware that simulates the latency imposed by such great distance. And, yes, it is a valid simulation. It was an interesting exercise and with the YottaYotta distributed block server, the PolyServe (HP) and RAC were totally oblivious to the topology. It was a cool project, but that technology has had a difficult time catching on. In the interim, mainstream vendors have stepped up to offer stretch clustering technology and in the name of business continuity, folks are considering these sorts of solutions-but they are expensive. To that end, most shops would tend to buy, at most, a two-legged SAN. Therein lies the problem. Such a configuration could suffer a disaster on the leg of the SAN that has the majority of the CRS voting disks resulting in a total outage of the solution.

The remedy for this problem is to implement a third leg of storage for more voting disks to ensure an n+1 majority are available, but at what cost? The solution is to implement an inexpensive NFS share in which to host these additional voting disks. And, yes, you can use a simple low end Unix/Linux host as the NFS server for this purpose-so long as the host is running Solaris, AIX or HP-UX, or Linux. The following is a link to a paper that covers Oracle’s recommended/supported approach to this solution with Oracle Database 10g Release 2.

Using NFS for a Third CRS Voting Device

The paper is clear about the fact that using some plain Unix/Linux server to host NFS shares for Oracle files is limited to this specific purpose:

Oracle does NOT support standard NFS for any files, with the one specific exception documented in this white paper.

The paper appears to have a small contradiction about mount options-specifically stating that the noac option is required for Linux (see Figure 1) servers which seems to contradict Metalink 279393.1. I’ve sent an email to the authors about that. We’ll see if it changes.

In my recent rant about Oracle database installation difficulties, I provided a link to a video in which fellow OakTable Network members Morten Egan and Mogens Norgaard captured how difficult the task really is.

Well, they’re at it again. You’ll see Morten “The Nose” Egan start out this new video taped Oracle installation by configuring a SAN with what looks like the HP Array Configuration Utility, but then my eyes are getting as bad as my blogging frequency. I couldn’t miss the Windows Disk Manager though-not even on fast forward.

I think we should start calling him Morten “The Hair” Egan. The link to the video follows:

Unconventional Oracle Installation Part II

I thought a comment on one of my recent blog entries deserved handling in a blog entry. A reader posted:

Have you done any comparisons of the HP DL585 with an HP DL580? Is the DL580 a NUMA machine? Which one would you by today for a RAC cluster?

I’ll answer these out of order. The DL580 is not a NUMA system. Although it stands to reason that if HP continues the DL580 product line into the future to the point where they bake in the CSI interconnect then at that time the DL580 would be a NUMA system. So, the short answer to whether or not a DL580 is a NUMA system is no, it is not. I think long answers are more fun.

In my series of posts about Oracle on NUMA, I think I must have said it about umpteen times, but I’ll say it again concisely in this post. I’m talking about what “NUMA-aware” software means. I routinely hear that Oracle is NUMA-aware. It is, and it isn’t. The reason I say this is because there are widely varying degrees of NUMA-awareness that varies between hardware platforms and Oracle ports. I made the point in my recent post about Oracle Database 10g 10.2.0.4 that 10.2.0.4 contains NUMA-related fixes, and it does. However, that isn’t saying it is the fullness of NUMA-aware, because it isn’t. However, the only question that matters is whether it is sufficiently NUMA-aware for today’s NUMA systems, and I’d have to say that it is.

I’ll give a hint: No Linux Oracle release can be fully NUMA-aware until processes (e.g., shadow processes, PQO slaves, etc) can quickly and cheaply detect what CPU they are currently executing on and prefer memory resources based on that locality. Way back in 1996 I was in Advanced Oracle Engineering at Sequent and we were in the late stages of producing the first commercial NUMA system. It was my early Oracle work on Sequent’s NUMA that begat the GETENGNO(3SEQ) API, which was an extremely inexpensive call for processes to check what CPU they are executing on.

Let’s fast forward to today. The Linux development folks are considering the Linux corollary for Sequent’s GETENGNO() with the vgetcpu() call. The problem is that the call is very, very slow compared to the 4-6 cycles that Sequent required to inform a process what CPU it was executing on. Nonetheless, the point is that until vgetcpu() works, and Oracle exploits it, the pinnacle of NUMA-awareness has not been met. And while that may not matter given today’s AMD situation, it will certainly matter when Intel system are NUMA (e.g., CSI based). I guess I shouldn’t equate Linux NUMA with AMD since IBM’s x3950 is a building block for large NUMA systems, and there are others as well. But I was focusing on commodity-level NUMA systems which the x3950 most certainly is not.

There are a lot of factors in selecting hardware, but since I’m asked about DL585 vs DL580, I’d say DL580-so long as it is a DL580 G5. I have tested DL585 and DL580 side by side. However, that was a pretty old DL580 G3 (1066 MHz FSB). I see that the DL580 is now fit with the “Penryn” Xeons (e.g., 5460), which have a front-side bus speed of 1333MHz. There are G5’s that are fit with “Tigerton” Xeons which are 1066 MHz FSB. I’ve seen benchmark results that suggest there is some 21% to gain from going with a 5460-based G5 over a 7350-based G5. So, look closely at the specification. Also, I think a shrewd shopper would try to read the crystal ball to see when the DL580 G5 will be fit with the Xeon 5462 which has a 1600 MHz FSB. As always, with Oracle you want big pipes.