Oracle acquired Pillar Data Systems on June 29, 2011 and I was quite silent on the matter because the topic was not blog-worthy. But, I’m not blogging about the blog-worthiness of old news. The day after the news regarding the Pillar acquisition, Oracle executives held a webinar covering the latest turn in Oracle’s storage strategy. Some of the content in that webinar was, in fact, worthy of a blog entry.
The webinar slides can be found here: Oracle Storage Strategy Update June 29, 2011 (slides).
Fair or Foul?
Imagine for a moment that SAP spends the money to compete in the America’s Cup to race the Oracle boat. Imagine further that the captain of the boat fouled in some way during a race (I know nothing about sailboat races because I’m a mere mortal). It would take Oracle executives about 42 seconds to call out the issue. Well, as I see it, slide number 15 in the above-referenced Storage Strategy webcast slide deck is a foul and while it took me longer than 42 seconds to get to the matter, I’d like to address the matter now.
As the following screen shot of slide 15 shows, Oracle is specifically targeting EMC storage technology to contrast to Exadata. Forget for a moment that the design center for Exadata was DW/BI so the logical comparison would be to Greenplum. That’s saved for Part II. The problem with calling out EMC specifically is in doing so one tends to attract the attention of guys like me. You know, guys who know Exadata really, really well. I’m going to strive to choose my words carefully at this point as I’m sure Oracle did when they scooped all those words into slide 15.
In my assessment, Oracle used misleading information while poking the Slide 15 Stick™ at EMC. But not misleading in the manner you might expect. The misleading information isn’t what Oracle stated about the EMC product in the comparison. Instead, the misleading information is actually in under the Exadata column! So, I’ll call out these errata in the following list and then show the screenshot after that.
- Database IOs per second. This bullet grants a dubious 150,000 to “EMC Symmetrix” and clearly sets the stage for an OLTP-slanted comparison by citing the 1.5 million read IOPS (datasheet) capability for a single full-rack Exadata as per the information in the Exadata column. Don’t get me wrong, I don’t question the 1.5 million read IOPS capacity for 8KB random I/O because I held the position of Performance Architect in Oracle’s Exadata development organization for years and I’ve used my own test kits to get that many IOPS on my own Exadata lab gear. But stick with me for a moment. It’s really, really faulty marketing to pick random capabilities cited in the datasheet and plop them in a slide for your executives to present. The next row in the Exadata column cites a compression figure–and a Hybrid Columnar Compression (HCC) figure at that. Folks, Exadata cannot achieve the datasheet read IOPS figures (1.5 million) with compressed data—most particularly not HCC data. By citing the the 1.5 million read IOPS Oracle sets the stage that this is an OLTP/ERP-slanted comparison to EMC. After all, IOPS is not a DW/BI issue. So I’m scrutinizing the rest of the list from that position.
- Average Database Compression Factor. This is a new move for Oracle. I’ve not seen the claims for “average database compression” before. Generally, you’ll see wording such as “up to 10x compression” or “10x to 15x” compression when referring to the full HCC suite–which offers both query and archive compression. But, no matter. What you shouldn’t like about the claim of “average database compression” is the fact that Exadata is marketed as an OLTP machine yet HCC cannot be used for OLTP. So if you’ve bought Exadata for OLTP you’re probably wondering where your “average” 10x savings is. On the other hand, if you are using Exadata for OLTP, and wisely employing the only compression technology suitable for OLTP ( Oracle Advanced Compression (a.k.a, ACO) ) you are enjoying the same 2-4x compression you would have if your database was stored on any storage—even EMC! Imagine that. Oracle and EMC share tens of thousands of Oracle customers running OLTP or ERP and enjoying the same compression ratios across the board. So, stating zero for EMC in the compression class is, well, you know…
- Database bandwidth, high performance disks. This line in the table is bizarre. I know the Exadata bandwidth datasheet numbers by heart (shouldn’t come as a surprise). The full-rack X2 models with High Performance drives sustain 25 GB/s when scanning only the HDD disks. When scanning both HDD and flash cache concurrently the bandwidth number jumps to 75 GB/s—again, for a full-rack with High Performance drives. I can only surmise that the person who filled in this slide simply multiplied the HDD-only scan rate by the “average” compression factor (10x) to get 250 GB/s. That means the row should be called “effective bandwidth.” But then that isn’t the real problem I have with this point. The row is labeled “database bandwidth.” OLTP databases are databases and I assure you there is no 10x effective increase for OLTP, ERP database I/O bandwidth with Exadata because (and I can’t repeat this enough) there is no HCC for OLTP/ERP use cases. Perhaps the row should be called “HDD Effective Scan Rate” which has little to nothing to do with OLTP so I’m looping back to the 1.5 million read IOPS citation again.
- Database cache usable capacity. This one is really broken. First off, we all know that the “database cache” is first and foremost the Oracle System Global Area which all Oracle databases have regardless of the storage. That is the database cache. If the slide was meant to allude to “storage cache” then fine, let’s take it that way. The slide cites 53 TB for Exadata. Those are lovely characters and numbers but they don’t mean anything. The Exadata Storage Server (X2) has 384 GB of Exadata Smart Flash Cache in each of the 14 storage servers in a full-rack configuration. That’s 5,376 GB. I think, here again, the cited number is a mash-job of physical capacity and the DW/BI compression ratio commonly achieved with HCC (10x). The title of the slide is “Better than EMC for Database I/O.” OLTP databases are indeed databases and OLTP is not compatible with HCC compression. So, the information is…a foul.
So, as in the case of my America’s Cup racing foul example, I cry foul. It took me more than 42 seconds to do so though.
Here is Slide 15:
Follow this link to Part II in the series.
“IOPS is not a DW/BI issue”. Could you please explain the reason?
Charlie,
The challenge for DW/BI workloads is disk bandwidth. Not IOPS. Even a single path of 10 GbE can handle over 100,000 random 8K Oracle IOPS. Even when Exadata is concurrently scanning HDD and flash at the full-rack rate of 75 GB/s the IOPS rate is less than 80,000.
The Exadata 1.5 million IOPS datasheet number is random 8K reads. Scans use sequential large multi-block reads. If you have a DW/BI deployment laden down with indexes then, sure, you will probably be sensitive to IOPS for index range scans…but indexes are not what Exadata is about.
So, if an Oracle slide is posing the value of 1.5 million (read) IOPS think OLTP/ERP, not DW/BI.
Kevin,
You said that you are not sure if that is a reference to EMC Symmetrix VMAX or EMC Symmetrix DMX-4 but in slide 15, it clearly states that it’s EMC Symmetrix V-Max.
You are absolutely right that these numbers are not for OLTP systems, but I don’t see any reference to OLTP systems in this slide. You say that exadata is marketed for OLTP but I know almost all exadata customers (in my country, Turkey) and all of them use exadata for DW. We also do not know if presenter mentioned that these numbers are for DW systems.
Why should the comparison be to Greenplum? The slide is about the performance benefits of the storage technology used by Exadata for Oracle users. It targets to the customers who use EMC for storage (like us). It’s not for comparing Oracle to another database.
Regards
Gokhan
Hello Gokhan,
I feel silly that I didn’t catch the fine print where VMAX is specified (bottom of the slide). I’ll edit that bit out. The point does not change the essence of the blog post though. I think you’ll agree on that.
As for the preponderance of DW/BI use cases for Exadata customers in a particular region, well, it just happens to go that way sometimes. In fact, I’ve presented to, and dined with, the Turkcell folks. Good people putting Exadata to use for it’s intended purpose: DW/BI. On the other hand, however, the *very first* customer ship for the X2-8 model is in fact in your country (Ministry of Education) and there was no small amount of bloodshed early on there regarding cursor mutex thrashing which is the signature earmark of OLTP or ERP. Most of those bugs regarding 11.2 cursor mutex thrashing have been fixed though.
Regarding my assertion that this slide *must* be taken as a OLTP-slanted slide is, as I pointed out in my post, the mere mention of the Exadata datasheet 1.5 million read IOPS makes this an OLTP-slanted slide. Since you know almost all the Exadata customers in your country, and by your words they are majority DW/BI, you must also surely know that their I/O profile is not random single block reads! See my previous comment to Charlie on this matter as well.
The slide is Mulligan Stew. My mention of Greenplum addresses the fact that they threw in an effective scan rate number (the 250 GB/s) into an otherwise OLTP-slanted slide (again, the mere mention of 1.5 million IOPS). If the use case is DW/BI, and thus effective scan rate is a critical value proposition, then the comparison needs to be to Greenplum in which case there is no comparison for effective scan rate. A moderate Greenplum (16 nodes) deployment can easily render effective scan rates in the 2TB/s range. However, since the slide is about OLTP I won’t mention that. Oh, hold it, I just did
Finally, you’re right that slide 15 is about Oracle database. Slide 11 sets the stage. My mention of another database product was just to address the out-of-place citation of the 250 GB/s effective scan rate on slide 15.
As in any competition, crying “foul” will always be contested by the other side. If I have to stick to one main point it is about the 0x versus 10x compression ratio figures. I make the point clear that the 10x is only about DW/BI and Oracle on EMC enjoys the same compression ratio as Oracle on Exadata for OLTP so surely the value under the EMC column should not be zero.
In closing, since slide 15 establishes (in Oracle’s eyes any way) Exadata as better than the best block storage out there (EMC VMAX) for Oracle database I/O, what purpose does Pillar serve? That’s blog-worthy.
“The slide is Mulligan Stew.”
“IOPS is not a DW/BI issue”
THANK YOU, Kevin!
I’ve lost count the number of times some marketing droid has walked to my management with the tired idiocy that “you need more IOPS for your DW, buy Exadata”.
How much of this nonsense can this industry really take?
Enough is enough!…
On another note: we just upgraded to a VMAX. Looking forward to use it, some really good stuff there.
Noons,
I think “the voice of reason” is not a single mouthpiece. I only aim to help make sense of some of the messaging. When Oracle marketing started that Exadata random read IOPS value prop thing I knew I was going to have to find a better situation. There was no interest in metering the message due to the inability of the platform to scale physical writes at a commensurate rate…so…thus it went…
But will it run Mongo DB ? I hear that Mongo DB is web scale
http://www.xtranormal.com/watch/6995033/mongo-db-is-web-scale
Advanced Compression is an Oracle technology. However HCC is an Exadata specific feature. So I guess we can say Exadata achieves better compression than EMC-VMAX. (Unless EHCC has no extra benefits.)
Hello Waseem,
The slide is about Oracle on different storage. Therefore ACO is a constant. Further, the slide brings up the 1.5 million IOPS value prop which is solely an OLTP value prop. Therefore this is an OLTP on EMC versus OLTP on Exadata slide. You cannot use HCC with OLTP. So, the only compression you would use with Oracle on Exadata is the same compression you would use with Oracle on EMC. The slide says EMC has no compression. It is intellectually dishonest.
I hope I don’t have to spell that out again though.
I don’t think you know much about the oracle storage.Oracle has delivered over 30 million TPC and the need storage. World record TPC with world record storage. So?
Ahud, please get informed before you spew [
edited], OK?Kevin INVENTED Exadata!
Check your facts and knowledge, you idiot!
Thanks for stopping by, Noons…always a pleasure. Sorry I had to strike out a bit of your comment.
I need to point out that I do not lay claim to inventing Exadata. Exadata lived several “lives” before I joined the development team in 2007. One of the former lives of Exadata (aka Storage Appliance for Grid Environments SAGE) was Sun “Thumper” (aka x4500) connected via multiple paths of gigabit ethernet. That x4500 is not to be confused with the modern x4500 which is a 4-to-8 socket Nehalem EX (soon to be Xeon E7 formerly) box. The Thumper had 48 drives in it and a couple of Opterons. It made for a good initial platform for early Exadata but the ratio between spindles and cpu was way out of whack.
The real inventors of Exadata are three guys that just sort of came up with the idea and started to crack out the code. One of those “concept inventors” is still in Exadata development–an utterly brilliant developer (Architect). He tends to remain a bit private so I won’t mention his name but his pragmatic and honest approach to the realities of software and systems is worthy of praise. It was a pleasure working with him. The other two of that early mind-trust have moved on from Oracle to VMware, in one case, and start-up entrepreneurial efforts in the other.
All that aside, however, Ahud brings up the storage used for the 30 million SuperCluster TPCC which has very little to do with this post since that particular storage “solution” is not even a real product as much as a sort of DIY kit with servers, COMSTAR (http://hub.opensolaris.org/bin/view/Project+comstar/WebHome) and F5100s. I should probably blog that topic some time. The rules of TPCC state that you have to benchmark real product (or at least product that will become real by the stated system availability date). So if you pop open the FDR and get the line item for the storage (it’s over 20 million dollars btw) and decide to buy that exact “thing” I believe Oracle either has to sell it to you or withdraw the published result. This is just how TPC works and it is not big deal.
I have seen no evidence that the storage used in that SuperCluster TPCC will ever make it out the door as a genuine product. None of that matters though. TPC-C configs generally don’t look anything like a product intended for an IT shop. They really are just benchmark-special sort of configurations… at least at the highest nose-bleed level. That is why there are no Exadata TPC-C or TPC-H results. Products like Exadata don’t possess the arms-race characteristics these sorts of benchmarks tend to require.
NOTE: The SuperCluster page on oracle.com (http://www.oracle.com/us/corporate/press/192208) states that in CY2011 product would materialize that is “based upon the architecture used in Oracle’s new TPC-C world record.” Important to note, however, is that particular offering is *not* based on the storage used in the TPCC. Oracle announced a recipe that uses ZFS Storage Appliance as opposed to COMSTAR heads fronting Sun F5100 devices.
Kevin,
Brilliant post. My client was the very first Exadata customer in entire EMEA region. And there experience has not been very positive due to all the bugs and patching complexity. Now my client is thinking about Greenplum as well for their DW strategy. I guess, i better start learning Postgres now.
Competition is good for the IT industry. I won’t argue that. While I mentioned the EMC Greenplum product in this post as a way to address the Oracle Storage Strategy Update claims of scan rates in the 250GB/s range, I still want to drive home the fact that this specific post is about Oracle Database on EMC versus Exadata and most specifically OLTP as Oracle plugs in the 1.5 million RIOPS number which can only set the stage for an OLTP comparison.
I wish for success for your client.
Thanks for stopping by, Vishal.
Hi Kevin,
I agree that some of the points in the above slide are questionable, especially
a) ‘Database bandwidth 250 GB/s’ and
b) ‘Database cache usable capacity 53 TB’
When we look at the (quite good & fair, IMHO) whitepaper about x2-8
http://www.oracle.com/ocom/groups/public/@otn/documents/webcontent/173705.pdf
we say 25 GB/s uncompressed raw disk bandwidth. So a) can only be done with scanning compressed data, as you pointed out already, while b) is on the Database Layer, not on the Storage Layer.
I do not think, though, that it is so unfair to list metrics that apply to OLTP and DW on the same slide.
We have indeed these impressive IOPS metrics (1.5 Million) on Exadata as well as impressive compression rates (higher than on the slide even, typically), and some customers will well have hybrid Databases where both is relevant, so why not quote that?
The unfairness is maybe in the fact, that we just offer HCC only on Exadata, although it would have been technically possible to do HCC on any storage. Zero for EMC is also not fair, because customers could use non HCC compression on EMC to achieve more moderate compression rates like 4 times.
On the bottom line, I think it is true that the Exadata Oracle DB/Hardware combination delivers the best metrics for OLTP & DW – although not necessarily on the same system simultaneously – compared to any other Oracle DB/Hardware combination that uses similar hardware. It just doesn’t sound so impressive without numbers
Hi Uwe, please let me comment inline:
Uwe: “So a) can only be done with scanning compressed data, as you pointed out already,”
Kevin: I think you mean can only be done with uncompressed data
Uwe: “The unfairness is maybe in the fact, that we just offer HCC only on Exadata, although it would have been technically possible to do HCC on any storage. ”
Kevin: “technically possible” Uh, yes. Just ask all the participants in the 11g Beta program who graciously put Hybrid Columnar Compression through their testing..none of which had Exadata. HCC is an intrinsic database feature but Exadata happens to support decompression in storage. That fact has to do with a couple of your other points but I’ll take that up in a different blog post.
I meant the quoted rate of 250 GB/s is only possible when scanning compressed data but counting the uncompressed volume – which is of course a little flawed…
Uwe, that’s just an effective scan rate number. No worries.
So, how many RIOPS (of the 1.5 million maximum actual) do you suppose you can get when Exadata is also concurrently sustaining complex queries involving deeply compressed HCC data? I know the answer to that…just wondering if you do. The answer has a lot to do with my objection to Oracle marketing throwing OLTP and DW/BI value prop numbers in together as if they are concurrently sustainable. All systems have limitations…believe it or not!
Kevin, no I don’t know – let me guess: It is significantly less, right?
But that doesn’t make the two metrics on the slide wrong regarding IOPS and Compression Rate, achievable using Exadata Storage.
It is not said on the slide that you can get these metrics at the same time or even on the same system. To me, it seems that you just interpret that into it.
Uwe,
I partially agree with you. The problem is that most people (that includes Oracle employees) do not know which components play most importantly in the delivering of these datasheet numbers. Allow me to explain.
The Oracle datasheet number for RIOPS is 1.5 million full-rack. I do not argue that (and you know I don’t argue that). Furthermore, Oracle datasheet numbers are genuine db_file_sequential_read numbers as opposed to some synthetic I/O generator like ORION. And that is commendable. The problem is that going anywhere near 1.5 million RIOPS even driven from the lightest, most-synthetic simple SQL kit swamps the CPU in the database grid (give it a go…perform 16,000 physical reads per second database host core and smell the silicon burn). You can’t lean on that value prop while handling complex queries because, as too few people know, the most cpu-intensive aspects of SQL processing do not occur in Exadata storage cells.
Joins, sort, aggregation occurs in the database grid. I contend that you either get high IOPS or complex queries but neither perform up to expectations when run concurrently. It takes very little time to saturate storage cell CPU when querying against HCC data and with the new “passthru” functionality (feature?) there is a significant (up to about 40%) amount of the data that flows from CPU-bound cells up to the database grid without having been filtered or decompressed! That only further exacerbates the general shortage of CPU available in the solution for join, sort, aggregation effort.
Your tests probably don’t show you that yet. That’s why I posted on your blog some time back requesting that you focus more on complex queries than these little fully-offloaded scans everyone seems to blog about. Scanning disk for a non-existent needle in a haystack is not interesting anymore.
Kevin,
“That only further exacerbates the general shortage of CPU available in the solution for join, sort, aggregation effort.”
Are you now stating that exadata has insufficient cpu power in general, or only when you combine oltp workload with dwh workload?
Freek,
That question deserves an entire blog entry. Anyone who has sat through any of my lectures on Exadata knows how solidly I hound on the Data Flow Dynamics and what components are involved for what sort of workload. I have not “changed my story” as a result of leaving Oracle, nor will I. And when I say lecture, that includes the authors of all currently published Exadata books as I was the technical reviewer of each.
To answer your (good) question, I’ll say that I specifically wrote “general shortage of CPU available in the solution for join, sort, aggregation.” Consider the fact that the X2-2 has 96 WSM-EP cores in the database grid and 168 in the storage grid. That’s 36% of all CPU to do join/sort/aggregate but 64% of all CPU for IO/filtration/projection. That’s an Asymmetrical MPP model. Some queries will favor that balance, others won’t. For example, which of the following two queries do you think will prefer 36% CPU available for join/sort/agg?
Query A:
select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) from customer, orders, lineitem
where c_name like ‘Mickey Mouse’ and
o_orderkey in ( select l_orderkey from lineitem group by l_orderkey having sum(l_quantity) > 314 )
and c_custkey = o_custkey and o_orderkey = l_orderkey
group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice order by o_totalprice desc, o_orderdate;
Query B:
select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) from customer, orders, lineitem
where
o_orderkey in ( select l_orderkey from lineitem group by l_orderkey having sum(l_quantity) > 314 )
and c_custkey = o_custkey and o_orderkey = l_orderkey
group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice order by o_totalprice desc, o_orderdate;
Unless you have a lot of customers named Mickey Mouse there is no SQL processing work for the database grid in Query A. In that case, 96:168 might seem like an acceptable fit for CPU delineation…well, that is, until you consider the fact that during the entire query execution time of Query A you have 96 WSM-EP processors doing nothing for you. On the other hand, in the case of Query B you get some the same I/O and filtration/projection effort from the 168 CPUs in the Exadata storage cells plus additional overhead to prepare and send data from the filtration workers in storage over iDB (40 Gb IB) into the database hosts. That is, the 168 cores in storage will be a little busier than in Query A but they add no value for the join/sort/agg spelled out in the query.
So, I’ll answer your (good) question with a question. What do you think is more CPU intensive, performing a hash join or filtration (e.g., string compare “Mickey Mouse” != “Winnie The Pooh”) ?
Finally, the topic is complex and Hybrid Columnar Compression spins it on its head.
Kevin, I really appreciate your explanations!
You are of course right with your remark that the demos on my Blog are (over-)simplistic, compared to real-world scenarios. But that lies in the nature of my profession: I am an instructor, and my goal is (mostly) to explain concepts & techniques in an easy understandable way.
Reality is often too dirty & complex for this approach
But inspite of that nature of my postings & demos (I am aware of their limitations, believe me), I hope that they are useful for my audience. At least sometimes, I have this impression 
Hi Uwe,
I understand your position. When I urge you to blog about complex queries (instead of simple scans) I do have education in mind. Some folks want to learn how to tell time, others how the clock works and others still what the clock is made of.
Here’s the link of the webcast (which these slides belong):
http://bcove.me/lzhh01ib
At the question and answers part, the first question is about why to position Exadata against EMC Symmetrix (30:10)…
Thanks for the link, Gokhan.
Hi Kevin,
This is Tanveer. For last 1 year i’m following your blog to learn about Exadata. And truly speaking, i’ve learned a lot about Exadata from your blog. Thanks a lot..
Not to mention but I think i’m learning more from your blog since you left Oracle.
))…….Now the information flow is even more interesting…