If you have Oracle Database deployed on Linux and have not yet settled upon collectl as your primary tool for system-level performance data collection, well, we need to have a long talk!
Disclaimer
The views expressed on this blog are my own and do not reflect the views of Oracle Corporation.
The views and opinions expressed by visitors on this blog are theirs, not mine.
Recent Posts
- Oracle Exadata Storage Server Technical Deep Dive Series – Part II: Requires a Citrix CODEC.
- Oracle Exadata Storage Server Architecture: Impossible To Back Up?
- Oracle Database File System (DBFS) on Exadata Storage Server. Hidden Content?
- Aren’t Customers Choosing Oracle Database Machine?
- Recorded Webcast Available: Exadata Storage Server Technical Deep Dive – Part IV.
- Oracle-Enhancing Solaris Features. Memory Lane.
- Staging Data For ETL/ELT? Flat Files Appear Magically! No, Load Time Starts With Transfer Time.
- Webcast Announcement Clarification. Exadata Technical Deep Dive Part IV.
- Oracle Data Warehouse Performance Issues? Solve It The Old-Fashioned Way With A Third-Party Accelerator!
- Oracle Exadata Storage Server Technical Deep Dive – Part IV
Recent Comments
Blogroll
All information and materials provided here are provided "as-is"; Oracle disclaims all express and implied warranties, including, the implied warranties of merchantability or fitness for a particular use. Oracle shall not be liable for any damages, including, direct, indirect, incidental, special or consequential damages for loss of profits, revenue, data or data use, incurred by you or any third party in connection with the use of this information or these materials.
Thanks for the tip. Any comments about why it is better then, for example, using NMON ?
Forget it
As said, thanks for the tip and forget the stupid question about nmon.
Kevin, do you have any clue/idea how to monitor on Linux
1) read(2)/write(2) avg syscall time per running process (I/O completition, or lets better say to VM – yup, without O_DIRECT)
2) how to measure seqential vs random ratio for process/disk ?
3) VM ratio hit?
E.g. i could use something similliar to “strace -r -s 0 -e read,write -p ” for point 1 but would like it to being less intrusive.
For second, is there is any other option on Linux that writing dedicated ptrace(2)-based utility tracing lseek(2),read(2),write(2) ??
3) Any idea??????
Jakub,
Perhaps you might investigate OProfile? Strace single-steps processes so it is worthless in this regard.
Jakub – This probably isn’t exactly what you’re looking for, but a feature of collectl that I don’t think has gotten a lot of use its ability to show I/O by process, assuming of course you have a kernel > 2.6.22 that was also built with process accounting enabled. See http://collectl.sourceforge.net/Process.html for more details.
The reason I bring it up is you’ll at least be able to see a processes I/O rates as well as the number of I/O systems calls its doing if you select the I/O summary format.
And of course you can see it at the individual thread level as well if you so choose by including –procopts t.
-mark
Kevin,
Regarding single-stepping, isn’t the strace system call tracing done via trapping the syscall entry and return only and not via single-stepping the whole execution path?
Tanel.
…I’m talking about ptrace() PTRACE_SYSCALL option here…
Tanel, I think you are right. I wonder if my wires were crossed between it and truss (or if even truss does that)…hmmm…I’ll have to investigate and clarify. I’ve got single-stepping in my mind somehow…need to figure out why
You’ve been debugging too much lately?
I think truss doesn’t do single-stepping either, at least in Solaris, as it uses /proc/pid/ctl interface which supports setting breakpoint traps at specific system calls (there are ctl flags PCSENTRY and PCSEXIT).
Nevertheless, stracing / trussing critical (background) processes in production should require careful thought first as you can cause trouble to traced process (missed signals, failed IOs) as I’ve once evidenced here: http://blog.tanelpoder.com/2008/06/14/debugger-dangers/
Jakub, in addition to OProfile, if you’re on recent Linux (RHEL 5+ equivalent) then you can look into SystemTap. It is getting mature enough now for professional (non-hacker) use – on my CentOS 5.2 install it’s kernel and userland parts are preinstalled so I didn’t need to start patching and recompiling kernel (for which I just don’t have the time and will).
I’ve just been doing too much. Period.
Why don’t use SAR that is standard in all unices and well know to all system admins ?
mdx4 – I guess you still don’t get it and maybe won’t until you try collectl out and see for yourself. Can sar produce data in a format you can easily plot? Can sar collect data on infiniband? slabs? and other ‘less common but still critical’ types of data? can sar run at sub-second monitoring levels? can sar provide much of its output on a single line for easy reading? can you easily feed sar data to other programs over a socket in real-time? far too many other things to list. I think sar is fine at what it does, but there is so much more it doesn’t do and that’s why I wrote collectl.
-mark
Ok, thanks for valuable hints. I think that SystemTap is the long time solution, but I wanted to look at old systems too. It even seems that Oracle wrote some part of public SystemTap scripts
I’ve started developing ptrace(2) (lseek/read/write, pwrite is currently missing) based fileops aggregation utility to trace I/O characteristic on fds , when I’ve hit strange issues in production (as Tanel mentioned) for Java apps, one even DoSed my whole system. Maybe sometime I’ll have time to finish & release it.
However I still get no idea how to calculate VM ratio hit over time…
I recently discovered /proc/buddyinfo, which shows how memory is fragmented, thought it useful and added it to collectly so now you can literally see in real-time how many fragments of what sizes are available. However I don’t have a real good feel for how much extra overhead is being generated by having too many small fragments and not enough larger one. Could it make a bigger difference for heavy data movers like oracle? Is this new look at memory fragmentation of use to anyone?
-mark
I wasn’t sure where to post this to I thought I’d tack it onto the end of this string.
I just released a new version of collectl that might even get Kevin’s attention because it now has a data collectl API. This means if there are any readily available Oracle specific stats, it shouldn’t be too tough (if you’re a perl programmer) to add them into collectl.
Why would you want to do this? That’s easy – imagine looking at some oracle data and cpu/disk/interrupt/ctx/etc data ALL at the same time either on your terminal or maybe even in a plot – yes, this mechanism allows your custom data to be recorded in plot format too.
You’re seeing Oracle I/O rates drop? Now you can watch linux stats and see if any resources are starved and causing your problems. Of course you could still do that without this feature but just think how much harder it would be.
-mark
Hi Mark,
Well, you know I’m a huge fan…didn’t know if you were aware of this or not, but collectl made the grade for inclusion into Exadata packaging. You really produced a winner with collectl!