I didn’t have the time to post this week. I was very busy doing a lot of stuff. Some of them are a post worthy though. On Thuesday almost my whole day was filled with the AMIS Academy day. During this event my AMIS colleagues or I present to our fellow colleagues and / or have the time to talk. A lot of us are consultants and this day is a chance to catch up again. The AMIS Academy is a free event that anyone can attend, also not AMIS people. During the day, free food is present, so I you have a chance in the future, take you pick and attend.
This mini conference had nine presentations in three parallel tracks. Some topics presented were SOA, BI, Oracle RDBMS, Excel, XML DB, Java Web Development. My first track was Sjoerd Michels presentation about the Oracle Web Services Manager and how it maybe could be used as a testing framework being the hub of al lot of processing. The second one was my colleague’s “Herman goes RMAN” about, you guessed it, how RMAN is deployed within the AMIS environment. My last session was one concerning how to use / use cases based on materialized views and it current functionality. The day ended at the AMIS bar around 21.00 hours.
On Wednesday I had to go to a customer with database problems. The database was not really corrupt database wise, but the web application that has its metadata in tables (one of these famous “database independent” applications), made it possible to corrupt its own metadata content / tables via one of its screens… As we discovered, it was possible via a simple web screen to drop a column of one of its own tables, this resulted in “security” errors in the web pages. Then of course, Murphy came along…
The support department of this web application firm had the brilliant solution: “restore the database, then it will be all fine again”. No attempt to restore the situation to correct it via DDL or DML statements. Just: “restore the database…”. This document store application from Canada had store approximately 80 Gb of data in the form off Word, Pdf etc in the Oracle database. At hand I had a full dumpfile (logical failsave), a full backup set made with RMAN (spfile, datafiles, archives, controlfiles) and, as we found out, also still a Oracle Veritas backupexec process was running during the night. Physical tape backups were made on a different location, 15 kilometers from the database / server room (although with a dedicated line, still not as fast as I wanted to transport backup data).
Because we were fiddling around with free space for over a year now on our SAN, we decided over the months to make less and less redundant backups. What was almost fatal was that we didn’t test in between if our startup situation was still provicient regarding our initial restore tests. As I found out, I ended up with controlfiles with not enough information about the backups made over the week. I had a complete RMAN backupset, but no way to get the datafiles out of the backupset. It would always ask for the current backupset instead of the backupset I wanted to restore. After 5 hours hacking and trying to find a solution, I just renamed my backupset for the backupset it wanted to see.
To my surprise, this worked, and RMAN restored happily my 90 Gb of datafiles I wanted from the day before, despite it did it with the incorrect file. After this (because the files were consistent) the restore went in a breeze. We had even time to check what went wrong, by destroying / corrupting the database logically a second time while I put all important sessions in trace so we could deduct from the database, what actually dropped the column. The two tracefiles were analyzed via TKProf and send via a commentary e-mail to the Canadians…
The last two days (Thursday / Friday), I had to setup a performance test environment to see what actually happens under test strain with an Oracle Application Server (Java Edition), containing a ADF application. To perform this test, Apache JMeter was used to simulate concurrent user sessions. On my database server, situated in a SUSE 9.3 E. Server, I installed as an extra, the IBM NMON tool. With this tool it is very easy to get a global glance to see what is happening on the server. We didn’t end up yet with some results, but we ended up on Friday with some better set-up of the situation we wanted to simulate: a ADF application under real stress with 2000 sessions banging on the Oracle Application Server and the result, among others, on the database.
The first indications gave me the idea, that I had a typical “Anjo Kolk” situation as once presented by him during and AMIS Query. It looked like, while increasing the data sources / JDBC connections that the database had so many things to do during, probably the same actions (our test set-up wasn’t really random yet regarding difference in processing), that the Linux machine completely went CPU bound trying to setup data source, that the database wasn’t able to service anymore. Anyway, this one is still going, as on Monday we will go on with our testing.
In between all this I got an e-mail from Anjo Kolk regarding the Miracle Express, for the dutch guys, who will attend Miracle Open World in Lalandia, Denmark, next week. Anjo arranged some small coaches to drive the lot of us to Denmark, already lovingly called “The Miracle Express”. As was told by Carel Jan Engel two weeks ago during a meeting, he will drive one of these small coaches and maybe the other one by Anjo?. I hope we will attend the seminar in Lalandia because it was announced that these coaches would have some specimen of the famous Miracle liquids on board. I hope I can write some onsite blog posts on the AMIS Technology Blog, starting next week Thursday.
😉
