CastleCops® Executive summary 21 Feb 2008

Need help? Click here to register for free! Absolutely zero advertisements on this site!

**$9736.22 of $21422.68**

Help CastleCops serve the community on new servers, Donate Here to reach our goal.

	Donation/Premium

	Donations. Become Premium Today!

	Security Central

	· Home · PIRT/Fried Phish · MIRT · SIRT · Deutsch · Wiki · Newsletter · O16/ActiveX · CLSID List · Contest2007 · Downloads · Feedback (send) · Forums · HijackThis · Hijacktrend · LSPs · My Downloads · O18 · O20 · O21 · O22 · O23 · O9 · Premium · Private Messages · Proxomitron · Reviews · Search · StartupList · Stories Archive · Submit News · WsIRT · Your Account · Acceptable Use Policy

Survey

Forum FAQ

Usergroups

Profile

Select FavForums

Executive summary 21 Feb 2008

All -> FavForums -> perf

[del.icio.us!] [digg it!] [reddit!]

View previous topic :: View next topic

Author

Message

Paul

CastleCops Founder

Joined: Feb 22, 2002
Posts: 27351

Posted: Thu Feb 21, 2008 4:08 pm Post subject: Executive summary 21 Feb 2008

As there are ten pages to the original 'Optimizing Performance' thread, I figured lets create a summary for folks when they are needed.

In a nutshell, I'm not sure what exactly the problem is. But, here are some details:

Latest mysql, apache, php, and freebsd (not 7). 8 GB RAM, half a terabyte free disk space raid.

top

Code:

last pid: 70194; load averages: 16.77, 19.57, 27.66
248 processes: 19 running, 227 sleeping, 1 zombie, 1 lock
CPU states: 74.9% user, 1.5% nice, 19.5% system, 4.1% interrupt, 0.0% idle
Mem: 2933M Active, 2233M Inact, 388M Wired, 333M Cache, 214M Buf, 1795M Free
Swap: 16G Total, 4K Used, 16G Free

PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
40376 mysql 77 20 0 1247M 1243M kserel 337:48 13.18% mysqld
69807 www 1 114 0 198M 110M RUN 0:21 2.93% httpd
69366 www 1 114 0 238M 130M RUN 0:35 2.83% httpd
69975 www 1 114 0 238M 110M RUN 0:14 2.83% httpd
69443 www 1 113 0 238M 111M RUN 0:32 2.69% httpd
69235 www 1 113 0 238M 110M RUN 0:30 2.59% httpd

iostat -xC

Code:

extended device statistics cpu
device r/s w/s kr/s kw/s wait svc_t %b us ni sy in id
da0 1.0 6.7 9.2 237.5 0 51.7 3 73 0 17 2 7
da1 6.1 17.9 305.6 691.4 0 344.9 13
pass0 0.0 0.0 0.0 0.0 0 0.0 0
pass1 0.0 0.0 0.0 0.0 0 0.0 0

vmstat

Code:

procs memory page disks faults cpu
r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id
20 2 2 3232812 2378844 10400 0 0 0 10492 80 0 0 3520 27480 8256 73 19 7

my.cnf

Code:

[mysqld]
#___KEY BUFFER___#
key_buffer = 1G
key_cache_division_limit = 100
#___VARIOUS BUFFERS___#
myisam_sort_buffer_size = 64M
join_buffer_size = 2M
read_buffer_size = 2M
sort_buffer_size = 5M
read_rnd_buffer_size = 2M
#___THREAD AND TABLE CACHES___#
table_open_cache = 1024
thread_cache_size = 200
tmp_table_size = 25M
max_heap_table_size = 25M
table_definition_cache = 1024
#___QUERY CACHE___#
query_cache_type = 1
query_alloc_block_size = 16K
query_prealloc_size = 8K
query_cache_limit = 2M
query_cache_size = 128M
query_cache_min_res_unit = 1K
#___TIMEOUTS___#
wait_timeout = 100
interactive_timeout = 100
#___LOGS___#
log = 0
log_warnings = 0
general_log = 0
log_output = FILE
slow_query_log = 1
log_slow_queries = 1
log_queries_not_using_indexes = 1
#___FORCE MYSQL TO USE INDEX___#
max_seeks_for_key = 1000
#___MISC___#
socket = /tmp/mysql.sock
server-id = 1
max_connections = 200
concurrent_insert = 2
low_priority_updates = 1
thread_concurrency = 2
max_allowed_packet = 16M
myisam-recover = BACKUP,FORCE
skip-locking
skip-innodb

Standard httpd.conf settings. php.ini mysqld persistency is off.

mytop

Code:

MySQL on localhost (5.1.23-rc-log)
Queries: 92.9M qps: 1315 Slow: 0.0 Se/In/Up/De(%): 92/00/00/00
qps now: 2069 Slow qps: 0.0 Threads: 17 ( 4/ 56) 97/00/00/00
Key Efficiency: 99.4% Bps in/out: 0.7/128.3 Now in/out: 62.1/ 5.5k

Load has risen to 50s. I've only found one zombie process, and that looks like its due to load.

Some forum sql queries I've re-written, and forum thread views have been adjusted in how they work to help offset access to critical tables. Additionally, some further enhancement to IRT code done as well.

As slow query logging is enabled, I tried to "fix" some code. But their run time is extremely fast anyway, because I'm also logging queries that run without using index. I may disable that, and just monitor slow queries instead of both.

Forum search has temporarily been disabled.

Even with the consitent high server loads, the site is fairly traversable.

Paul

CastleCops Founder

Joined: Feb 22, 2002
Posts: 27351

Posted: Thu Feb 21, 2008 4:11 pm Post subject:

Forgot to mention above that mysql-server was built with these settings: 'BUILD_OPTIMIZED=yes BUILD_STATIC=yes' The server is also a 64 bit system, dual proc single core AMD. _________________ Paul Laudanski - http://www.laudanski.com http://www.linkedin.com/pub/1/49a/17b

Paul

CastleCops Founder

Joined: Feb 22, 2002
Posts: 27351

Posted: Thu Feb 21, 2008 4:46 pm Post subject:

Code:

tw_cli /c0 show

Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy
------------------------------------------------------------------------------
u0 RAID-1 OK - - - 372.519 ON OFF
u1 RAID-1 OK - - - 372.519 ON OFF

Port Status Unit Size Blocks Serial
---------------------------------------------------------------
p0 OK u0 372.61 GB 781422768 5QH0A8T9
p1 OK u0 372.61 GB 781422768 5QH09GK7
p2 OK u1 372.61 GB 781422768 5QH09F30
p3 OK u1 372.61 GB 781422768 5QH09WWA

_________________
Paul Laudanski - http://www.laudanski.com
http://www.linkedin.com/pub/1/49a/17b

AlphaCentauri

SIRT Handler
Premium Member

Joined: Nov 20, 2003
Posts: 2705

Posted: Thu Feb 21, 2008 4:47 pm Post subject:

It's very speedy for me right now. Last night there was a period (maybe around 10pm -500) when it would not load at all and I got a error message from my browser about not being able to connect. When that stopped, it was back to speedy. At the time, I was able to log in with Firefox but not Seamonkey, and it may have just been the site was starting to come back and working intermittently, or it may have had something to do with the log in function. My password is already loaded in Seamonkey so I automatically go to the forum page. With Firefox I only got to the log in page, and by the time I looked up my password, both browsers were working, so I couldn't tell if Firefox would also have hung after I logged in.

Bill_Bright

General

Premium Member

Joined: Jan 16, 2004
Posts: 8930
Location: Nebraska, USA

Posted: Fri Feb 22, 2008 12:37 pm Post subject:

There are still some inconsistencies, but overall there is much improvement. Beyond that, I appreciate very much the openness of these discussions - obviously, bringing this to the forums for open discussion has resulted in a learning process for those following and participated, but more importantly, the site "feels" alive again! _________________ Bill (AFE7Ret) Freedom is NOT Free!

Trpm

Security Expert
Premium Member

Joined: Jan 16, 2004
Posts: 1663

Posted: Fri Feb 22, 2008 3:35 pm Post subject:

Really nice to see you in here again Bill, no sh*t you have been missed. Paul remember your weakest points are the slowest and most used parts of the system, hard drives and front side buss. Rex _________________ I'm a reasonable person just ask Eeyore or Christopher R.

Bill_Bright

General

Premium Member

Joined: Jan 16, 2004
Posts: 8930
Location: Nebraska, USA

Posted: Fri Feb 22, 2008 4:42 pm Post subject:

I've been around, just not as visible, and yes, much I miss too. But thanks. _________________ Bill (AFE7Ret) Freedom is NOT Free!

Trpm

Security Expert
Premium Member

Joined: Jan 16, 2004
Posts: 1663

Posted: Fri Feb 22, 2008 4:47 pm Post subject:

Rex _________________ I'm a reasonable person just ask Eeyore or Christopher R.

j79zlr

Security Expert

Joined: Jul 16, 2004
Posts: 971

Posted: Sat Feb 23, 2008 1:30 am Post subject:

I was wondering what had been going on here. It seems like for the last month everytime I've tried to visit here it takes minutes for pages to load. _________________

johnlgalt

Special Response Team
Premium Member

Joined: Feb 27, 2007
Posts: 1410

Posted: Sat Feb 23, 2008 1:46 am Post subject:

The system is definitely much much faster - the fastest I have ever seen it since I started hanging out here regularly. I, too appreciate the openness in what has been going on, and more so from the perspective that you are not only keeping the constituents in the loops, but that anyone else that in the future might start running into issues such as your own will now have a repository to read through to garner their own ideas on site improvement. _________________ <img src="http://www.castlecops.com/zx/johnlgalt/johnlgalt%20sig.png"> <img src="http://www.castlecops.com/zx/johnlgalt/John%20L.%20Galt%20%20CPU-Z.png">

	All -> FavForums -> perf	All times are GMT
Page 1 of 1

You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


	2002-2008 � Computer Cops LLC dba CastleCops®. All rights reserved. Acceptable Use Policy. Use signifies your agreement. 130ppm 0.465s (0.112s) Making cybercriminals unhappy since 2002*