PDA

View Full Version : CSC Server Performance *** UPDATE: Problem Linked to Google Chrome



DevilsTonic
Wed Jul 27th, 2011, 09:03 PM
ADMIN EDIT: This only seems to be a problem with Google Chrome. For instructions on how to fix it, scroll down to post #25 or click here (http://www.cosportbikeclub.org/forums/showthread.php?p=593634#post593634).


Ralph / Bob, I know that you guys have many more important things to worry about with your real jobs. I just want to put this out here as I'm sure many people are experiencing the same delays in performing the simplest of tasks on CSC. This only started after the site was moved to the new servers. I'm not sure if it's a hard coded speed/duplex setting on the new hardware or maybe latency with the database.

I'm seeing delays in server response when clicking to view a thread, reply to a thread or to create a new thread. It took over 20 seconds after clicking the "New Thread" button to create this thread, I got no response and clicked it several more times before I was directed to the new thread page.

I've also noticed that there are many double posts. I'm not sure if that is because people are moderating their own posts (doubtful) or because they post, the server response is too slow so they attempt to post again and end up with a duplicate.

In running a continuous ping to the server I'm seeing inconsistent response times ranging from 9ms-200ms. I know that you are doing this on your own time, and I thank you for all the time and effort you put into keeping CSC running. I just wanted to make you aware in case this is something you may want to look into....in your copious three seconds of spare time of course!

Ricky
Wed Jul 27th, 2011, 09:09 PM
Problem: server connected to cisco switch port that is set to negotiate at a specific speed, and the network card in the server is set to auto negotiate.

Solution: set server NIC to negotiate at the same speed that the switch port is set to

TurboGizzmo
Wed Jul 27th, 2011, 09:15 PM
Problem: server connected to cisco switch port that is set to negotiate at a specific speed, and the network card in the server is set to auto negotiate.

Solution: set server NIC to negotiate at the same speed that the switch port is set to

Ha! Thats the first thing the network engineer consultant said to me (personal network issues)....most of network issues he deals with is a speed/duplex/negotiation issue.

PS Editing this post took 664ms for editpost.php to load

Vellos
Thu Jul 28th, 2011, 09:06 AM
For me when the page lags it will not load even if I try waiting for it. I have to keep trying and 1 out of 10 times it will instantly load.

CaptGoodvibes
Thu Jul 28th, 2011, 09:46 AM
60 percent of the time it works everytime

http://www.youtube.com/watch?v=zLq2-uZd5LY

grim
Thu Jul 28th, 2011, 10:19 AM
60 percent of the time it works everytime

http://www.youtube.com/watch?v=zLq2-uZd5LY

Where is the cookie? and i didnt get the invite to the pants party!

rforsythe
Thu Jul 28th, 2011, 11:47 AM
Problem: server connected to cisco switch port that is set to negotiate at a specific speed, and the network card in the server is set to auto negotiate.

Solution: set server NIC to negotiate at the same speed that the switch port is set to

Negative. Server is connected to a gigabit switch and everything is negotiated properly, and in any event the virtual machine running the web server talks to the VM running the database through a virtual switch where duplex doesn't really exist. That doesn't affect the ISP link however.

Folks I know it's annoying, and I will get it resolved as soon as I can. We are on a better server, but the ISP not so much as that got changed as well. The dupe posts are probably from people clicking submit multiple times waiting for reloads, though oddly enough I don't see these issues myself when I'm remote.

t_jolt
Thu Jul 28th, 2011, 11:59 AM
Ralph i know you've probably already looked, but could it be blocking on the Database? or disk queues? I know those disks are way faster then the old ones you were using, but maybe its a thought? or could their be proc contention with the vm's?

Tyrel

Zach929rr
Thu Jul 28th, 2011, 12:01 PM
I have experienced no problems at all.

King Nothing
Thu Jul 28th, 2011, 12:08 PM
waaaaaaaaaaaaaa It takes like a second for the threads to load. waaaaaaaaaaaa :cry::cry::cry::cry:

people will bitch about anything.

Sean
Thu Jul 28th, 2011, 12:10 PM
I have experienced no problems at all.That's because you're mentally slow. Your thought process and the servers are now in sync.

Joe
Thu Jul 28th, 2011, 12:50 PM
Since I don't understand any of this, but I'm very grateful for them...

http://nerdarama.com/wp-content/uploads/2009/02/ogre.jpg

DevilsTonic
Thu Jul 28th, 2011, 01:21 PM
waaaaaaaaaaaaaa It takes like a second for the threads to load. waaaaaaaaaaaa :cry::cry::cry::cry:

people will bitch about anything.No bitching here, just making a site comment.

~Barn~
Thu Jul 28th, 2011, 03:08 PM
I have experienced no problems at all.

+1

Ricky
Thu Jul 28th, 2011, 04:15 PM
Negative. Server is connected to a gigabit switch and everything is negotiated properly, and in any event the virtual machine running the web server talks to the VM running the database through a virtual switch where duplex doesn't really exist. That doesn't affect the ISP link however.

Folks I know it's annoying, and I will get it resolved as soon as I can. We are on a better server, but the ISP not so much as that got changed as well. The dupe posts are probably from people clicking submit multiple times waiting for reloads, though oddly enough I don't see these issues myself when I'm remote.

Strange... This is identical to issues I've seen on other web servers that had to do with port negotiation. But yeah, if it's virtual, that makes it a different ballgame. Some people notice it, some don't. Jitter is massive when pinging the server. It was never this way before the switch. It's so bad for me, that each time I click, I usually have to click 5-10 more times for the page to load. Otherwise, it times out completely.


waaaaaaaaaaaaaa It takes like a second for the threads to load. waaaaaaaaaaaa :cry::cry::cry::cry:

people will bitch about anything.

And yet, here you are bitching about people bitching about something. Thanks for proving your own point. Holy fucking hypocrisy.

gtn
Thu Jul 28th, 2011, 04:55 PM
I haven't experienced any of the issues discussed above.

I just ran a set of 100 pings, and the hi/avg/lo were 129/92/87. No packets were lost.

Seems more like a peering issue with the ISP.

I have Comcast for an ISP at home. Sprint and Verizon at work. Though I haven't run the ping test from work, I do occasionally sneak a peak at CSC from there and haven't had any issues since the move.

LivingPools
Thu Jul 28th, 2011, 05:42 PM
No problems here, and I really appreciate the software upgrade to 'New Posts' functionality!

DevilsTonic
Fri Jul 29th, 2011, 07:01 AM
You may be onto something Ralph. Since you mentioned the browsers I started testing it in various scenarios. It just seems to be Chrome both in OSX and in Windows. I'm connecting to the site with Safari right now and I've tested with IE and Firefox and I haven't experienced the same delays. At least I can get my fix. That's all that matters! Thanks Ralph.

rforsythe
Fri Jul 29th, 2011, 09:07 AM
This seems to be a caching issue with Chrome specifically, and not unheard of on the Internet. My guess is Chrome is trying to remember some old location for files, and it can't find it there so it all but stops. For those of you using it, please clear all cache and page history out of it and see what happens.

Tylar
Fri Jul 29th, 2011, 09:49 AM
This seems to be a caching issue with Chrome specifically, and not unheard of on the Internet. My guess is Chrome is trying to remember some old location for files, and it can't find it there so it all but stops. For those of you using it, please clear all cache and page history out of it and see what happens.

Done that a few times, no joy.

I'm just sticking to Safari when I need to read about drama....because I just have to have it.

rforsythe
Fri Jul 29th, 2011, 09:57 AM
Done that a few times, no joy.

I'm just sticking to Safari when I need to read about drama....because I just have to have it.

Yeah, according to one article, clicking delete doesn't actually do it - you have to go manually nuke the files. Google it if you care that much.

cptschlongenheimer
Fri Jul 29th, 2011, 09:58 AM
This seems to be a caching issue with Chrome specifically, and not unheard of on the Internet. My guess is Chrome is trying to remember some old location for files, and it can't find it there so it all but stops. For those of you using it, please clear all cache and page history out of it and see what happens.

Did these. Helped at first but after closing a few threads and reopening I started seeing timeouts again. Read a tech bulletin that said DNS pre-fetching in chrome may be the issue. Trying now with that turned off. So far, all is good?

Tylar
Fri Jul 29th, 2011, 10:01 AM
Yeah, according to one article, clicking delete doesn't actually do it - you have to go manually nuke the files. Google it if you care that much.

Google is starting to act like the collective...


Did these. Helped at first but after closing a few threads and reopening I started seeing timeouts again. Read a tech bulletin that said DNS pre-fetching in chrome may be the issue. Trying now with that turned off. So far, all is good?

I'll try this. Two browsers for the single task of avoiding work is unacceptable.

cptschlongenheimer
Fri Jul 29th, 2011, 10:23 AM
Google is starting to act like the collective...
I'll try this. Two browsers for the single task of avoiding work is unacceptable.

Agree on both. :D

Since I've changed this setting, no more timeouts :) but I still cannot upload an attachment from chrome or IE

How to:
Click the wrench
options
under the hood
uncheck the box next to "Predict network actions to improve page load performance"
close tab
you may still have to flush the cache afterwards

rforsythe
Fri Jul 29th, 2011, 10:28 AM
I'll see if there are weird DNS settings in play that might be affecting the prefetch on my end. I can't imagine why there would be, but you never know.

Attachments are still broken. I've had a few more pressing things to deal with.

cptschlongenheimer
Fri Jul 29th, 2011, 10:36 AM
No worries. Not pressuring ya, Ralph. Just trying to give ya helpful info.

0N1X
Fri Jul 29th, 2011, 05:42 PM
Attachments are still broken. I've had a few more pressing things to deal with.

For me, performance is top notch, but sometimes (or more than sometimes) I can't view attachment images. Both IE and Firefox.

DevilsTonic
Sat Jul 30th, 2011, 08:30 AM
Modifying the prefetch settings in Chrome (OSX) seems to have remedied the issues I was seeing.

vort3xr6
Mon Aug 22nd, 2011, 04:14 PM
Working fantastic on Netscape Navigator over here.

Sean
Tue Aug 23rd, 2011, 07:47 AM
Thanks for the Chrome update, that works much better for me now. :up:

bulldog
Tue Aug 23rd, 2011, 08:05 AM
I think the problem here is all the porn DevilsTonic downloads; virus! :lol:

bulldog
Tue Aug 23rd, 2011, 08:15 AM
And solution to Chrome issue can be found here
http://www.cosportbikeclub.org/forums/showthread.php?t=42019

Mother Goose
Tue Aug 23rd, 2011, 08:15 AM
I have issues w/ Firefox sometimes. It just sits there and thinks about opening the page. I'll right click the thread and open in a new tab and it opens right up. Weird.

nextraztus
Thu Aug 25th, 2011, 06:11 PM
For the record, this option disables the phantom domain name lookups that Chrome does in the background. Basically, what happens under the hood if you dig around is that you get a bunch of requests something like this:


------------------------------------------
CONNECT_JOB (id=71154) [start=Thu Aug 25 2011 17:55:46 GMT-0600 (Mountain Daylight Time)]
------------------------------------------
t=1314316546660 [st=0] +SOCKET_POOL_CONNECT_JOB [dt=4]
t=1314316546660 [st=0] +SOCKET_POOL_CONNECT_JOB_CONNECT [dt=4]
--> group_name = "cpngackimfmofbokmjmljamhdncknpmg:65535"
t=1314316546660 [st=0] HOST_RESOLVER_IMPL [dt=4]
--> source_dependency = {"id":71155,"type":7}
t=1314316546664 [st=4] -SOCKET_POOL_CONNECT_JOB_CONNECT
--> net_error = -105 (NAME_NOT_RESOLVED)
t=1314316546664 [st=4] -SOCKET_POOL_CONNECT_JOB

------------------------------------------
HOST_RESOLVER_IMPL_REQUEST (id=71155) [start=Thu Aug 25 2011 17:55:46 GMT-0600 (Mountain Daylight Time)]
------------------------------------------
t=1314316546660 [st=0] +HOST_RESOLVER_IMPL_REQUEST [dt=4]
--> address_family = 0
--> allow_cached_response = true
--> host = "cpngackimfmofbokmjmljamhdncknpmg:65535"
--> is_speculative = false
--> only_use_cached_response = false
--> priority = 0
--> source_dependency = {"id":71154,"type":4}
t=1314316546660 [st=0] HOST_RESOLVER_IMPL_CREATE_JOB
t=1314316546660 [st=0] HOST_RESOLVER_IMPL_JOB_ATTACH [dt=4]
--> source_dependency = {"id":71156,"type":8}
t=1314316546664 [st=4] -HOST_RESOLVER_IMPL_REQUEST
--> net_error = -105 (NAME_NOT_RESOLVED)
--> os_error = 11004
--> os_error_string = "The requested name is valid, but no data of the requested type was found.\r\n"

------------------------------------------
HOST_RESOLVER_IMPL_JOB (id=71156) [start=Thu Aug 25 2011 17:55:46 GMT-0600 (Mountain Daylight Time)]
------------------------------------------
t=1314316546660 [st=0] +HOST_RESOLVER_IMPL_JOB [dt=4]
--> host = "cpngackimfmofbokmjmljamhdncknpmg"
--> source_dependency = {"id":71155,"type":7}
t=1314316546660 [st=0] HOST_RESOLVER_IMPL_ATTEMPT_STARTED
--> attempt_number = 1
t=1314316546661 [st=1] HOST_RESOLVER_IMPL_ATTEMPT_FINISHED
--> attempt_number = 1
--> net_error = -105 (NAME_NOT_RESOLVED)
--> os_error = 11004
--> os_error_string = "The requested name is valid, but no data of the requested type was found.\r\n"
t=1314316546664 [st=4] -HOST_RESOLVER_IMPL_JOB
--> net_error = -105 (NAME_NOT_RESOLVED)
--> os_error = 11004
--> os_error_string = "The requested name is valid, but no data of the requested type was found.\r\n"

It's not that Google is acting like the collective here, it's that most ISP's try to be 'helpful' and watch your DNS queries. When you go to a site that doesn't exist (usually a typo) they have "partnered" with some search company that they then happily forward your request to as a query. This company then turns your misspelling into a nice ad-filled landing page with your ISP's logo on it.

When Chrome does these requests first, it's EXPECTING garbage/failures to come back. If it gets a "real" IP address it knows that your ISP's DNS cannot be trusted. I'm not sure what it does at that point, I haven't reviewed that chunk of source code. Usually this happens very very quickly. According to my tools the CSC DNS server handles a lookup in about 50ms on average, my ISP usually handles it in about the same. The internal resolver on your computer is practically instantaneous. Granted, there is a lot of caching and things that go on in this process, but it's still very fast for non-authoritive lookups.

I'm still determining if I'm having page load issues after changing this setting. I'm going to keep hunting for the root cause of this because no other site I frequent has this problem (and that's a LOT of sites and I use Chrome almost exclusively at work/home/play/etc). This could be some obscure bug that needs to be reported to the Chromium team.

nextraztus
Mon Sep 5th, 2011, 09:17 PM
http://code.google.com/p/chromium/issues/detail?id=95461

I've given up. Passing it along to the Chromium Devs.

~Barn~
Mon Sep 5th, 2011, 09:31 PM
Nice work, Nex. :up:

rforsythe
Mon Sep 12th, 2011, 06:33 AM
Nex please do let me know what you find out. DNS has not yet been moved, so the only thing that changed was the A record when the web server was relocated. I see both my pri/sec DNS servers are consistent, so I'm really not sure what is causing this either.

For you non-Chrome users, does the site seem a little faster today? I did some optimization over the weekend.

zPurpleRoom
Mon Sep 12th, 2011, 12:08 PM
Yes....the site is faster. Thank you. I was just thinking to myself that the site was faster when I read your post....lol. Great job! :up: