Do Not Panic

By Shamus Posted Sunday Mar 3, 2013

Filed under: Notices 169 comments

Yes, the site theme has changed. No, this is not a permanent change. I’m just mucking around trying to figure out why the site is so ridiculously slow. I’ve fussed with the database, I’ve turned off all my plugins, I’ve checked the error logs, I’ve scoured the WordPress help pages, and I’ve said swear words in a loud, authoritative voice. I’m basically out of ideas at this point.

SOMETHING is making it incredibly expensive to show pages. It gets bad when lots of people are commenting. It gets horrible to the point of uselessness when I add a new post. We’ll see what this does.

EDIT: Initial test: Inconclusive. I’m going to let this sit for couple of hours and watch performance as comments appear and I edit posts.

EDIT: Well, I stripped the site down to the bones. Default theme, no plugings. (And then, just plugins to rebuff spam.) The site loaded fine, but posting comments still takes over a minute. Actually, the comment goes up right away, but the person leaving the comment gets stuck waiting for the page to refresh. The same thing happens when I dd a new post or update an old one. Basically, submitting a form results in an epic page load for no reason in the world. It could be the form. It could be modifying the database. It could be rebuilding the affected pages. I don’t know.

There are 235,000+ comments on the site. That’s a lot, but in the grand scheme of things they shouldn’t clog up the works that badly.

I really am at a loss. There’s no reason for this. I contacted my web host and the machine itself seems fine. No rogue processes eating CPU/memory/bandwidth. It’s just another stupid WordPress blog.

EDIT: On the advice of the tech support guru at my host, I dropped the Super Cache plugin and switched to W3 Total Cache. The difference is pretty stark. I’m hopping around the admin pages with a speed I haven’t seen in over a year. Comments seem to be working well. We’ll see how this goes.

Thank you for your continued patience.

 


From The Archives:
 

169 thoughts on “Do Not Panic

  1. Indy says:

    Faster but not by much. Nothing to say that it was a change in theme that improved it.

    EDIT: This post took 80 seconds to go through.

  2. Zukhramm says:

    Too late, I’m already panicking.

    1. Zukhramm says:

      The site is now so fast I posted this comment before I wrote it.

  3. MrGuy says:

    “Sheriff, his is no time to panic.”
    “This is a perfect time to panic!”

  4. Daemian Lucifer says:

    Whoa,that was jarring.

    Ok,lets see if this helped.Timing the comment posting now.

    1. Dragomok says:

      I know this is not actually related to site’s performance, but up until now I have thought that the demon in your avatar is male.

      1. Daemian Lucifer says:

        Yeah,I chose a succubus way back when HOMM5 was about to be published.Before that,I had an ifrit.This way,I support the notion that all demons are actually genderless,no matter how they appear.

  5. Shamus says:

    Testing this for myself.

    1. Shamus says:

      Ugh. The comment actually appears quickly, but the page where you MAKE the comment sits there for no good reason. I gave up watching it after 2 minutes.

      Sigh.

      1. Daemian Lucifer says:

        That was my experience as well.The pages load quicker now,but when you post a comment it just gets stuck.Opening a new tab will display it without a hitch.

        EDIT:5 minutes.Thats how long it took for the tab with my previous comment to finally reload.

        1. Indy says:

          And quite suddenly, my 80 seconds doesn’t seem that bad.

          See ladies? It can be a good thing….

          1. Dragomok says:

            Less than a minute for me.

      2. Good luck tracking down the issue.

      3. ClearWater says:

        My comment took 1 minute to the second to start loading. I’m guessing there’s something in your comment handling code that’s doing a connection to something else and probably timing out after a minute of trying. Maybe an incorrect host name or IP somewhere?

        Edit: ok, not to the second this time. That makes a connection time-out less plausible.

      4. Shamus says:

        More plugins disabled.

        1. Shamus says:

          And still more…

      5. Shamus says:

        This is ridiculous. I’m down to almost nothing here – this is basically the default setup of a band new blog and it’s still taking over a minute to post a comment. WHAT ARE YOU DOING, WORDPRESS? WHERE IS THE TIME GOING?!?!?

        1. Kyte says:

          Didja try optimizing/vacuuming the DB? Sometimes the problem’s on the backend. I hear WP isn’t very efficient with its queries.

      6. Shamus says:

        That’s it. No more plugins. And we’re still over a minute to post. WTF?!?!?!?

        1. TheUnHidden says:

          Since i couldn’t post as the site went down as i hit the “post” button, and we now have the theme back, let’s see how long this comment takes.

      7. Steve C says:

        If you don’t wait the pre-req 2 mins while it’s thinking then you lose the ability to edit that comment after it’s posted. So if you comment, then refresh, it’s quick but permanent. Perhaps check the editing code?

      1. Sam says:

        Maybe the database is having problems?

    2. ShamusLoggedOut says:

      So: Default WordPress these. No plugins. This is it. Totally barebones. The site is still performing like complete trash. My last comment is still hung in another tab, going on something like five minutes.

    3. Deadfast says:

      Just testing myself…

  6. newdarkcloud says:

    Testing.

    EDIT: Took about 2-3 minutes.

      1. Piflik says:

        Comment appears instantaneous, but the tab doesn’t load. I get about 90 B/s. Don’t think it’s a database issue since the comment is there immediately if I load the page in a new tab.

  7. X2-Eliah says:

    Best. Theme. Ever.

    Edit: Eh, still quite slow. No change, even. Imo.

    Wat a bit and see if it gets slower as the comment count grows, maybe?

  8. EagleEye says:

    Take a picture of this and add it to the post. The site looks hilarious like this.

    Also, what WordPress plugins do you have installed?

  9. AJax says:

    Can anybody edit the comments after posting?

    1. Indy says:

      The user who posted them (briefly) and moderators i.e. Shamus.

      1. Indy says:

        Oh, you meant now that it reset. Nope. And I can’t edit my post to remove this wrong information.

        1. MrGuy says:

          I just edited a post (the post below, from one dot to two.) So post editing is working for me (or, at least, was working a few minutes ago).

          Or it could be that, in playing around with the site, Shamus might be turning features like “post editing” on and off looking for the slowness…

          Edit: If you can see this, editing posts is STILL working for me…

    1. Indy says:

      Your gravatar looks extremely worried about that fullstop you just posted.

  10. Aquarion says:

    You might find handy http://www.tracelytics.com/. I’ve used it at work, basically you install an Apache mod on the back end and some inserted JS on the front end, and it watches the PHP code’s execution model to see where it spends most time (rendering, database calls, whatever) and then gives you pretty pictures about what’s wrong with your site.

  11. Brandon says:

    I am happy to time a comment. It will be a fine excuse. I take it this isn’t a Gravatar/Wavatar thing like the last time you went hunting for a bottle neck? I guess that was page rendering, not comments, so never mind.

    –And about 1 minute to post.–

  12. Wulfgar says:

    hey, it can stay forever. i like this look.

    over 60s to post a comment

  13. anaphysik says:

    [/HEADLESSCHICKEN]

    1. anaphysik says:

      AHHH IT CHANGED AGAIN!!! EVEN THOUGH I KNOW SHAMUS IS JUST TESTING STUFF METHODICALLY I CAN’T HELP BUT #HEADLESSCHICKEN SOME MORE!!!

        1. anaphysik says:

          (also, no edit ability)

  14. vamphri says:

    Posting has never really been an issue, but I should mention that for me the die cast auto plays. Im not sure if that is a quicktime plugin setting on my end or if its universal though…

    EDIT: 40s to post and 20s to edit.

  15. Shamus, I don’t know if this is relevant, but when I visit this old page showcasing the first episode of Spoiler Warning, I see the old theme. Maybe something in your archives is to blame for the slowdown?

    If not, I think the solution is to post more SW and Diecast episodes until the problem clears up.

    1. 4th Dimension says:

      You need to clear your cache or Ctr+F5 reload that page. You are getting the cached version.

      1. I did, though now it’s irrelevant because he’s changed it back.

        Though it does give me another non-progammer idea: What if something like those old Viddler links is doing this somehow? Some little bit of code that’s phoning home or having a hard time getting the zeroes and ones it thinks it needs?

  16. Tomato says:

    Looks like the mayans were right after all.
    I know adding more plugins on top of this mess might not be a very bright idea, but have you tried WP-Optimize? I’ve been told it’s pretty good for cleaning up the inevitable garbage that piles on a long-running WP install like yours.

  17. Mike Riddle says:

    FWIW.. The initial page display was MUCH faster than before..less than 2 seconds versus 20 seconds

    comment took 44 seconds to post

    first edit took 1:45 to post

  18. Attercap says:

    What’s the DB backend for the site? Have you tried any database cleaning and/or optimization?

  19. Tsi says:

    I’m from France and it took only a few milliseconds…

    1. Tsi says:

      and less than 10 secs to post apparently

      1. Tsi says:

        In fact, the post is nearly instant. It’s only the loading of the resulting page that takes time.
        I can see my posted comment if i refresh the page once i press the post buton.

  20. slipshod says:

    The front page loaded much, much faster.

  21. Ravens Cry says:

    It seems a significantly faster, all things considered.

    1. Ravens Cry says:

      OK, last time the site loading was faster, but submitting comments seemed to take about as long. Let’s see what happens . . . now!

      1. Ravens Cry says:

        Might be just my end for all I know, but comments still took forever to go through.

        1. Ravens Cry says:

          Testing, testing. Glad you found something that works better.
          Edit:
          Oh, wow, that IS faster. Much, much faster.

  22. Thomas says:

    So much white, what is this? The ending to Human Revolution?

    EDIT: That comment went through smooth

    EDIT EDIT: The edit, not so much

  23. Ciennas says:

    Really scared me for a second there with the everything disabled button. Thought the site had been rpossessed by hobo squatters, like they do elsewhere.

    Instantaneous loading, though. Just swap the trolley picture with something familiar.

    Anyway, to test comment posting in this mode.

    57 seconds from my end.

  24. Tony Kebell says:

    Aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaahhhhh!!!!!!!!!!!!

    (1:30, approximately)

  25. ClearWater says:

    For a moment I thought I’d ended up on the wrong site.

  26. KMJX says:

    Is this the twentysided blog? Am I in the right place here?

  27. wheals says:

    Whoa, yet another theme

  28. Mintskittle says:

    Test comment is testing.

    1. Mintskittle says:

      The original tab is still thinking about it. The post seems to have gotten through between 3 and 5 minutes.

  29. Even says:

    Welcome to alternate testing reality.

    By the way, is it correct that http://www.shamusyoung.com/twentysidedtale/?page_id=16473 and http://www.shamusyoung.com/twentysidedtale/?page_id=16458 show the original theme with ads and everything? Because that’s totally happening in my Firefox at the moment.

    The latest Dishonored post shows the previous site theme with the picture of a tram and the one before that still rolls the original theme.

    1. Mintskittle says:

      The first one there shows the default theme for me, but the second one is the proper Shamus-site theme.

      1. Mintskittle says:

        and now this page has the proper theme.

        Also, site load in about 30 seconds. Gravatar/Wavatar the process hog?

  30. bloodsquirrel says:

    IT’S THE ENDTIMES! REPENT! GIRD YOUR LOINS! SMOTHER YOUR CHILDREN IN THEIR SLEEP TO SPARE THEM THE HORROR!

    Also, are you guys doing another Diecast? I was perfect happy in life before them, but now I feel angrily entitled to more.

  31. Also testing. The blog just looks kind of zen now. Makes me feel a little more at peace.

    1. My comment seemed to have gone through immediately, but the site completely stalled loading afterward.

      1. One more test, just for fun and such.

  32. krellen says:

    The page still loaded pretty slowly for me. Since the front page seems to load just fine – it’s only subpages that are really slow – I suspect the comment database is to blame.

    Incidentally, this page seems to be loading with the old theme (I tested it in three different browsers, so I’m pretty sure it’s not a cache problem on my side.)

  33. Johan says:

    The moment I saw the site today I PANICKED

    But then I saw your post

    Slow site is a common theme, thought it is always interesting to see followup posts on WHY it’s slow, I seem to remember one time it was the Date and Time eating up everything?

    Either way, good luck

  34. ShamusLoggedOut says:

    Yet another test.

  35. ShamusLoggedOut says:

    Posting comments is still taking in excess of 5 mins for me.

  36. anaphysik says:

    “Hilariously” (and I do use that term lightly here) the current theme breaks youtube embedding. I wonder what quirks the other test themes may produce.

    1. anaphysik says:

      (Oh, and Shamus followed @DisclosureAlert. *SQUEEEE*. Now if only Aldowyn would edit the eps like he should have a week ago, so that we weren’t stuck with me figuring out ways to produce outtakes <_<)

  37. ShamusLoggedOut says:

    Hmm. That one was less than a minute.

  38. ShamusLoggedOut says:

    Another test.

  39. Charles Henebry says:

    I hadn’t noticed it running slow.

  40. According to a website speed test ([url]http://www.webpagetest.org/result/130303_WD_EQF/1/details/[/url]), 4 of your swf previews eat most of the speed.

    I had a similar problem with my site, when getting data from twitch.tv took 21 of the total 23 seconds loading time. Maybe look into that?

  41. Helpful Lurker says:

    I’m a lurker, but the page has never been slow for me. However, I have adblock (turned off for your site) and Stop Autoplay for Firefox. I’m commenting now to see if I experience a delay, and also if culling the slow from the not-slowed might help us figure out the cause.

  42. Stacy says:

    Testing comment function.

  43. Stacy says:

    Testing comment function again.

  44. Bryan says:

    But, but, but… no large friendly letters on the cover! No cover to speak of! How am I supposed to not-panic in these circumstances?!

    Ahem.

    Not knowing much about the system that processes an HTTP POST and spits out HTML, the fact that normal page loads are running at a reasonable speed, and the DB entries that users’ posts turn into are showing up if the page loads in another tab, I have to assume that something in the POST handler that runs after the DB insert is causing grief here. But I don’t know what it would be (never run php myself).

    Maybe grab a backup copy of the scripts, then add some logging to wp-comments-post.php? (Debug by printf, basically?) I’m assuming that’s the one that’s taking forever to return the redirect with the fragment-id of the new post in it. And since the post shows up on other-tab loads of the page, I assume (without knowing for sure) that the post-redirect GET is taking a reasonable amount of time to finish. Especially if that’s the same URL (except for the fragment-identifier) as a “normal” page load, though now that I say that I don’t know that it is. Hmm… going to try something with this one, hang on…

    1. Bryan says:

      …Nope, same URL after the redirect after making a comment (except for the fragment identifier, which doesn’t actually get sent to the server) is the same as before the comment. So yeah, it has to be the wp-comments-post.php script before the redirect gets flushed to the client, and after the database insert, I suspect.

    2. Bryan says:

      …Nope, the URL after the redirect after making a comment (except for the fragment identifier, which doesn’t actually get sent to the server) is the same as before the comment. So yeah, it has to be the wp-comments-post.php script before the redirect gets flushed to the client, and after the database insert, I suspect.

  45. File this under “spitballing and hoping something sticks.” A few searches for this kind of problem suggested disabling something called “subscribe to comments.”

    If that helps, huzzah. If not, well… the dumb suggestion was free, right? :)

  46. Also, isn’t your site keeping track of how many posts everyone’s login name has made and displaying them as dice? What if you shut that off? Maybe it’s hanging having to go through the entire site’s comment DB to get those totals.

    1. Moridin says:

      The counter counts replies to the individual post, not replies the commenter has made themselves.

  47. newdarkcloud says:

    One thing I’d like to add is that sometimes I’ll post a comment, and the comment counter on the home page will go up, but not on the permalink for whatever article I commented on.

    Then, after some seemingly random time period, the comment will be posted and the comment counter will be what it should be.

    Refreshing the page doesn’t seem to help, so I’ve no clue what’s causing it.

  48. This might be a dupe-post, but the first attempt didn’t go through.

    Could the problem be the comment counter for each user? If the site has to go through the DB and tally how many comments we’ve made before our posts go up, could that be the source of the bottleneck?

    As further “evidence” (though it’s only anecdotal), when the comment counts have been missing their graphics or only appeared to count the ones in this post, the site seemed to load a LOT faster.

  49. Jace911 says:

    Am I the only one who’s worried that Shamus is going to start going through the site’s source code line by line and find OBEYOBEYOBEYOBEYOBEYOBEY stuck in there somewhere?

  50. JPH says:

    doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof doof

  51. Jokerman says:

    Just posted a comment on the last SW episode and after about 5 mins i get a message saying…

    “Duplicate comment detected; it looks as though you've already said that!”

    So its trying to post multiple comments, at least when you post with Chrome.

  52. StashAugustine says:

    … and keep a towel with you at all times.

  53. Deadfast says:

    The whole site is now slow as hell for me Shamus. It takes up to 20 seconds for the server to tell me that die*.png doesn’t exist.

  54. Erik says:

    Have you tried using XHprof? (https://github.com/facebook/xhprof)
    Its originally designed by facebook and gives you great insight in whats taking howmuch time, what paths are executed and howmuch memory is used.

    Helped me out a lot when trying to find performance issues in 6 million lines of code :)

  55. Rax says:

    Is it okay to start panicking now?
    This is the state of the site on my end as of about 2 minutes ago:
    Screenshot on imgur
    (note that it claims the gw2-post and the “do not panic”-post have finished loading)
    Those still working tabs at the bottom have been like this for more than 5 minutes (without posting any comments obviously). Other sites load that number of tabs fine so it’s probably not a problem on my end.
    Between doing the screenshots and writing this two of those tabs have resolved to tell me they couldn’t connect to the databse, the other tabs seem to have loaded fine (except for the comment-counter-picture-thingy and avatars)

  56. Rax says:

    The following comment was written about 20 minutes ago, about 10 minutes after hitting “Post Comment” the site collapsed completely, I really like this “error”.
    It seems to work again, so let’s try that comment again, everything in it still holds true:

    Is it okay to start panicking now?
    This is the state of the site on my end as of about 2 minutes ago:
    Screenshot on imgur
    (note that it claims the gw2-post and the “do not panic”-post have finished loading)
    Those still working tabs at the bottom have been like this for more than 5 minutes (without posting any comments obviously). Other sites load that number of tabs fine so it’s probably not a problem on my end.
    Between doing the screenshots and writing this two of those tabs have resolved to tell me they couldn’t connect to the databse, the other tabs seem to have loaded fine (except for the comment-counter-picture-thingy and avatars)

  57. Hieronymus says:

    I’ve noticed that when your site is loading, it locks-up Firefox for a couple of seconds (most noticeable by opening new (empty) tabs). I’d been assuming it has something to do with the embedded videos.

    I’d try two things:
    1) Disable objects (or however you have videos/ads set up to load).
    2) Take a look at how your MySQL database is set up (or have someone who is more familiar with them do it, I HATE MySQL). If everything in your database is stored in the same chunk of table, then things would become exponentially slower for every addition. If this is the case, consider making several, smaller tables to increase efficiency.

    1. Neruz says:

      Holy crap what did you do?! The latest post has vanished into thin air and comments aren’t nested anymore, huargh.

      -EDIT-

      Took about 2 – 3 seconds for the post to happen.

      -DOUBLEDIT-

      Okay now everything is back to normal. What the hell is going on?!?

  58. X2-Eliah says:

    Site takes about a minute or more to just load an article page (no hot comment-on-comment action).

  59. X2-Eliah says:

    Also, hey, these tools may or may not help. Just in case you weren’t using/checking these stats out.
    http://www.webpagetest.org/result/130303_EW_J2K/
    and
    http://tools.pingdom.com/fpt/#!/LWe7C0paf/http://www.shamusyoung.com/twentysidedtale/do-not-panic/

    (there’s more tool-pages like these, ofc)

  60. Even says:

    Enter the testrix.

    1. Even says:

      Well that was instant.

      1. Even says:

        Website seems to be faster now than it’s ever been in recent memory. All pages so far have loaded in less than 3 seconds. Let’s see how this post goes.

        Edit: Instant. Neat.

  61. MelTorefas says:

    Page took a very long time to load for me, much more so than usual. Now, lets see how this comment goes.

  62. MelTorefas says:

    About 2 minutes. Yikes.

  63. mockware says:

    Maybe you should check on your database and make sure you don’t have a corrupt index on your comments table. That could cause the DB to act very sluggish.

  64. Chris says:

    All you need to do is stay calm and kill the bugs. in this case find the bug so it can be killed.

  65. Stacy says:

    Testing comment function yet again.

  66. Shamus says:

    This post is horribly mis-named. Today was, in fact, an optimal time to panic.

    1. Soylent Dave says:

      Via UK, on Chrome.

      Site loads markedly faster, comments page loads SIGNIFICANTLY faster (by 30s or more).

      Your tower advert on the right is broken (Chrome has been trying to load it while I’ve been typing… oh, no, it’s given up “this web page took too long to respond”)

      Edit: Comment posts in about 10s (used to be more like 45);

      And the advert works now that the page has refreshed, so that’s presumably an error at their end then.

      Huzzah!

      Edit #2: Editing comments is instantaneous. The whole thing: edit box appears right away, edit goes through instantly.

      I know that’s how it’s meant to work, but it’s been SO much slower than that for so long, it’s made me dread needing to edit a comment.

      Huzzah x2!

  67. Wulfgar says:

    test

    less than 10s

  68. Deadfast says:

    I know this is probably a silly thing to say but that has never stopped me before: make sure you do regular DB backups.
    You know, “Nice 235,000+ comments you got there, it’d be shame if something were to happen to ’em.”

    EDIT: That was neat and quick!

  69. Let’s test together. From the UK on Chrome.

    Edit: Went through immediately. Everything much quicker.

  70. Zagzag says:

    Let’s see how fast it is now

    EDIT: Instant! Nice one Shamus!

  71. Jace911 says:

    Testing captain…

    1. Jace911 says:

      Holy crap, that was fast. Looks like it worked.

  72. Sean says:

    Got here too late to see that you had fixed it, but in the future if you’re looking to debug things, XDebug is your friend.

    The only downside is you need shell access to install it, if it isn’t already installed. However, if it’s already installed or you have shell access, it is a lifesaver for debugging PHP stuff. Not only can it let you step through live code line by line, you can also tell it to write out profiling data so you can use something like KCacheGrind to see exactly what’s taking so long.

    If you don’t have shell access, another option is to do a database dump, and run a copy of your WP install locally. That way, you’re running with the same data your site runs with, and you have full control over the environment, so you can easily do stuff like try out XDebug to see what’s going on.

    Saved my life when I was a PHP developer ( and will again soon, my new job is PHP ).

    Anyways, glad to hear you got it all sorted out!

  73. Fang says:

    Really? Post go fast?

  74. Thomas says:

    Fast posting? I don’t want to miss this

    EDIT: That was even better than I could have imagined

  75. Astrolounge says:

    I don’t post comments often, so let’s see how long it takes.

  76. Fawstoar says:

    Hmm, I’m going to test this for myself if you don’t mind. I wasn’t noticing the site/comments being particularly slow; even on iOS it runs fine.

    Edit: Woaaaaaaah. It posted my comment before I could finish blinking.

    No, but really, it’s much improved. Nice work.

  77. Ben says:

    I just posted a comment with a now-unneeded suggestion – the comment went up blazing fast. Congrats! Now relax.

  78. MrGuy says:

    W00t!

    Edit: holy crap that was fast!

  79. krellen says:

    I want in on the testing bandwagon too.

    Edit: Holy Crap, Shamus. Your site is running better than it has in years, I think.

  80. JPH says:

    durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf durf

    1. Asimech says:

      You’re sounding like my neighbour’s stereos.

  81. Daemian Lucifer says:

    Hey,my last comment loaded in under 10s.Was it a fluke,or have you actually fixed it?

    EDIT:IT WASNT A FLUKE!

    Yay Shamoose,you fixed the blog.

  82. Din Adn? says:

    Oooh, the comments are fixed? I wanna try this out also!

    EDIT: Wow. That’s like, faster than a lot of other pages out there even. Awesome.

  83. Neko says:

    boop.

    Edit: Wheeeeeeeeeeeeeeeeeeeeeeee

  84. Aldowyn says:

    Hmm so you just changed a plugin and it fixed everything? I wonder if the old one just couldn’t handle the load (for some reason) or it was a compatibility issue or what was up with that. *shrug*

    *edit* Requisite follow up. That went pretty well.

    **edit** so did that.

  85. it could be a PHP SEO hack on your site. This is where rogue php is inserted into your site’s config files – independent of theme and plugins – and piggybacks on your site’s pagerank to grab seo mojo for spam, usually pharmacy links.

    The reason the cache speeds things up is because you aren’t executing all the php on your site anymore with every pageload. But you’re still infected.

    here’s an article on the so-called pharma hack that gives more details,

    http://wp.smashingmagazine.com/2012/10/09/four-malware-infections-wordpress/

    any WP install with high pagerank will attract this exploit and there isn’t any preventivemeasure you can take, only after-the-fact cleanup. I’ve reinitialized my own geekblog from scratch a few times just to get rid of it.

    — Aziz

  86. Paul Spooner says:

    Glad to hear that the comments are working well again! Strange how software is still so opaque. Ideally, there would be a “debug mode” where you could get exhaustive logging… but I suppose with all the crazy bits tangled together that might not be as helpful as it sounds. In any case, the experts oppinion turned out (as so often is the case) to be the deciding factor. Good on your for asking.
    Edit: took about three seconds instead of sixty. Looks like the fix is everything that you advertised!

  87. McNutcase says:

    Man, the things you miss when you’re racing against time trying to get two tractors from Paris to Bratislava with only one rest stop, and it starts raining hard at 3am somewhere in Eastern Germany as you blast your Scania towards the Czech border at something in excess of 80km/h (not half bad considering these things are limited to stop delivering power at 90km/h and are a real handful to control anywhere past 70…)

  88. Syal says:

    Joining the test group because I can’t bear the thought of not having my name in the comments.

    EDIT: Cool, posting didn’t kick me off the site entirely like it’s been doing.

  89. milos says:

    I’ll feel like I’m missing out if I don’t post something too.

  90. Kdansky says:

    For obvious reasons, this is a test.

    Edit:

    * 10 seconds until posted
    * 5 seconds until edit window opened.
    * Edit2: About 15 seconds until Edit window saved.

    Better than before, but still far from fast.

  91. william dutton says:

    what version of wordpress are you up to? the latest?

    p.s. this post was really fast when i sent it

  92. Tom H. says:

    This is a test comment. Please bear with it; I’ll follow up with a meaningful analytic post.

    There are two (EDIT: three) tools inside Chrome that help analyzing situations like this. (Disclaimers: I work with them and on them most days. There are similar tools in most other major browsers.)

    Control-shift-I brings up the Developer Tools; using the Timeline there to record a trace lets you know what javscript, network, and parsing/rendering events are going on inside the browser. We regularly tell developers to use it to look at their site’s performance, and have been using it recently to look at e.g. a 2-second regression in loading time for one site in a prototype version of Chromium.

    Taking a new tab to chrome://tracing lets you record a trace showing the internal C++ calls on all the threads active on Chrome. It’s probably overkill for this kind of slowdown, and is usually only good for recording 10-20 seconds of activity; I use it mostly to look at slow rendering.

    Oh, and there’s a third tool that could conceivably be relevant: chrome://profiler gives you aggregated statistics about the C++ events inside Chrome, particularly the browser process. Good for hunting down weird IO issues.

    1. Tom H. says:

      I’m only seeing a 12-second delay, not the minute+ that seems to be reported throughout the thread, so I can’t diagnose much.

      Unfortunately there isn’t a way to do file attachments here, but using a timeline recording (write the comment, switch to tracing tab, hit “Record”, switch back to this tab, hit “Post Comment”) I can see things like:

      Once I click the button to post my comment, there’s a two second *gap* in the record where the browser doesn’t seem to be doing anything, but there also isn’t a pending event or other reason for it to be waiting for a server.

      3.8 seconds after the recording starts, Chrome sends a request to load http://www.shamusyoung.com. It takes 1.25 seconds to get a response back, then parsing happens and we fire off a bunch of subsidiary requests. Some of those take 2.5 seconds to return.

      At about 6.5 seconds into the trace, we start the largest HTML parse. This fires off dozens of requests for PNGs.

      Various ad networks and syndication networks make javascript calls that don’t complete for a long time, but don’t seem to be heavy on the CPU; they’re just firing off requests or timers that take a while to complete.

      There are a bunch of little things you could to to speed up the site, but I tend to live deeper down in the renderer than this, so don’t have advice to give that you can’t find on plenty of websites, and don’t see any smoking guns. Even if you don’t have your own server to wire SPDY into, a lot of their suggestions can be done manually (For instance: sprite-sheet your die*_* PNGs so you make one large image request instead of dozens of little ones! Since that code’s stable, you need to fuss around with sub-sampling the sprite sheet once, but you’ll save a LOT of network traffic and round-trip-times).

  93. Scampi says:

    Page loads a lot faster than ever for me(what means: only 5-6 secs loading a category site, only 3 secs loading the start site)-now: checking comment speed…

    Edit: This post: 4-5 secs (instead of the previous multiple minutes). Whatever you have done to make the site THIS fast: it’s amazing.

  94. Aquarion says:

    Could it be the query that runs to find out if this comment needs to be moderated? Because that’s would be a text match on the email address, and looking at my old WordPress install, it doesn’t look like it’s got any indexes on it at all.

  95. anaphysik says:

    So I hear things are great again (again since ~200x)?

    EDIT: YES THEY ARE!!! 3sec to load!
    EDIT2: And even less to edit!

  96. NotACat says:

    Is this why the navigational images to the top-right are failing to appear on any but the home page? (well, for me, at any rate…)

    1. Shamus says:

      Can you post the url you’re seeing at the top? It SHOULD be http://www.shamusyoung.com/twentysidedtale/?p=18914

      If it isn’t then you’re probably going to http://www.shamusyoung.com/twentysidedtale/do-not-panic or somesuch. NOBODY should be seeing that, but I fiddled with a setting yesterday that made this happen for about an hour and some people are STILL seeing it.

  97. Thomas says:

    There was a post of the latest Fallout which has disappeared right? I’m not going mad?

  98. Thomas says:

    I can’t see comments on this page until I comment myself. (I’ve tried refreshing and ctrl+shft+r which should clear the cache for the page right?)

    1. Thomas says:

      Okay http://www.shamusyoung.com/twentysidedtale/?p=18914&cpage=2#comments displays correctly, but the main page is linking to this for me:
      http://www.shamusyoung.com/twentysidedtale/do-not-panic/#comments

      Which isn’t showing correctly. When I comment it goes to the right page afterwards

      EDIT: All okay again, and the article has shown up

  99. Nick-B says:

    Wasn’t there a fallout 3 video up above this post earlier, and… over 160 comments on this post alone this morning? now we’re down to barely 110+

    Oh, and test.

    EDIT:

    Ok, now it’s back up to 160+ posts, the place where dice was earlier is now back to dice (it was showing some ridiculously high number, like 25056 posts or something, incrementing), but seems to be responding nicely. Posting the comment took mere seconds (rather than the minute plus it took before), and will post if this comment edit takes longer.

    Am on firefox, with most default ad sites blocked via noScript.

    Second edit:
    Quick post as well. main site now shows the “DO NOT PANIC” post as having 160+ comments and the fallout 3 article as well. strange. Almost as if when I loaded up the site just 10 minutes ago it was a cached version until I clicked through to this post.

  100. Did the SW post for the Little Lamplight episode disappear for everyone else, or is it just me?

    Edit: Okay, it’s there. I think I just needed to clear my cache and reload. Odd that it reverted to a previous version of the page sans F3SW, but hey, it’s the interwebs.

    Also, I didn’t see most of the page 2 comments until I commented as well. Cache again?

  101. anaphysik says:

    So I’m still having /a/ problem with the site. It wasn’t remembering my field entries (name/email/site). Sounds like a cookie problem, and indeed I saw no shamusyoung.com cookie in my list. So I’m trying to rebake one?

    EDIT: And I guess rebaking worked (I can even edit after reopening my browser!). It was especially odd as it seemed there was no reason for that cookie to disappear on my end :S

  102. Ben says:

    I’m morbidly curious how slow it is.

  103. hborrgg says:

    Not that I’m panicking, but did an entire SW post just disappear?

  104. anaphysik says:

    Shamus, your site still hates me D:

    EDIT: It starts working again whenever I make a post, but making a post in this thread every time I come here is not an ideal solutions D:

    1. hborrgg says:

      Shamus Help!

  105. hborrgg says:

    oops, it’s all gone again

  106. Kerin says:

    Since it’s hanging after a comment is added, but the comment appears right away, either:

    -there’s something happening after the post action that shouldn’t be. If you modified any core WordPress code, this is likely the cause, or

    -everyone’s right about it being database related. This is where I’m leaning too.

    What I’d recommend you do is go right to the source and start logging slow queries that hit the database:

    http://dev.mysql.com/doc/refman/5.1/en/slow-query-log.html

    This will tell you immediately whether your queries are taking a million years to complete, which will in turn tell you exactly where to look (or, at worst, rule out the database.)

    I develop and debug WordPress plugins pretty regularly, so if you get results that you don’t grok I’d be happy to offer what I charitably describe as exertise. ;)

    1. Kerin says:

      Actually, the comment posted ULTRA fast so maybe you’ve sorted it. :)

  107. X2-Eliah says:

    So hey, for about 12 hours up until now, this site seemed as if frozen in time for me – the Fallout 3 ep19 post didn’t show up and the comment count on this article was stuck at 109 posts. Now it all ‘caught up’, I guess, but.. Well, it seems like either the new cache plugin is not passing up-to-date things, or that there’s some database connection issues.

  108. InternetCommenter says:

    I don’t know if this is related or just something on my end, but this site always took ages to load for me (twenty seconds or so, whereas every other page on the Internet loads near instantly), up until I installed NoScript, at which point it’s now as fast as everything else.

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun. Your email address will not be published. Required fields are marked*

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>

Leave a Reply to StashAugustine Cancel reply

Your email address will not be published.