Broken Stuff and Security Concerns

By Shamus Posted Wednesday Jan 10, 2018

Filed under: Notices 51 comments

Yes, the forums are down. Yes, I realize you can’t edit your own comments. Let’s talk about that.

On my Linux-based webserver, there is a user account linked to me. This “shamus” account owns all the files: All the PHP scripts to drive the blog, all the scripts to run the forums, and all the images and other random files that makes the site operate. Under normal circumstances, the entire file structure is designed so that only my user can upload, delete, and modify files.

However, you need to make some exceptions. For example, I run a WordPress plugin that makes weekly database backups. This plugin needs to be able to save these backups, which means that I need to make the backup directory writable for all users, not just the “shamus” userPHP, MySQL, and other processes are owned by the root user.. Otherwise, the backup plugin would run but it wouldn’t be allowed to save the resulting backup to disk.

So I need to make a few spots on the machine where processes not owned by me can put files. This alone isn’t enough to compromise the security of the machine, although it’s often considered something to be avoided if you can help it. The danger is that it may provide an attack vector for potential hackers. If there’s a vulnerability in either WordPress (the software that runs the blog) or PhpBB (the software that runs the forums) then they would be able to write files to these directories.

Here is a ficticious example of how something like this could work: Let’s say the forum offers a feature where users can upload their own profile image. You’re supposed to upload a JPG or PNG image file. These files end up in /forums/profileimages/. In order for this feature to work, I need to set the permissions of /forums/profileimages/ so that anyone can write to that directory. Let’s say the people who wrote the forum software didn’t do their job and the forums don’t make sure that what the user uploaded was actually an image. Like, maybe they uploaded a PHP script. This allows them to put new pages on my site, and those pages can do all sorts of nasty things.

Now, they can’t just put those pages anywhere. Those pages can only end up in /forums/profileimages/, and only the attacker will know about them. Once the upload is done, the attacker can then manually type in the URL like so:

shamusyoung.com/forums/profileimages/badpage.php

This will cause the script to run and do whatever it’s supposed to do. This doesn’t give the attacker full control over the machine. (They can still only put new files in directories I’ve had to leave open.) They can’t re-write the blog or attack visitors directly, but this is still an alarming situation that allows them to see a lot of stuff they shouldn’t.

This is a very simplified explanation. The actual method of attack is a lot more complex and to be honest most of it is beyond me. But this is the idea in broad strokes.

A couple of months ago PeterHe doesn’t comment often so you might not know him, but Peter has been providing technical and hardware support to this site for a long time. and I discovered some files on the site that were not owned by the “shamus” user. Files like this:

lprvpluh.php
pvkmnwoj.php
onrvyxwg.php
ukwwtgwx.php

Always the same pattern: A PHP file with a gibberish eight-character name, probably generated at random. These files contained highly obfuscated PHP code and were not part of the normal file structure of either WordPress or PhpBB. More importantly, they are obviously malicious in nature.


Link (YouTube)

Peter and I have been battling this mess for the last month or so. We deleted all the suspect files, tightened up directory access, and then hoped we’d fixed the problem. Then a few weeks later the mystery files would show up again and we’d have to start over.

Last week the files showed up for the third time, and so we went to maximum paranoia level. We wiped WordPress clean and started over with a fresh install. We uninstalled the forums completely. This machine is now as locked down as we can make it. There are no directories with write access. This would break several of the WordPress plugins I use, but since I haven’t installed any plugins that’s not a problem yet.

If the problem returns, then I’ll need to contact my host and have them wipe the machine clean and start over. I’d hate to do that, since it would result in a ton of downtime. (The blog has about 1.2 gigabytes of images, and I don’t have a very fast upstream connection. That would be a long upload. Not to mention the time required to restore the databases and re-install everything.)

I’ve deliberately left out a lot of details on the off chance that the attacker actually reads the blogThis is unlikely. These kinds of attacks are often done by bots.. So if you’re thinking of asking, “Why don’t you guys just X?”, then keep in mind we probably did X but I’m leaving it out of this explanation.

So that’s why the forums are gone and all of our quality of life plugins are missing from the blog. It’s a known issue. We’re still investigating. If all goes well, then we’ll eventually get back comment editing and all the other little plugins we’re used to.

 

Footnotes:

[1] PHP, MySQL, and other processes are owned by the root user.

[2] He doesn’t comment often so you might not know him, but Peter has been providing technical and hardware support to this site for a long time.

[3] This is unlikely. These kinds of attacks are often done by bots.



From The Archives:
 

51 thoughts on “Broken Stuff and Security Concerns

  1. Daemian Lucifer says:

    You should still contact your host and ask them at least what are they doing about the hardware vulnerabilities that have popped up recently.Specifically the spectre and meltdown vulnerabilities that were disclosed these days.Theres not much you can do about it,but at least youd be informed.

    1. CoyoteSans says:

      I mean, it’s not like THEY can do much either at the moment. At the very best, they might have a patch for Meltdown and one variant of Spectre, but I don’t think the chipset makers have even put out the microcode patches that Spectre requires just yet.

      Although I’d say these vulns are so new the probably of them being both 1) actively exploited in the wild and 2) Shamus’ site getting whammied with them this soon is pretty damn small.

      1. Daemian Lucifer says:

        They arent that new,they are just newly leaked to the public.But yes,there probably arent many bots floating around that exploit the them.Still,its nice to know what to keep your eye on,and at least be prepared to burn down everything if the unlikely happens.

        1. acronix says:

          My understanding of them is that they can only be exploited by people who are physically at the computer.

          1. King Marth says:

            This is emphatically not the case. Meltdown and Spectre are specifically most dangerous for computers shared by multiple users, such as web site hosts.

            The actual impact of these exploits is that the attacker learns all information in the compromised computer’s memory, including everything the operating system would normally keep private (including the data owned by other users active at the time). The exploit does not allow the attacker to modify the target, though it might reveal sensitive information such as passwords if they are stored in memory, which could then be used to make changes.

            Also, run updates. These are very fundamental issues with hardware design, but there are software countermeasures which significantly reduce the attack surface.

            Weird, the spam checkbox is gone. It makes sense given this post, but still odd to see.

          2. Phill says:

            Nope. As long as the attacker is able to run code on the computer, they can in theory run the attacks. But for the average home user there isn’t much point: there are much easier vulnerabilities to exploit. It is more a problem for cloud computing services where you can use it to spy on data in processes that other people are running.

            1. Alan says:

              In theory it can be exploited from Javascript, which means it potentially threatens nearly everyone. Browser makers are shipping workarounds that hopefully render that path irrelevant. (In Firefox’s case, I gather they’re making the various timing functions less accurate as the attack relies on very precise timings.)

              1. Decius says:

                Isn’t the implementation very specific to the hardware involved?

                1. Pete_Volmen says:

                  Quite the opposite, for Spectre anyway. All x86 (and x64 by extension) systems are vulnerable. There’s talk that arm could be in trouble too.
                  Intel is more vulnerable to specific version of Meltdown (which itself is a subset of sorts of Spectre), but while AMD is a bit safer, it’s still not safe.
                  Part of what makes it so dangerous is that Alan is right. So long as you can run javascript on the system (through pretty much any browser for example), you can get access to data. Depending on the specific attack it might be very slow (everything needs to go through cache, and reading/writing from it for every bit of data is slow as hell), but that’ doesn’t save the victim much pain.
                  Problematic is that there’s no real patch that could fix it. It’s down to the way CPUs are built these days (running code before it’s needed if they expect to have to run it later, for example). There’s workarounds for current CPUs, but there’s more overhead, making them (marginally) slower, and it’s not exactly a real fix.

                  1. Pete_Volmen says:

                    Shamus, I’m pretty sure i escaped the italics. but it doesn’t seem to work

                    I definitely escaped both italics and bold in this message. if the current sentence isn’t regular, the escaping doesn’t work. If it does, just ignore it. Hope it’s user error in this case.

          3. guy says:

            That’s apparently a bit complicated; any code can use the exploits to load protected memory into the cache, but then they have to get that information from the cache to somewhere useful to them. Some of the methods they can use to do that do require physical access, but most definitely not all of them.

  2. Bubble181 says:

    OK, but, like, why don’t you guys Y? :p

    1. Daemian Lucifer says:

      Y is so last year.In 2018 Z is all the rage.

      1. Ciennas says:

        No, we went grecian about Three days in. Theta Delta 9 is the true solution for all of lufe’s problems!

        1. evileeyore says:

          Shamus, y’all should go hexadecimal and implement option 0x19.

          1. Erik says:

            Execute “End of Medium”? That sounds ominous. I’d recommend 0x1D, “File Separator”. At least that keeps all the suspicious files separate. :)

  3. Arstan says:

    I really hope everything sorts out. Best of luck to you and Peter!

    1. Christopher says:

      Same, hope it works out.

  4. ElementalAlchemist says:

    The blog has about 1.2 gigabytes of images, and I don't have a very fast upstream connection. That would be a long upload.

    Your host doesn’t offer FTP backup? If it comes to nuking the site from orbit, it’s probably time to find a better host.

    1. Peter A says:

      Actually it’s more about how the web server is managed. If the host is going to force you to set world-wide writable directories in order to make WordPress function, you’ll never be rid of this problem. Linux has group level controls. As long as your web server process and the user that is executive any background processes are in the same group and the directory is owned by that shared group (regardless of user), then it’s writable by both without being worse by the internet at large.

      If your host can’t help you set that up, it’s definitely time to switch hosts.

      1. Matt van Riel says:

        This is true. However, unless you’re running a dedicated server and have full root access, this isn’t the sort of tweaking you can do as a user normally. My Dedibox lets me create users and do basically anything I like (I run VNCServer with a desktop environment and WINE for certain programs I can’t run locally right now because I’m between homes and reliant on tethering). But my hosted accounts (where I have my sites) only let me set permissions per-folder using CHMOD, nothing deeper.

        I don’t remember ever coming across a host who’d actually go to the trouble of setting up specific Linux users and other backend stuff before, but I suppose if you’re paying for a managed solution that should be something you can do? I’m on opposite ends of those poles, with both an unmanaged hosting package and a fully dedicated server, nothing in-between, so I’m not sure what that grey area in the middle of my own experiences is capable of ;p

        1. ElementalAlchemist says:

          Doing a lookup, I’m guessing this is the host – http://www.peak10.com – and presumably Shamus is using one of their managed services.

      2. Chris says:

        I suspect (and really, really hope) that the “writable by everyone” bit was just an oversimplification for brevity and the sake of the non-techie readers. Actually granting 777 to a website directory would be madness.

  5. Eichengard says:

    Wait, who’s Peter? Have I missed a person appearing? I feel like I’m in one of those “you remember the new guy, right?” TV show episodes where they try and convince us a new character has been around since the start.

    1. Matt van Riel says:

      As Shamus said, he doesn’t post much. I think I’ve seen him directly involved in a conversation maybe twice in the last three years :)

      1. Eichengard says:

        Ahh, somehow I missed the footnotes to this article I in first read through.

  6. Dev Null says:

    So if you're thinking of asking, “Why don't you guys just X?”, then keep in mind we probably did X but I'm leaving it out of this explanation.

    I’m going to do it anyways, just because I’ve seen _exactly_ those symptoms before. So even though you’ve probably already done it: did you disable XMLRPC? Like, down at the .htaccess level?

    Yeah, thought so. But I had to ask; there was some canned push-to-hack script running around a few years back that exploited xmlrpc and generated filenames that looked exactly like that…

    Carry on, and good luck.

  7. Dreadjaws says:

    Have you tried turning the forums off and on again? How about unplugging your PC? Yeah, that’s the limit of my expertise in coding.

    I mean, I can do a couple of “If… Then” lines in basic, but somehow I don’t think it’d help.

  8. PeteTimesSix says:

    I see your satirical hacking-related video and raise you a better version.

    1. Philadelphus says:

      I was going to post that exact video! :D

    2. Droid says:

      I only recently found out about DoorMonster, and they are everything I never knew I needed in my life, and nothing more than that. In their 5 years’ worth of backlog, I don’t think they ever released a video (not counting community or announcement/organisation videos) that didn’t make me chuckle.

  9. Fade2Gray says:

    This “shamus” account owns all the files

    The context and quotation marks made me read this as “Sham-us”.

    Also, I now feel completed to point out that I am not a robot.

    1. eaglewingz says:

      I also am not a robot.

      Though I may be an android.

      1. Daemian Lucifer says:

        Ok then,lets test the two of you:

        THIS STATEMENT IS FALSE

        Now,eaglewingz:youre in a desert,walking along the sand,when all of a sudden you look down and see a tortoise.You reach down and flip the tortoise on its back.The tortoise lays on its back,its belly baking in the hot sun,beating its legs,trying to flip itself over,but it cant.Not without your help.But youre not helping.Why is that,eaglewingz?

        1. Fade2Gray says:

          Ok then,lets test the two of you:

          THIS STATEMENT IS FALSE

          Oh, Maker! My life is a lie!

  10. FluffySquirrel says:

    Don’t think I’ve ever had a client with a wordpress which didn’t get hacked at some point. It’s very popular with script kiddies I assume

  11. Redrock says:

    Gosh darn it. I really miss the edit button. I mean, if things stay like that, I may have to start actually checking my comments for typos before hitting the ‘post’ button. And that would just be awful. That would almost be like work.

    1. Viktor says:

      There have been studies. Changing the formatting of something makes your proofreading more accurate. Basically, when it’s the same formatting, your brain assumes “oh, I’ve already read this, let’s just skim.” A new font or background tells your brain “this is new text, better read carefully”. That’s why you don’t spot the typos until after you hit “Post”.

      1. Ander says:

        Interesting. I should implement the Comic Sans proofreading technique.

      2. Fade2Gray says:

        This must explain why, no matter how many times I reread my comments before posting, I almost always find typos after posting. Often multiple times in a row. I once got myself flagged as a spammer on a website for editing the same comment to remove typos too many times in a row.

        1. MichaelGC says:

          If you really need to be sure a document ends up with no typos, print it out and proofread a paper version. Whilst it remains electronic the pesky 0s & 1s are free to jump around when you're not looking, but if you run off a hard copy they know they have to settle down and behave, else you'll have them rang to bights. Or … I'm pretty sure that's why the method twerks, anyway.

      3. Philadelphus says:

        I can absolutely confirm this in my own experience. When writing on my own blog I can reread something in the editor multiple times and not notice anything, then spot multiple typos as soon as I hit the preview button. The ability to preview before posting something is a wonderful, wonderful thing.

  12. Brian says:

    Since I tend to deal with the lots of files problem somewhat often, may I recommend instead of using your personal computer/net connection for the site backup/restore process, instead use a VM somewhere else? Using (something like) Amazon to ssh/ftp/scp/rsync into your site means that you can leverage their storage and their connections (and glacier, so their backup) which could significantly reduce site downtime and increase options. And once you’re done with the instance, you don’t need to keep it around.

    1. Hmm! That’s pretty darn clever actually.

  13. MichaelG says:

    I use Hosting Matters and have been hacked at least once. A directory full of Canadian pharmacy ads appeared on my machine. I deleted it and complained and nothing (that I know of) has happened since. But I just run static web pages with no DB, so not much to disturb there.

  14. Tometzky says:

    Hi. I don’t have much time before I have a ridiculously small amount of sleep left tonight, but here’s some advice from a sysadmin with programming, security and webmastering experience:
    * almost always it’s some automatic worm that’s compromising your site, not a dedicated hacker;
    * almost always it’s either vulnerable (outdated or unsupported) plugin you use (like 90%), vulnerable (outdated) version of WordPress (like 5%) or a vulnerable (outdated or unsupported) library in your theme;
    * it’s counter-intuitive but your limiting write access to WordPress directory actually greatly increases the probability of a successful hack, as WordPress can’t upgrade itself automatically with these constraints;
    * you will not be able to keep up with WordPress proper / Plugins / Theme security updates manually – you really should allow it to update itself and it’s plugins automatically;
    * you have to clean your site very soon, or browsers will start to refuse to load it;
    * you can check the modification dates of the malware files and compare them with HTTP server logs – more often than not the URL used for compromise will be there and can point you to the name of the plugin which was used to compromise your site;
    * check your list of users and force a change of all their passwords – it could be overwritten by a hacker to ensure future access;
    * if a site was compromised and a worm was able to run any code on site then it’s very hard to be sure what was changed by it to maintain access to the site – but based on these random named files it looks quite primitive, which is good;
    * install your operating system security updates regularly and reboot the server regularly;
    * install Wordfence – it will alert you if there are suspicious files in your WordPress installation and can check all your WordPress files if they are unchanged.

    1. Matt van Riel says:

      “you have to clean your site very soon, or browsers will start to refuse to load it;”

      Not just that, Google will also delist it. Then you have to go through a mildly annoying process to get them to list your site again once you’ve cleaned it.

  15. “WordPress plugin that makes weekly database backups”
    If you are able to run CRON jobs on the server I’d suggest getting a “standalone” wordpress backup script (could be written in PHP even) that is run by the CRON job service and just get rid of the plugin (which is dependent of running when a user triggers it right?).

    Once thing to make sure of (tat is easy to forget) is to place config/password stuff needed, outside the webroot (in other words, make sure hat the webroot is a subfolder on the server). That way someone can’t download a config file etc.
    Sure you can use .htaccess to “lock” certain folders or files but it’s easy to forget/screw up.

  16. Zaxares says:

    Yikes! Chances are it’s small scale stuff, but you can never be too careful. I hope the latest measures manage to halt this stuff in its tracks, Shamus.

  17. adam says:

    I can’t believe you put up with that shit. I couldn’t do it. I’d have lost patience years ago and thrown my monitor out a window.

    If I had to host a blog I think the most complication I could deal with is throwing up a simple blog on something like GitHub Pages (e.g., Jekyll) and manage my comments with a 3rd party solution like Jeff Atwood’s (https://www.discourse.org). Just wash my hands of the hard stuff.

  18. Even Alander says:

    It can also be worth checking your own (plus those that contribute) computer(s) for any viruses. Came across this once where malicious scripts (javascripts) would be added to some free text form plugins in WordPress once. We found a pattern where the scripts would re-appear every time a specific user had done changes to the site. After cleaning up that computer we didn’t see it any more.

    Would agree though with some of the other comments here that this is more likely than not bot(s) creating users and uploading files and then triggering them. A good thing to do with static resource folders is have them being served directly by Apache with mime types added (only image), that way the PHP files will simply download, not run.

Thanks for joining the discussion. Be nice, don't post angry, and enjoy yourself. This is supposed to be fun. Your email address will not be published. Required fields are marked*

You can enclose spoilers in <strike> tags like so:
<strike>Darth Vader is Luke's father!</strike>

You can make things italics like this:
Can you imagine having Darth Vader as your <i>father</i>?

You can make things bold like this:
I'm <b>very</b> glad Darth Vader isn't my father.

You can make links like this:
I'm reading about <a href="http://en.wikipedia.org/wiki/Darth_Vader">Darth Vader</a> on Wikipedia!

You can quote someone like this:
Darth Vader said <blockquote>Luke, I am your father.</blockquote>

Leave a Reply to Roger HÃ¥gensen Cancel reply

Your email address will not be published.