Stories
Slash Boxes
Comments

Dev.SN ♥ developers

posted by NCommander on Tuesday February 18 2014, @12:18PM   Printer-friendly
from the holy-fucking-shit dept.

So, as I write this, day one has officially come to an end. I'm still somewhat in shock over it. Last night when I was editing the database to change over hostnames and such, I was thinking, man, it would be great if we got 100 regular users by tomorrow. Turns out I was wrong. By a factor of ten. Holy cow, people. I'm still in a state of disbelief, partially due to the epic turnout, but also because our very modest server hardware hasn't soiled itself from the influx (the numbers are, well, "impressive" is a way to put it). Anyway, I wanted to do a bit of a writeup of where we stand now, what works, and what doesn't. Check it out (and some raw numbers) after the break! Warning, it is a bit lengthy.

As some of you may have noticed, we've had a couple of hiccups today, but all and all, it was a huge smashing success for day one, and you, the community, made it possible. Here's our version of the good, the bad, and the ugly.

The Good!

Articles, Commenting, and User Profiles: As the basic life blood of any website, these features need to work 100%, be usable for both staff and for the community. As many are aware, we had an issue with story submission; the problem here was that Slashcode has a "Human Confirmation" module which basically went nuts after we launched. That's been fixed, and submissions should work from any user, and from ACs. As for user registration, this is working, but we've had issues with emails either getting placed into the spam bin, or dropping into the nether regions. We're still looking into this, and hope to have it addressed soon. As for user features (referred to as the Zoo in code), things like Friends/Foes/etc. are all working as expected.

Moderation: Of all the bits of Slashcode that needed tweaking, this is the only one that required a rewrite. As it's difficult to test any code that depends on a huge group of users, we went live knowing that this was going to need ongoing massaging. I won't go into the nitty-gritty here, but suffice to say we're keeping a very close eye on it. Basically, modpoints are issued if you're logged in and actively browsing on the site, and expire four hours after being issued. The UI doesn't accurately reflect this, as this change was made a couple hours ago, and the templates don't provide enough granularity to show hours vs. days. For those who are curious, I did a write up the original slash system (and why it had to go) here, and a second one going into details behind the current implementation, and the thought process that went into it here. Comments welcome.

Site Responsiveness: Appears to be good all around, despite the very modest hardware we're running on. Slash does plenty of caching with memcache, and varnish is helping loads. I don't expect any short term issues with us getting overloaded based on our current capacity and the resources we're utilizing. If there's one thing slash excels at, it is at being an incredibly scalable chunk of perl.

Development Portal: Up and running on our local FusionForge installation. Like the main site, this one also having some email issues, but we're manually approving accounts as they come in. The current latest code can be found there, as well as our bug trackers. The SSL issue is known, and we've got a proper certificate waiting to be installed. The git repository for the site is there (as well as mirrored on github) if you want to dive in. Feel free to drop in at ##altslashdot on irc.freenode.net if you have questions.

Statistics: Slash's internal stats engine is mostly working, but is slightly skewed due to Varnish. We can get realistic numbers by combining slash's numbers with those from the host itself. Needless to say, I was impressed.

The So-So!

Journals: On a technical level, these are working as best as we could expect. The problem is there's a lot of missing functionality, either in stuff that you expect to be there, and isn't, or stuff the other site has that we don't. The biggest glaring omission is the missing "Submit for Submission" button. I'm not sure if it's just not existent, or the HTML for it fell off the templates. It would not be the first feature we had to reconnect to the rest of slash. (Next/Prev article was another one). To be honest, overhauling this entire section is on my wish list, and I have some thoughts on this that I will eventually write down, discuss, and perhaps even implement.

Topics: Oh boy ... for a simple feature, this proved incredibly irritating to get to work, resulting in us hosing the site horribly for several hours. In the end, it required implementing the topics one by one in the database by hand then trying to get backslash to cooperate. This feature is somewhat over-engineered due to the fact that every article requires at least one "Topic" and a "Nexus", which is a section of the site which can exist as a subdomain. For the most part, these are implemented, but are all sharing the same icon (the art wasn't quite available by go-live). Expect to see icons sometime soon. The topics.pl page however is glitching out, for reasons I can't completely understand. For the most part, it's functional.

UTF-8: Believe it or not, we do support it (and it was pretty simple to get working, but it's not 100% bug free. This seems to be mostly related to the internal HTML validator slash uses on incoming comments, but the results are mixed based on your operating system, your browser, and the phases of the moon. If you want to see the UTF-8 in actual for yourself, check out here and here.

The Bad!

Search: Mostly hosed. The backend requires Sphinx, and possibly patching MySQL. It seems to be broken on the other site, as well, when I tested it, so at least we're not alone in this department. It does work for searching based on authors/comments by user, thus it was left in place. Just don't expect it to work for general queries right now.

The Skin: We've had ongoing CSS and theming issues since slash was installed. Considering what we had when we started, it's somewhat impressive that it's usable at all. PrestonL and many, many others on IRC contributed CSS fixes, and a proper fix for the user's page went live late today. I need to go through the logs and get an exact list of contributors so I can all give them the proper kudos and put them in the AUTHORS file.

Static HTML Pages: Slash comes with a fair number of static pages, like the About and Help pages. Most of these pages are there, but are either horribly out of date (for example, anything related to slashcode), horribly wrong (moderation), or just nonexistent. Basically, if it is documentation related to the site, the odds are that it's either 404ing or just wrong. Fixing this is mostly a matter of writing them and putting them in the correct places.

The Ugly! Welcome to the top (or perhaps bottom) issues currently plaguing the site today, those that have no easy resolution. Help is especially needed for all the below.

D2: As many people no doubt noticed, we launched with the old D1 system. While perfectly functional, it has a *very* 1997 feel to it, and there are users that do prefer the more dynamic D2 system. This wasn't some conspiracy to launch an entirely Web 1.0 website (which happens to mostly work with NCSA Mosaic), but due to the fact that most of it is missing. We have the actual code and backend stuff, but none of the javascript that makes it work. Furthermore, even if we did, it would be the rather ugly version that had the "floating box of doom" (since that code is from that era). The switch in the UI creeped back in after go-live, but functionally it might as well not be there. I'd like to give the D1 system a SLIGHT facelift (similar to the old Greasemonkey script for Slashdot), yet make sure this site always works correctly. Ideas and volunteers welcome.

Firehose: It's essentially the same story as D2, except I'm not even sure that all the backend code is there for it. If we want it, it's basically a burn and rewrite.

Tags: Unlike the other two, we've got most of bits for them. They're just not exposed anywhere on the skin. It's probably possible to bring them back if the community feels that they add value, but it will require someone who knows HTML, javascript, and Template::Toolkit to do it.

Meta-moderation: This is a case where the implementation itself is questionable. We've got all the code for this, but it's broken. When the other site rolled out firehose, they broke this. It appears all the logic is in place, it just needs minor brain surgery. The problem is, the old moderation system was built on a fundamentally wrong premise that got lucky, and I'm honestly not sure many people ever metamodded in anything approaching significant value. This entire feature needs a rethink in light of the moderation rework.

Theming Engine: Slash's theme engine is an example of evolution, not of intelligent design. Basically, if we want multiple themes, it requires multiple Apache instances, running different vhosts for different Nexuses. If I lost you, the take away is its pretty damn ugly. Its not unsalvagable, most of the themed bits live in the database, and it might be possible to make themes user-selectable (it appears some work was done to implement this in slash before being abandoned).

Backslash: Most of you are probably "WTF is backslash." It's slash's version of an admin panel, and it was clearly never designed for mere mortals. We've smacked most of the bugs out of it, but it remains somewhat awkward to use, and for suadmins, it is possible to actually break the site with a misclick (we've already done it and put slashd in an infinite loop). For the most part, the story editors is "relatively" usable, and I finally smote the long standing 500 error. For end users, the biggest thing on this to know is that the two-man system isn't enforced. Articles will be flagged yellow or red (depending on the author) on the master list, but nothing actually prevents those from going live. Furthermore, there's very little in terms of sanity checking, and it's far too easy to accidentally publish something. I just ask the community bears with us as we fight our backend tools.

Mod_perl: And we get to the 600-pound gorilla in the corner. Slash is tied to mod_perl 1.3 and mod_perl is tied to Apache 1.3. If you know Apache, you can guess the problem. If not, it's basically summed up as this: 1.3 is EOL, and has been for some time. There are no "known" CVEs, which helps, but I think everyone wants us to get to supported server software ASAP. This is compounded by the fact that mod_perl development appears to have ceased, and no version exists thats compatible with Apache 2.4. While not an IMMEDIATE problem, it's something we need to figure out for longer term plans for. There's been talk on going to FastCGI or a similar solution, but no definite long term plans have been made. At least one serious effort was historically made to try and update the codebase to 2.0, and MOST of the Apache binder stuff is relatively self-contained. At least we're not alone on this, the few slashcode sites that we're aware of (aside from the other site, which Netcraft reports running Apache 2.2) are still running 1.3.

Final Remarks, and Server Information, and Day 1 Stats

Wooo, still with me? Good. The long and short of it is we're doing pretty well overall, but there's a lot of work that still needs doing. If you want to jump in, grab the code and join us on IRC, all contributions are welcome. After this post goes live, I'm going on a hiatus to try and get my life/work balance back in order. I hope to return in a week or two to help the dev team digging. As always, I'll be available on IRC should something (like the mod script) needs my attention.

A couple^Wsome^Wlot of people have been asking about the server hardware, who is UID #1 (Anonymous Coward), and what the day one counts were. I'm pleased to announce that we are hosted on Linode, specifically, two Linode 2048s running Ubuntu 12.04 LTS. Average server load during day 1 was between 10-20% percent, spiking occasionally when we had to flush caches out for config changes.

In the last 24 hours, we used over 7 GiB(!) of bandwidth, served to 3630 distinct IP addresses. As of midnight tonight EST, the daily stats email dropped out of slashd, so I'm going to repost most of it here. Please note the bandwidth stats from slash are skewed as it is behind Varnish. For the most part, only the actual HTML of a given page is uncached for logged in users. Enjoy.

SoylentNews Stats for 2014-02-17

                   UIDs      IPIDs      Pages
        total:        -          -      76164 (2123.8 MB)
 static total:        -          -       5722
gstatic total:        -          -       7827
  grand total:      916       3630      78372 (2140.9 MB)
 secure total:        -          -          0

   total hits:   103059

        posts:      267        333
     comments:      607       1502      17478 (412.1 MB)
        index:      868       1818      14931
     articles:      738       2298      12508 (573.6 MB)
       search:      224        568       1841 (31.4 MB)
     journals:      276        394       1572 (25.2 MB)
        users:      652        812       5390 (105.0 MB)
          rss:       55        172       2208 (17.1 MB)
        other:       43         52       2086 (37.0 MB)


     formkeys:     1732 rows total
     comments:      837 posted yesterday
  submissions:       28 submissions
 sub/comments:     82.1% of the submissions came from comment posters from this day



------------------------
                            Yesterday   | 2 days ago | 3 days ago
    Avg Hits Per Article:          962.2|       168.4|         0.0
Avg Comments Per Article:           58.0|        16.4|         6.2

Pages From RSS By Section
------------------------------------------------
Section		         Pages     UIDS    IPIDS
           Main Page       524       36      169

For Main Page
                  Pages      IPs   Bandwidth    Users
        total:    76164     3600   2123.8 MB      916
        index:    14931     1818    644.1 MB      868
     comments:    17478     1502    412.1 MB      607
     articles:    12508     2298    573.6 MB      738
       search:     1841      568     31.4 MB      224
          rss:     2208      172     17.1 MB       55
        other:     2086       52     37.0 MB      916

-----------------------

Top stories viewed by article.pl:
  1885 14/02/17/0148235 Dopefish   Welcome to the World of Tomorr
   865 14/02/17/1745207 mattie_p   What "News for Nerds" Sites Sh
   725 14/02/17/0745210 Dopefish   Environmentalists Concerned Ab
   613 14/02/16/2220240 NCommander Announcing UTF-8 Support on So
   608 14/02/16/1331209 NCommander Massive Site Progress - Status
   560 14/02/17/145204  LaminatorX Australia spied on US law firm
   531 14/02/17/0818240 Dopefish   MIT Students Subpoenaed Over T
   514 14/02/17/1453216 LaminatorX Find those bugs!
   510 14/02/16/2318242 Dopefish   AIDS Deniers Use Bogus Copyrig
   413 14/02/17/0814201 Dopefish   Wondering Where Microsoft Bing
   393 14/02/16/0731223 mattie_p   Global Warming Finally Debunke
   385 14/02/17/180204  Cactus     Desalination Project in Califo
   292 14/02/17/1734251 mattie_p   Coal Mine Fire Threatens Power
   292 14/02/16/237205  Dopefish   CD Projekt RED Considering The
   277 14/02/12/0715245 NCommander Welcome to SoylentNews!
   268 14/02/16/0327202 Cactus     Kickstarter Hacked
   257 14/02/17/1630228 mattie_p   3D Printers for Kids
   210 14/02/13/0412219 NCommander Reworking Moderation Access
   182 14/02/16/2059230 mattie_p   Amazon Electric Fish Inspire U
   176 14/02/15/189225  mattie_p   Tuna Hearts Damaged by Oil Pol
   175 14/02/16/217209  Dopefish   How Facebook Can Be Addictive
   172 14/02/14/201241  mattie_p   Google Buys Lease On NASA's Fa
   170 14/02/16/2032248 mattie_p   China Plans World's Longest Tu
   167 14/02/16/1115247 Dopefish   PPACA/Obamacare Website Develo
   160 14/02/16/2119211 Dopefish   Germany Beefing Up Spy Defense

-----------------------

Hour        Hits        Hits/sec
  00        1027            0.29    ########
  01        1144            0.32    #########
  02        1282            0.36    ##########
  03        1171            0.33    #########
  04         726            0.20    ######
  05        2891            0.80    ########################
  06        4595            1.28    ######################################
  07        3356            0.93    ############################
  08        3377            0.94    ############################
  09        2940            0.82    ########################
  10        2895            0.80    ########################
  11        2991            0.83    #########################
  12        3622            1.01    ##############################
  13        3992            1.11    #################################
  14        3896            1.08    ################################
  15        3853            1.07    ################################
  16        4019            1.12    #################################
  17        4110            1.14    ##################################
  18        3881            1.08    ################################
  19        4504            1.25    #####################################
  20        4558            1.27    ######################################
  21        4685            1.30    #######################################
  22        4112            1.14    ##################################
  23        4745            1.32    ########################################

 
This discussion has been archived. No new comments can be posted.
Display Options Breakthrough Mark All as Read Mark All as Unread
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • (Score: 2, Insightful) by Zz9zZ on Tuesday February 18 2014, @01:48PM

    by Zz9zZ (1348) on Tuesday February 18 2014, @01:48PM (#1752)

    Thanks for making this happen, I didn't expect a functional alternative so quickly!

    Starting Score:    1  point
    Moderation   +1  
       Insightful=1, Total=1
    Extra 'Insightful' Modifier   0  

    Total Score:   2