Recent Comments

I hardly ever use my tablet, now that I have Shamu-boy Neil.

isn't it unfair to compare a nationwide network of AT&T and later RBOC maintained telco copper, on which POTS ISPs ran freely, with allowing/imposing competition on local evil cable company monopolies who all ran their own infrastructure and connected to each other and the Internet per se only as an afterthought to delivering TV?

Nationwide versus local doesn't make any difference... You wouldn't want to do long-distance dial-up, with the high rates being charged, and couldn't ever do long-distance DSL, so only local really ever mattered for internet access.

As far as who ran what, cable companies got government granted benefits, in the form of monopoly/franchise rights, eminent domain and right-of-way access. They've really had just as much help from the government as telcos when they built-out the POTS, DSL and fiber infrastructure.

Remember, as mentioned, DSL was originally open access, too. You could have Earthlink as your DSL provider, just as easily as Verizon/ATT/etc. That is why SBC (now ATT) DSL was $15/month while cable internet was still usually $50/month. SBC had to directly compete with 3rd party ISPs (like Earthlink) offering DSL over SBC's own lines, and they drove the price down in order to get those customers. It was only a mid-2000s FCC rule change that allowed them to lock things up all over again, after-the-fact.

Finally, cable companies wouldn't be asked to give away their lines for free if the rules were changed... They'd still be charging ISPs a reasonable connection fee and line service/maintenance charges. The cable companies can still continue to provide their own internet service directly, too. They just wouldn't be able use their monopoly to FORCE vertical integration of all their services upon their customers. Would it be okay if cable companies took it a step further, blocked Netflix and Magic Jack, and only allowed their own streaming video and VoIP service to travel over the internet service they provide? Would the fact that they built it out be sufficient justification for their desire to make more money at their customers' expense? It's shades of the same issue.

And did you notice that Time Warner, who has to compete with Earthlink, is among the only ISPs with low-priced internet service plans? You can get internet service from them for $15. Charter doesn't offer anything under $40. Comcast's lowest tier is $40. Cox starts at $48. I'm specifically excluding 12/24-month contract promotional prices for new customers.

You have 500 servers for the same reason you have raid: Redundancy.

And if you let a few of them stay down for no reason, you've got that much less redundancy.

And when your contract with a big company calls for 30% excess capacity, and those couple servers being down during high traffic happened to let it fall under that level, you get to explain exactly why you don't feel the need to properly monitor and maintaining those servers...

Or because being perfectly matched to your workload would mean 499 servers one day (or hour, or minute), 503 the next, 32 on weekends, etc. It really sounds like this guy is arguing from a position of ignorance, and assuming that a big company with lots of servers has frequent failures that they don't care about.

"On occasion" across many hundreds of servers quickly becomes a daily occurance.

Of course I've already said that a dozen times now... Willful obstinance and ignorance doesn't make you look smarter.

You have 500 servers for the same reason you have raid: Redundancy.

I'm not sure when it started, but I feel that pipedot comments have gotten more venomous over the past weeks... And I see a lot of ACs replying in a really rude way, who still get modded up somehow.

I don't care how right you may be. You don't have to be an asshole about it.

You seem to know more about this than I, but isn't it unfair to compare a nationwide network of AT&T and later RBOC maintained telco copper, on which POTS ISPs ran freely, with allowing/imposing competition on local evil cable company monopolies who all ran their own infrastructure and connected to each other and the Internet per se only as an afterthought to delivering TV?

It would be great if competition were there (and in a few cases there are things like Earthlink over Time Warner cable, thanks to legal settlements related to that merger) but it doesn't seem like the same thing as saying "there's already a national voice network and you can use modems on it to talk to the ISP of your choice". There just IS NO similar national network of cable infrastructure on top of which to run all these theoretical competing ISPs.... No?

I guess we're not reading the same site. You found the one really worthless troll post that I can recall seeing here in the last couple of months. And due to the overall low participation many threads consist of half or more ACs, and yet the conversation to my mind has been pretty damn good. (Certainly better than Soylent, which I haven't actually visited in weeks now.) Crank up your settings to make us 0-scorers disappear and tell me just what kind of content is left here.

For the record I've enjoyed talking to you in several threads.

Because you have monitoring systems in place that report such status information, and because any decent admin will configure a service manager to only restart the process a few times in a short period, before giving-up. "Monitoring logs" is only something you do at home... It doesn't scale. You can't do system monitoring that way.

You don't directly monitor logs on production machines past a certain number. Most companies get by with just 2 or 3 email servers and these machines are often monitored manually.

The most rock-solid stable and reliable service will crash, on occasion, in ways that do not need nor would benefit from investigation

Nor from automatic restart since it only happens "on occasion" and anything critical will be monitored. And because we are talking "critical", an administrator will be on call anyway.

Ironically impolite, given your other comments here.

Just calling it like I see it. A spade is still a spade.

If you're getting crashed processes because of hardware errors you can't just restart the process.

A service crashing isn't evidence of a hardware error, and when that does happen, trying to restart it a couple times won't hurt your efforts in any way.

Services will and do crash for no reason... All kinds of strange timing issues will suddenly cause services like crond to just quit after a few hundred days of operation, and continue to work perfectly after being restarted. Go ask someone running a compute cluster how much time they put into investigating every single service crash on their hundreds of thousands of servers...

If its a single Apache server out of 500 thats behind a loadbalancer that can detect the failure and route around, then yeah let it stay dead.

Not actually a good plan... If you have 500 instances of Apache, it's because you NEED 500 instances of Apache, and a couple of them going down is likely to cause measurable slowdowns at peak times. If you have many more servers than you need, you're wasting money to compensate for software limitations.

I think everyone who complains about it should join in with uselessd and see that through. That approach makes sense to me. Forking Debian seems like a waste of time and energy.

I think most everyone can agree on that point.

Ironically impolite, given your other comments here.

In any case, are you saying that a process should restart whenever it fails, for whatever reason? A server can go down for a lot of reasons, anything from a dead harddrive to a CPU overheating because the heatsink got dirty. Or even more "weird" reasons like cosmic rays, in fact, I bet that a large enough cluster of servers could work reasonably well as a cosmic ray detector, just watch for ECC errors in RAM. If you're getting crashed processes because of hardware errors you can't just restart the process. You need to fix the hardware error, which is probably a long, drawn out process since you need to get someone on site first.

If on the other hand your processes are crashing because of stupid stuff, like you forgot to rotate logs, or you've got a memory leak, or some software-only issue, then you're kind of an idiot for letting that get to a production server.

What you do when a service fails usually depends on your setup. If its a single Apache server out of 500 thats behind a loadbalancer that can detect the failure and route around, then yeah let it stay dead. If you only have a single webserver, then yeah save the relevant logs, and auto restart that thing pronto.

I think too many people assume that there setup is the best for every situation when discussing systemd. Well, its not. There are a lot of cases where it makes everything much easier, others where it makes things possible, others where it makes little difference.

Most complainers have setups where it makes little difference.

I think everyone who complains about it should join in with uselessd and see that through. That approach makes sense to me. Forking Debian seems like a waste of time and energy.

Establishing a record that can be easily linked back to you through a single slip isn't much better than not using a handle at all.

You don't need to keep the same one for 20+ years like me, the barrier is quite low. Besides, it's extremely easy not to let personally identifying information slip... Unless you're Hodor.

Again, I think the ACs here are particularly polite and cogent

Ugg... Polite and cogent like him?:

https://pipedot.org/comment/2TD2

I'd go at it from the opposite direction, and say it's rare to see real information or insights from ACs. There isn't a lot of trolling and flaming in general because |. is still a small site. As it gets bigger, I have no doubt the ACs will be just as irritating as on other sites.

I think it depends on the size. I have the first gen Nexus 7 and I almost exclusively use it in portrait mode. With a larger tablet maybe I would find landscape mode more comfortable.

So you've nicely explained EXACTLY why some of us choose to be ACs some, or in my case, all the time. Surely you see that? In your case your handle is you online. Establishing a record that can be easily linked back to you through a single slip isn't much better than not using a handle at all. Might as well succumb to Facebook style real names.

Again, I think the ACs here are particularly polite and cogent (or at least the moderation has made it seem that way), whereas I've found the registered posters less so.

Also, for some reason I have thought until just now that your name was "evilPiper" for some reason, which makes even LESS sense than your actual handle that bryan just pointed out. :)

I really hate what has happened with Linux the latest decenium.

The truth is that we had a more functional desktop in the days of kernel 2.4. During that time even kernel versions meant stable and uneven meant development kernels. With 2.6 Linus broke this scheme as he considered it to slow down the development. We had really many broken kernels. Picking a working kernel was like a gamble. The latest years however he has managed to get some kind of stability back. Just when other things are beginning to get messed up in the whole community.

Then came the whole KDE messing with KDE4. And for those prefering a full blown DE, KDE3 was the only serious option. GNOME was not having enough of features and it was slow. Opening one folder full of pictures caused GNOME to crawl. KDE had a thumbnail cache and behaved a lot better. GNOME crashed often and had all kind of strange bugs.

Also, while most cherish the xorg fork from xf86, I am not happy with it. We had working graphics for many years. Sure the development was very slow comparing to what it is now for xorg, but it was stable. Now many times I have to fight quite a bit to get the graphics to work on even standard computers.

Slowly GNOME became quite useable. Then it was time to mess up that one too in the worst way splitting the already splitted Linux community even more. Now we have GNOME3, Cinnamon, Mate and many other DEs.

Then also came the Pulse audio change. Suddenly, having a working sound system was nothing to take for granted. Still, even today adjusting microphones, selecting recording sources and other settings are non-trivial. All this worked a decade ago. Meanwhile the BSDs never did the OSS->ALSA->Pulse conversion. They continued to develop on OSS and now OSS is able to play from multiple sources and everything works as it should.

Granted SystemV is not perfect. I have seen nasty race conditions with SystemV but I seldom have seen a system having trouble to get to a working condition. Now when systemd got introduced things began to break and things are more difficult to debug.

I hope they fork and bring back some sanity. Do as the BSDs did with OSS: Improving on what you have rather than throwing out and beginning from scratch. Do we want many years of experimentation with the init system rather than improving what we have in a stable pace?

I started out in Unix in the very early 1980s. I was drawn to it because of the toolbox approach and the readability of the configs and startup scripts. I started with BSD 4.2 on PDP 11s, then dualport OSX from Pyramid, followed by some Xenix and Suns running SunOS 3.x (BSD based). Over the years I have worked with pretty much every flavor of Unix, BSD, or GNU/Linux you can think of. Most of them were simple to deal with when moving from one to the other because the configs and startups were readable by anyone with even moderate skill in the art. I could teach a newbie how to understand the startup system and troubleshoot startup problems in very short order.

At some point commercial Unix vendors decided to "make things easier" and we started to get things like SCO Unix overwriting startup files and configs from it's GUI admin tool. It stored stuff in a private DB and just over wrote changes put in files by hand. NOT GOOD. Then of course with Solaris 10, which brought us the great ZFS, Sun decided to go to a monolithic startup database, which if corrupt means the system will not boot. Other commercial vendors have done similar stupid things over the last 10 years.

I think if the Debian core team wants to go against 30+ years of good solid proven engineering then they need to find a surgeon to give them a Rectal craniotomy.

If non-systems administrators can not figure out how to administer a system who the hell cares? Every time I have been called in by startups where the programmers tried to develop code and figure out system administration as well as design how the programs interact with the system it has been a total FUBAR.

The problem these days is everyone who has walked past a computer thinks they are a systems administrator or systems engineer, and vendors as well as, it appears, Team Debian , are feeding that fantasy and in the process destroying the versatility, agility, and maintainability of current POSIX systems. It is exactly this kind of stupidity that makes it more difficult all the time to properly administer systems as well as move between POSIX systems from different vendors.

JUST SAY NO TO FORCED INIT SYSTEM CHANGES.

Fixed. #2TET

Not terribly happy with the name, but had it forever so I keep using it on /. spin-off sites.

I certainly wouldn't want all my late night (and possibly drunken) rants and shouting matches, or all the times I've played devils advocate in a discussion over the past 20 years, to show up (out of context) in a job interview, ALL needing to be explained. So using a real name on most discussion sites just doesn't work for me.

What?! You mean Evil Viper isn't your real name?!

Some daemons are built to recover from crashes and restart their own worker processes. For example, Apache's main pid is mainly in charge of spawning new child processes to do the actual work. The children can even be configured to terminate themselves after serving X number of requests.

However, some daemons, such as mysql end up relying on a shell script to do this task. I've always thought of the mysqld_safe script as being an ugly hack. Wouldn't a real program be a better fit for this? And if you make a generic enough service monitor, couldn't you use it for more than just one program?

The traditional inetd process is another example. The post-fork method of operation is just too slow for modern tasks such as web page serving. But systemd does have some interesting ideas on how to fix it.

I am referring to the claims of that specific AC, yes.

I tend to disregard ACs in general. Back on /. I'd set my preferences to not see their comments unless significantly modded-up, nor ever get notifications about replies from them.

Even a pseudonym keeps people much more honest and polite, and certainly makes for a better community. It's unfortunate that ACs make up such a big proportion of commentators here... People not willing to even minimally stand behind what they say.

ep, you do know we're not all the same AC, right?

This isn't an issue of sysV vs. systemd

No, upstart is in there, too, and that's about all... It got voted down in favor of systemd across the board.

most people start off their argument by saying "we agree system V init needs to be replaced with something better. But this isn't it.

Open source software doesn't start with executives espousing grandiose ideas. Distros choose from what's out there. Somebody needs to churn out some code, and they needed to do it 20 years ago. This has been needed for a long time, and distros can't take a wait-and-see attitude when their big customers needed these features years ago and aren't going to continue waiting.

Committing to systemd is a big jump it's hard to back out of.

Big jumps, that get redone later, are pretty common in Linux. Big initrd changes, devfs to udev, dcop and dbus, oss with esd and arts to alsa and pulse, KMS, lilo grub and grub2, LVM, etc., they're always painful, and often stupid and pointless, but not world-ending.

Lastly, when you think about how much work it is to maintain Debian, threatening to fork it is a BIG undertaking

Actually, it's easy to make the threat. That's the problem with all these discussions... Talk is cheap, and every random misinformed random user can make lots of talk.

Many people are just buying-in to many of the unfounded rumors.

There is more: when a method requires you do steps A, B, C in order, you can do step A and then step C, and, when it is over, do step B (or just pretend that you have done it) because your boss told you to follow that step order or because you have to fulfil a norm that requires this method. With all the various methods there are now, you can invent infinite ways of getting around them. This is what is most rich and powerful about development methods :->

If something is crashing on a production server you have fucked up

Utter nonsense. You're just a kid with a linux box who has no large-scale experience but wants to pretend to be an expert on the internet. EVERYTHING crashes over a long enough time-frame. The most uber-stable and basic simple system software will eventually crash. Across enough servers, you'll see it happening daily.

No, but a NOC is the sysadmins' software "Hands and Eyes". If they're too stupid to do the work necessary, maybe you should pay them more?

If something is crashing on a production server you have fucked up. The end. Production isn't for testing, it's for stable code. The only exemption is if your production network is *so* big that it allows you to segment parts of it for testing, in which case you're accepting that things might break, so you can't really complain about being woken up at 3am to fix a broken service.

If a service is restarting itself all the time, how would you know?

Because you have monitoring systems in place that report such status information, and because any decent admin will configure a service manager to only restart the process a few times in a short period, before giving-up. "Monitoring logs" is only something you do at home... It doesn't scale. You can't do system monitoring that way.

How about you do that when it breaks and stops running the first time or two

Already addressed this, TWICE, in my post. Look for 'crond'. The most rock-solid stable and reliable service will crash, on occasion, in ways that do not need nor would benefit from investigation. Across many hundreds of servers running numerous services, this is a daily occurrence.

When a daemon crashes on a production server, we want to know why. We investigate and fix the problem before restarting.

Already addressed this nonsense, TWICE, in my post. Try again.

Funny... the only time I ever needed data center staff to intervene was after a botched systemd "upgrade".

A NOC isn't datacenter staff.

I'm a professional sysadmin on debian boxes starting from potatoes (I started as a developper). I'm now investigating alternatives for all servers my shop manages because I see too many desktop'ish stuff on fresh install done by my coworkers coming to our servers. I cannot stand to have much in pid 1 and need to reboot when some package updates. Our customer are start-up in early stages and most of the time they come to us when they encounter their first hickup somewhere "in the cloud"... Therefore I can just totally agree with your statement! And thank you Zafiro for this news as I don't read google+...

That's what I was wondering. If a service is restarting itself all the time, how would you know? Oh, by monitoring logs and fixing the problem? How about you do that when it breaks and stops running the first time or two, when you actually notice because you DIDN'T have a Windows style service manager restarting it for you?

Good point. Can't see any obvious remedy to the ability to throw off a thread... Making available the edit-history of a comment should help somewhat though.

Maybe a -1 edit abuse would help?

SysVinit scripts don't have any way to restart services that have quit/crashed. That is EXTREMELY important on servers, and it's absence is a notable missing feature on Linux.

When a daemon crashes on a production server, we want to know why. We investigate and fix the problem before restarting.

ANY service that you need running is "critical" and failure can't be ignored. Right now, these system restarts are typically performed by poorly-paid NOC personnel, who understand less about the services in question than systemd does

Funny... the only time I ever needed data center staff to intervene was after a botched systemd "upgrade".

Automatic service restarts are perfectly safe

Simple minded nonsense that will be easily countered by the first 0day exploit that takes advantage of it.

That's exactly the kind of stuff I was hoping to look into. Thanks for the tip.

Saw this on Google Plus, by the way, for what it's worth.

Evilviper, I think you're at risk of treating this as a duopoly, which is not. This isn't an issue of sysV vs. systemd. In fact, from what I'm reading, most people start off their argument by saying "we agree system V init needs to be replaced with something better. But this isn't it." Your criticisms of system v are on the mark, but many people - me included - would argue that systemd solves those problems but gives you a whole bunch of new problems. That uselessd looks pretty interesting, for example. I dunno.

On my desktop systemd is probably not a big issue, and I'd appreciate the faster boot time. On my servers though I want something that resembles system v init scripts. And while I'd like solutions to the weaknesses you describe here accurately, I don't want systemd to be that solution. I think these fork guys are of the same philosophy - they don't want systemd to become an imposed new standard, and to continue looking around while things evolve. Committing to systemd is a big jump it's hard to back out of.

Lastly, when you think about how much work it is to maintain Debian, threatening to fork it is a BIG undertaking: the hardware support, the enormous package repository, etc. That is a huge project and it's the foundation to Ubuntu, which is the foundation to hundreds of other things. What's that Hindu concept of the universe where there are turtles standing on top of monkeys who are on top of alligators ... all the way down to the elephants? This is like changing out the elephants - no simple feat!

As well, there are some ACs who are so interesting I wish they'd benefit from the karma as I mod them up! We've had some good, anonymous stuff here.

I object to the frequently repeated assertion that system admins don't want systemd, and that it only benefits desktop users.

SysVinit scripts don't have any way to restart services that have quit/crashed. That is EXTREMELY important on servers, and it's absence is a notable missing feature on Linux. There are various add-ons that do this, like daemontools, but they can't replace SysVinit, so you're stuck maintaining two mutually incompatible methods for running services.

I don't care about boot-up times, but not being able to have all system services automatically restarted (without human intervention at 3am), should anything happen to them, is a glaring failure on Linux, putting it a couple decades behind its competitors.

Debugging a system, and/or rebooting it every time it comes up but a network file system didn't mount in-time... Getting paged at 3AM every day, because after 2 years of uptime, crond happened to crash and across hundreds of servers that's a daily occurrence... etc. These are all very important to any server admins, and hardly matter to desktop users.

And to preempt the common responses:

You would NOT want to be paged at 3am just because crond crashed after 2 years of uptime. It's crazy to claim someone needs to investigate every such happenstance. It's also crazy to claim you should rewrite all your startup scripts so every system service is run out of daemontools. After all.. ANY service that you need running is "critical" and failure can't be ignored. Right now, these system restarts are typically performed by poorly-paid NOC personnel, who understand less about the services in question than systemd does. And needing to have NOC folks working around the clock is prohibitive for small shops (who have system admins who would like to sleep through the night) and increases the TCO for large shops, who made need a large number of NOC employees because restarting services becomes a full-time job to the exclusion of other job duties, given enough servers.

Automatic service restarts are perfectly safe. If there was any such issue, it would be looming over daemontools since forever, and the widespread adoption of systemd by every distro out there just serves to show the experts just might know something. Those claiming systemd is bad and useless have to come up with vast conspiracy theories to explain away the enthusiastic and widespread adoption.

I hate to jump into the systemd flame war yet again, where typically the least-informed and least affected shout the loudest. After all, there's no benefit to interrupting the detractors, because every distro out there is already on the side of systemd, and the ranting and moaning on sites like this won't change that.

Sysadmins take 'WAY too much crap from developers.

Almost every dot-com I have worked at or the past twenty-plus years listened 'WAY too much to developers, and allowed things to be rolled out, that shouldn't have been.

Almost every dot-com I've worked in the past ten years was started up by developers who called me in when things got too complicated for them to administer, and develop code, at the same time.

Almost every dot-com I have communicated with in the past two or three years seems hell-bent upon moving everything into 'The Cloud'. WHICH cloud? That changes about once every eighteen months, it seems.

Recruiters are now looking for these new-fangled 'devops' - these would be software developers, who do hardware operations - or is it hardware operators, who do software development? - to do twice the work, at half the price.

By my count, the average number of programming languages, technologies, vendor product lines, networking protocols, etc that one is now expected to master exceeds that of the average UN polyglot translator by a factor of ten.

It's pretty clear the people making up these job descriptions are only counting the money they will save by hiring one person to do two jobs and have NO CLUE what goes into either software development OR hardware operations.

We sysadmins need to start pushing back more!

And so it's great to see collective some movement against systemd.

Even if I am a BSD guy, at home, I still wrestle with Linux, at work - and so I share all of your concerns.

I'd add, though, that the complaint about preferring easily readable shell scripts is not really warranted. It is trivial, in systemd, to set up a service that just executes a bash script. In fact that's one of the first things a tutorial would show you.

Still, if they have a better alternative, they are welcome to work on it and make it available.

As an arch user, you know what I feel about the "evil" of systemd. :p

Still, it's good that these people are willing to put their money where their mouth is. If you don't like the way things are going, the right thing to do in the FOSS world is to just fork.

How about some utility-type distros once in a while?

* http://www.linux.com/directory/Distributions/livecd

Haven't ever seen System Rescue CD, OpenWRT or Zentyal mentioned here. Even "Android-x86" and "Chrome OS Linux" could be... weird and interesting subjects.

IMHO, the trick is to find a brand that has and maintains quality standards. I typically start by looking for companies that provide longer warranties than all their competitors. After that, sites that maintain fair (ie. unmoderated) customer reviews like Amazon, can be extremely helpful. It can also help to shop at smaller stores, where they stock only one or two models of each item, and their profits are directly affected if there are high return rates on an item, unlike big-box stores which are happy to stock cheap junk (Walmart/Best Buy).

Price drops are so extreme in electronics that buying early in the product cycle is often many times more expensive than buying later and just replacing it several times with similarly cheap items. Of course, finding inexpensive but reliable products later is an even better solution.

I suspect there are a few obsessives "on the spectrum". I've done my share of whining too but I agree completely with your frank assessment of this poster.

I continue to maintain that the ACs here are overall better contributors than the registered users, with a few notable exceptions on each side. Not sure why. Maybe just good moderation.

Thank you for keeping tabs on this, Zafiro. I probably would not otherwise know about it.

It seems like a losing cause as systemd is some kind of brain prion infecting distro devs, but here's hoping for traction.

Sorry, the French word for developer has two p's, which is where the confusion came from. I guess I will have to work on my chair dodging skills.

I did notice "ensure" should have been used after the fact, but I can't edit it now. It's really bugging me. >_<

Dude, it's "developers", not "developpers". You best be careful or Ballmer will throw a chair at you!

Also, in the initial article, it should be "ensure", not "insure". "Insure" is for insurance. "Ensure" to make sure.

I think the general mandate here is science and technology, and not so much of that drama-bait being posted elsewhere, so this is a direct hit. As for Friday Distro, I'm looking around for another good one. Last couple rolls of the dice have all been "Ubuntu but with LXDE and a great theme!" type distros - boring, and not that useful!