Hardware is Cheap, Programmers are Expensive

Open discussion on or around SGI; software/hardware related posts should go in the appropriate subforum.
Forum rules
Any posts concerning pirated software or offering to buy/sell/trade commercial software are subject to removal.
User avatar
Geoman
Donor
Donor
Posts: 820
Joined: Thu May 26, 2005 3:37 am
Location: Munich, Germany
Contact:

Hardware is Cheap, Programmers are Expensive

Unread postby Geoman » Sun Mar 26, 2017 6:43 am

https://blog.codinghorror.com/hardware-is-cheap-programmers-are-expensive/

Given the rapid advance of Moore's Law, when does it make sense to throw hardware at a programming problem? As a general rule, I'd say almost always.

Consider the average programmer salary here in the US:

http://royal.pingdom.com/2009/07/15/the-problem-with-using-hardware-to-compensate-for-slow-software/

...one of the reasons of course that retrocomputing even with most advanced SGIs won't make any sense today.
:Indy: :O2: :O2: :Indigo: :Indigo2IMP: :Octane: :Octane2: :Octane2:
SGI - the legend will never die!!

User avatar
Dodoid
Posts: 511
Joined: Mon Jul 04, 2016 1:36 pm
Location: Ottawa, Canada
Contact:

Re: Hardware is Cheap, Programmers are Expensive

Unread postby Dodoid » Sun Mar 26, 2017 9:09 am

Check out MenuetOS as an example of this. Written by one guy as a hobby project, contains stuff like games, a web browser, transparency support for window themes, and drivers for common hardware. Supports up to 64gb of ram, lots of cores, x64, audio, etc. The entire OS, including all of this, is 1.4mb and can even boot from a floppy. On the other hand, Windows 7 with all the updates is close to 40GB.
:Fuel: :Octane: :Octane: :Octane: :O2: :Indigo2: :Indigo2: :Indy: :Indy:

User avatar
ClassicHasClass
Donor
Donor
Posts: 2021
Joined: Wed Jul 25, 2012 7:12 pm
Location: Sunny So Cal
Contact:

Re: Hardware is Cheap, Programmers are Expensive

Unread postby ClassicHasClass » Sun Mar 26, 2017 9:37 am

Hey, I'm reading this on a 1GHz iMac G4, so I'm doing my part.
smit happens.

:Fuel: bigred, 900MHz R16K, 4GB RAM, V12 DCD, 6.5.30
:Indy: indy, 150MHz R4400SC, 256MB RAM, XL24, 6.5.10
:Indigo2IMP: purplehaze, 175MHz R10000, Solid IMPACT
probably posted from Image bruce, Quad 2.5GHz PowerPC 970MP, 16GB RAM, Mac OS X 10.4.11
plus IBM POWER6 p520 * Apple Network Server 500 * RDI PrecisionBook * BeBox * Solbourne S3000 * Commodore 128 * many more...

User avatar
Krokodil
Donor
Donor
Posts: 455
Joined: Fri Apr 17, 2015 2:32 pm
Location: The House of Particular Individuals

Re: Hardware is Cheap, Programmers are Expensive

Unread postby Krokodil » Sun Mar 26, 2017 6:09 pm

Dodoid wrote:Check out MenuetOS as an example of this. Written by one guy as a hobby project, contains stuff like games, a web browser, transparency support for window themes, and drivers for common hardware. Supports up to 64gb of ram, lots of cores, x64, audio, etc. The entire OS, including all of this, is 1.4mb and can even boot from a floppy. On the other hand, Windows 7 with all the updates is close to 40GB.


Yes, it's a pity that will probably never make it past the hobby OS level.
:Octane2: - :O2: - :Octane: - :Indigo2IMP:

User avatar
Elf
Donor
Donor
Posts: 85
Joined: Wed Oct 19, 2016 9:54 pm
Location: Pacific Northwest (US)

Re: Hardware is Cheap, Programmers are Expensive

Unread postby Elf » Sun Mar 26, 2017 9:49 pm

I think the problem with the "hardware is cheap" mentality is that it seems to assume the function between programmer time and attention vs infrastructure resources consumed is linear. Pay a little less attention to code, spend a little more on infrastructure, and vice versa. From working at several software shops, I find that when left to their own devices without optimization and architecture as a ranking priority, even "good" programmers tend to produce the kind of O(n^2) - O(n^3) (and greater) problems that you simply can't buy your way out of.

Real world example: A programming team is working on a SaaS application (J2EE backend + HTML5/JS frontend) that displays and performs parametric searches on a popular business data set. The dataset is few million rows of various types; nothing amazing. A few thousand active customers per day. The infrastructure is hosted in Amazon Web Services, designed properly greenfield, giving the ability to effortlessly scale the underlying infrastructure, as long as the application architecture accommodates it . No technical debt on the systems side.

Through a complete lack of caching and a severe abuse of SQL (20+ page long queries, ugly nested sub-queries many times deep, etc.), the application now runs on a set of r3.8xlarge RDS instances -- the largest size of vertical scalability available in Amazon. The end of the line. It regularly saturates this largest-size-available SQL database to 100%, causing the site to become unusable. Many queries which are directly relied on to serve up interactive data regularly run for minutes at a time. Caching the back-end results will not help, as the results of most API calls have highly user specific data mixed in, due to bad API design. There is no ability to horizontally scale the SQL master role (read/write functionality), due to hard limitations in the ACID / CAP commitments that an SQL database inherently makes. There is an ability to create read replica databases to spread out the read-only load, but the application never had the concept of distinguishing reads from writes, and is unable to use them, so even this limited model of horizontal scalability is impossible.

The developers put a lot of what should have been application logic into monstrous SQL queries because it reduced the amount of Java code they would otherwise have to write, to combine multiple different sets of results (or execute business logic on those results) before sending it back to the front end. It is now so entrenched that not only would it be a massive multiple-month effort to refactor the old queries, but it has become habit forming, and newly developed code still resembles the old code, even though the patterns have been recognized as problematic.

The high costs of running the infrastructure are noticed at the executive and board level as out of proportion for the work the application is doing. The product becomes unreliable (due to excessive resource consumption) for customers who pay thousands of dollars a month, despite near zero downtime in the actual supporting infrastructure. The application should be wholly rearchitected, including ditching the ugly queries entirely and presenting data out of a horizontally scalable non-SQL document store (like DynamoDB), but the time and attention to do this would set back feature development for months. People would throw more money at the infrastructure out of desperation, but there is literally no way to spend themselves out of the hole they are in.

I wish I could say that was a unique story, but every company I've worked at has had at least one or two projects that ended up like that. There is a lot that is good about the modern software development ecosystem, but I've increasingly seen software "engineers" (nee, developers) assume a lazy and undisciplined approach to their craft that simply wouldn't fly in any other technical field. Common attitudes like "we can buy our way out of our inefficient code" are just another deluded part of that mindset, where employees see themselves as highly paid rockstars and with an average tenure of about 1.5 years, usually move on before too many of their mistakes catch up with them.
:Indy: :Indy: :Indy: :Indigo2: :Indigo2IMP: :O2: :O2: :Octane: :Octane: :Fuel: :Tezro:
:Indy: [x19] :Indigo: [x7] :O2: [x4]

robespierre
Posts: 1523
Joined: Mon Sep 12, 2011 2:28 pm
Location: Boston

Re: Hardware is Cheap, Programmers are Expensive

Unread postby robespierre » Sun Mar 26, 2017 11:44 pm

Programmers working in HPC have always spent more effort to ensure high quality and performance, including using esoteric languages and "old-fashioned" development methodologies. The reason is that when you need the highest performance possible, "it will be faster and cheaper on next year's hardware" isn't an acceptable answer. When your computer costs $20M, the relative economics is still the same as in the 1960s: programmers are cheap, hardware is expensive.

Elf: your real world example perplexes me. This programming team of "rockstars" with no technical debt didn't see fit to hire a DBA? Without knowing the details, 20+ page SQL queries point to an undisciplined schema.
:PI: :O2: :Indigo2IMP: :Indigo2IMP:

User avatar
commodorejohn
Posts: 627
Joined: Tue Oct 02, 2012 1:22 pm
Contact:

Re: Hardware is Cheap, Programmers are Expensive

Unread postby commodorejohn » Sun Mar 26, 2017 11:59 pm

And this is why I do my own small part by actually trying to learn the material for which I'm developing code and come up with sensibly optimized solutions, and not relying on everybody else to do my work for me and accepting any preexisting efforts as the gospel truth - because no matter how many times the the open-source community repeats their mantra about duplication of effort, the fact is that I really have no reason to believe that whoever developed Magic Library X knew the material any better than I do, let alone had my own specific applications and goals in mind when developing it.

robespierre wrote:Elf: your real world example perplexes me. This programming team of "rockstars" with no technical debt didn't see fit to hire a DBA? Without knowing the details, 20+ page SQL queries point to an undisciplined schema.

If there's one thing I've learned from reading The Daily WTF, it's that nobody ever, ever, ever hires an actual competent DBA. If there's two things I've learned from reading The Daily WTF, it's that everybody who is not an actual competent DBA should be forcefully kept as far away from SQL queries as possible, by armed guards if necessary.
Computers: Amiga 1200, DEC VAXStation 4000/60, DEC MicroPDP-11/73
Synthesizers: Roland JX-10/SH-09/HS-80/MT-32/D-50, Yamaha DX7-II/V50/TX7/TG33/FB-01, Korg MS-20 Mini/ARP Odyssey/DW-8000/M1, Ensoniq SQ-80, E-mu Emax HD/Proteus-2, Casio CZ-5000, Moog Satellite, Sequential Circuits Prophet-600

User avatar
Elf
Donor
Donor
Posts: 85
Joined: Wed Oct 19, 2016 9:54 pm
Location: Pacific Northwest (US)

Re: Hardware is Cheap, Programmers are Expensive

Unread postby Elf » Mon Mar 27, 2017 12:17 am

robespierre wrote:Elf: your real world example perplexes me. This programming team of "rockstars" with no technical debt didn't see fit to hire a DBA? Without knowing the details, 20+ page SQL queries point to an undisciplined schema.

The greenfield infrastructure build-out had no technical debt (i.e. there was no infrastructure impediment to horizontal scalability), not the application development side which certainly had plenty and which was not ready to scale.

There was a DBA; actually a very impressive one, at that. He worked in the performance optimization group at Sybase, worked on various portability groups at Oracle, at national labs with HPC groups, etc. But the explanation is a very common one: the design and use of database queries and schema was seen as the domain of the developers, until it was much too late. The DBA was just told to keep the servers up. I think it was only after a year and a half in that there was any DBA review process in place for new queries, never mind the DBA designing them from the start. Then he became bitter and decided that nothing was ever going to change, and so didn't bother to force the issue of bringing things in line.

It wasn't just an undisciplined schema, which was one of a multitude of issues. Much of it came from trying to implement business logic within the SQL query. Things that would have been better implemented as a few successive simple queries with application code in the middle became monstrous multi-page disasters with CASE statements, multiple nested sub-queries, etc.

But honestly, what is a query writing DBA except a specialized developer? The mistakes made in the language of SQL aren't unique to the database domain. The misunderstandings that cause such things to happen just as easily manifest themselves in any other sufficiently abstract language.

That aside, the use of a traditional SQL database was a poor technology choice up front. A horizontally scalable document store like DynamoDB would have been a much better fit.
:Indy: :Indy: :Indy: :Indigo2: :Indigo2IMP: :O2: :O2: :Octane: :Octane: :Fuel: :Tezro:
:Indy: [x19] :Indigo: [x7] :O2: [x4]

robespierre
Posts: 1523
Joined: Mon Sep 12, 2011 2:28 pm
Location: Boston

Re: Hardware is Cheap, Programmers are Expensive

Unread postby robespierre » Mon Mar 27, 2017 12:44 am

A DBA is just a specialized developer (who conventionally has "ownership" over the database files, so can be mistakenly seen as just Ops instead of a designer). But (in theory) their training is more standardized than typical developers. Is there any specific theory that you can rely on a developer to know? I'm having trouble thinking of any. Extreme or Scrum are not theories in this sense. It's the difference between "a craft" and "a discipline".
:PI: :O2: :Indigo2IMP: :Indigo2IMP:

User avatar
Elf
Donor
Donor
Posts: 85
Joined: Wed Oct 19, 2016 9:54 pm
Location: Pacific Northwest (US)

Re: Hardware is Cheap, Programmers are Expensive

Unread postby Elf » Mon Mar 27, 2017 1:32 am

robespierre wrote:Is there any specific theory that you can rely on a developer to know? I'm having trouble thinking of any. Extreme or Scrum are not theories in this sense. It's the difference between "a craft" and "a discipline".

Right, I would classify Extreme or Scrum as project management methodologies rather than technical expertise. Certainly good to be familiar with as they have direct application to the field, but not quite in the same domain as technical knowledge.

Unfortunately your vocabulary was very well chosen! There is much I would expect someone to know, as a professional developer, but not much I can rely on them knowing. This is very different from traditional science and engineering fields, where a Masters or Doctorate degree in a scientific field implies a working familiarity with that field's theory and research. Likewise a Professional Engineering certification, in an engineering field, indicates not only standards tested competency in both theory and practice, but also an ongoing commitment to continuing education in their field's advancements. Even craft-practicing tradespeople are still held to standards and best practices (e.g. the National Electrical Code).

Those are the standards people get held to when designing roads, aircraft, datacenters, or even computer chips, but unfortunately software is a much different matter. Instead of "this is how you design a bridge according to mechanical engineering principles and known industry best practices," your average software programmer would say something like: "well, I copied and pasted parts of a few different bridge designs off of tutorial websites and StackExchange for NodeBridge 0.1 alpha, which is what everyone is using, and it ran and passed QA, so let's ship it."

I mean, I would idealistically expect mid-level to lead developers to know and apply things like:
  • Basic datastructures (lists / queues, trees, graphs)
  • Algorithmic efficiency (linear vs. exponential time, etc.)
  • Space vs. time tradeoff principles
  • A basic understanding of the different types of languages, imperitive, declarative and procedural
  • Features and conventions of the language they are operating in
  • Debugging and profiling tools for their language
  • Object oriented programming principles and design patterns (e.g. SOLID principles)
  • A top to bottom understanding of the abstractions that they use (i.e. being able to at least roughly visualize their code's execution in machine language)
  • Basic concurrency and parallelization techniques (threading and thread related data structures, inter-process communication, shared memory vs. mailboxing/messaging, statelessness, divisible workloads)
  • Different types of datastores and tradeoffs (relational, document, key/value, etc.)
  • A basic understanding of IP and friends (TCP, UDP, ICMP), routing, and sockets
  • The advancing state of their language's ecosystem, in terms of frameworks, etc.
  • Common application architecture and service design patterns
  • Automated testing (e.g. unit tests, mocking, automated UI and acceptance testing frameworks, performance testing suites, etc.)
  • Best practices for documenting their work, from source code to APIs and application architecture
For someone that makes near or at six figures, I think it's only responsible to meet similar standards to other engineering professionals.

But to your point, it is always hard to say what any one person actually does know. So many people these days get by just plugging together lego blocks until something turns on. I'm not really advocating for having some sort of professional standards organization for software developers, despite the effect that has had on other fields. I think that would be a bit heavy handed. On the other hand, I can't say I'm happy with the results today either.
:Indy: :Indy: :Indy: :Indigo2: :Indigo2IMP: :O2: :O2: :Octane: :Octane: :Fuel: :Tezro:
:Indy: [x19] :Indigo: [x7] :O2: [x4]

User avatar
tomvos
Donor
Donor
Posts: 127
Joined: Fri Jul 04, 2008 1:08 pm
Location: Aachen, Germany, Europe
Contact:

Re: Hardware is Cheap, Programmers are Expensive

Unread postby tomvos » Mon Mar 27, 2017 2:17 am

One aspect I like to add to this discussion: As long as you can solve a performance issue with faster hardware, you make it difficult for a developer to become a better developer. Because you don't become a better developer simply by switching into the Now-I-write-fast-code-instead-of-slow-code-mode.
As with every profession, it requires constant exercise to become good at something. And you need some challenges to improve, otherwise in most cases you simply apply the same solution that did work before.

Regarding the DBA topic. Even bad SQL can be improved a lot. SQL describes what to do, not how to do it. That's what the database does for you. Modern relational databases have a lot of tools available to tune SQL (However I'm surely biased here because I prefer Oracle over most NoSQL databases).


My favorite technique to tune performance is based on this article:
https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html
#1 trick: GNU grep is fast because it AVOIDS LOOKING AT
EVERY INPUT BYTE.

#2 trick: GNU grep is fast because it EXECUTES VERY FEW
INSTRUCTIONS FOR EACH BYTE that it *does* look at.

It turned out, this is very good advise. I succeeded in tuning quite a lot of systems by simply skipping non neccessary workload.
:Fuel: :Octane2: :O2: :O2: :1600SW: :Indy: :Indy:
Where subtlety fails us we must simply make do with cream pies.

User avatar
spiroyster
Donor
Donor
Posts: 135
Joined: Thu May 03, 2012 8:24 am
Location: Somerset, UK

Re: Hardware is Cheap, Programmers are Expensive

Unread postby spiroyster » Mon Mar 27, 2017 2:45 am

Ruby on rails need not apply... :lol:

So um... the first graph on the OP linked article o.0 Make sense to anyone?

User avatar
kjaer
Posts: 417
Joined: Wed May 07, 2008 7:47 pm
Location: Seattle, WA
Contact:

Re: Hardware is Cheap, Programmers are Expensive

Unread postby kjaer » Mon Mar 27, 2017 9:54 am

:OnyxR: :IRIS3130: :IRIS2400: :Onyx: :ChallengeL: :4D220VGX: :Indigo: :Octane: :Cube: :Indigo2IMP: :Indigo2: :Indy:

User avatar
commodorejohn
Posts: 627
Joined: Tue Oct 02, 2012 1:22 pm
Contact:

Re: Hardware is Cheap, Programmers are Expensive

Unread postby commodorejohn » Mon Mar 27, 2017 10:51 am

tomvos wrote:My favorite technique to tune performance is based on this article:

Hah, that's great. Choice quote:
The key to making programs fast is to make them do practically nothing. ;-)

This ought to be the epigram for every programming textbook.
Computers: Amiga 1200, DEC VAXStation 4000/60, DEC MicroPDP-11/73
Synthesizers: Roland JX-10/SH-09/HS-80/MT-32/D-50, Yamaha DX7-II/V50/TX7/TG33/FB-01, Korg MS-20 Mini/ARP Odyssey/DW-8000/M1, Ensoniq SQ-80, E-mu Emax HD/Proteus-2, Casio CZ-5000, Moog Satellite, Sequential Circuits Prophet-600

User avatar
foetz
Moderator
Moderator
Posts: 6501
Joined: Mon Apr 14, 2003 4:34 am
Contact:

Re: Hardware is Cheap, Programmers are Expensive

Unread postby foetz » Mon Mar 27, 2017 12:12 pm

Elf wrote:Common attitudes like "we can buy our way out of our inefficient code"

oh i can confirm that unfortunately


Return to “SGI: Discussion”

Who is online

Users browsing this forum: No registered users and 3 guests