General discussion

Hardware and software: benchmarking

Since I have and do use multiple systems on my boxes, I am very aware of system performance across the board; however, this is not true for most computer hardware related sites. What reasons are there for not using at least three unrelated operating systems to test performance?
I can only think of: apathy, sloth, and wanton ignorance.
Not even this site will test to such an extent.
Of the tests: Stress the box! Playing a game requires less of the cpu and memory than making one does. There is the test, in the creation not the execution of an application. Using systems that require little memory- such as minix- to those that are demanding- such as windows- on the same box would give a fair and unbiased review of the product. Add limiting memory to that and you have a great way of testing.

Discussion is locked
Follow
Reply to: Hardware and software: benchmarking
PLEASE NOTE: Do not post advertisements, offensive materials, profanity, or personal attacks. Please remember to be considerate of other members. If you are new to the CNET Forums, please read our CNET Forums FAQ. All submitted content is subject to our Terms of Use.
Reporting: Hardware and software: benchmarking
This post has been flagged and will be reviewed by our staff. Thank you for helping us maintain CNET's great community.
Sorry, there was a problem flagging this post. Please try again now or at a later time.
If you believe this post is offensive or violates the CNET Forums' Usage policies, you can report it below (this will not automatically remove the post). Once reported, our moderators will be notified and the post will be reviewed.
Comments
- Collapse -
I know and have known of people

whose sole aim appears to be to achieve the leanest, fastest and most indestructible computer on the planet. They spend almost all of their time tweaking and testing and very little...if any...having it do useful tasks. Happy

- Collapse -
unbiased benchmarking

I'm stating that a computer cannot be properly reviewed if only one operating system is used for benchmarking. This refers to the i386 and amd64 platforms but- come to think of it- let's add all available.
Think of it as a car that can use five different types of fuel. What is the mileage per unit of fuel? You can't answer this properly if you only use one type of fuel to test.
You can't properly test and review hardware if you only use one environment.

- Collapse -
I'm not sure that's entirely so

Much, I have to think, is in a computer's design to be best it can be in a specific area for a specific type of use. While it may perform a broad range of tasks quite well, it will be optimized for a more limited range. Change a part here and there and performance improves in some areas and suffers in others. Some processors work better with server applications than in running office suites and visa versa....and these will likely use a different operating system to take advantage of what the processor has to offer.

Since you used cars as a parallel, allow me also to do the same. A racer built for Daytona is very good at making left hand turns at high speed. That's in its design and the big oval is its "operating environment. Now, lets put this big bore machine against a smaller grand prix or formula 1 car with an engine 1/2 its size. Put them both on a twisty track with long straights and hairpin turns....the operating environment these smaller high reving engines work best in by design. Let's toss a "funny car" into the race too. It can surely gobble either one of these on a short straight track....the operating environment it was designed for. Then lets benchmark each car in the three operating environments. No one's gonna win all three events. It's also likely that the biggest winner in the acceleration contest will be the biggest loser in the other races.

So, I'd say a computer can very well be reviewed with only one operating system. It will be pitted against other computers in that same operating environment. Change the environment and change the hardware to match it and run computers against each other that way too. The potential user should know what they want to accomplish and make the selection accordingly. If you need a loaf of bread and the store is straight up the road, you don't need a car with quick steering that can pull 5 Gs in the corners to get you that PB&J sandwich. Happy

- Collapse -
cars as a parallel

*another trip down memory lane*

a few years back in the UK, they put a Mini-Cooper (1000cc engine) up against a souped-up Ford Galaxy(sp) (7-8 litre engine if i recall correctly) on a muddy moto-cross 'almost oval' circuit, the Mini won hands down

jonah

.,

- Collapse -
oops! Galaxie
- Collapse -
That muddy moto cross

I'm willing to bet they got that Mini all cleaned up days before the Ford looked presentable too. When younger, I competed in autocross events with the SCCA. Car's were classed for the sake of competitiveness on make shift tracks in big parking lots or abandoned oval tracks. There was some big American iron..."Cudas", Pontiac Trans AMs, Corvettes, etc. As well some European cars with much smaller engines. The big guys made the most noise but never turned in times close to the more nimble British, German and Italian cars. But when the event was over and all hit the highway, these muscle cars disappeared in a cloud of exhaust fumes and burning rubber leaving the other drivers choking on their smoke. Happy

- Collapse -
Os reviews

Cute analogy with the pb&j sandwich.

- Collapse -
Tests and retest

Let's take the same set of programs and- using on the same box with the same hardware- run them to use an extensive amount of memory.
Let the tests be done with memory at different limits: 256, 512, 1024, 2048, 4096 all being in megabytes. This would be the first set of tests.
Have only a barebones set up for each os.

Second set of tests: same programs run while a few user processes are running. Make each system with more options.
Third set of tests: have intensive use of cpu by user programs- a realistic and common scenario- and run the testing programs.

You will have the results under different environments.

CNET Forums