Home Tutorials Download Beta Store Forum Documentation KnowledgeBase Wiki Blog

ShiVa3D

Return to W.I.P

Shiva Benchmark

Work in progress

Shiva Benchmark

Postby error454 » 19 Feb 2011, 07:35

I had the need to benchmark shiva on a few mobile devices to get a target vertex count for our scenes. I made the .ste available if anyone is interested. If you have any suggestions I might consider adding them, particularly if my stats are skewed.

For now, this provides me a very reasonable vertex count that is believable.
http://wp.me/pO3oe-bW
error454
Platinum Boarder
Platinum Boarder
 
Posts: 666
Location: Portland, OR USA

Re: Shiva Benchmark

Postby Xdust » 19 Feb 2011, 15:24

error454 wrote:I had the need to benchmark shiva on a few mobile devices to get a target vertex count for our scenes. I made the .ste available if anyone is interested. If you have any suggestions I might consider adding them, particularly if my stats are skewed.

For now, this provides me a very reasonable vertex count that is believable.
http://wp.me/pO3oe-bW



Great job on this project. And now that I have been playing with it for a little. I realized this is just the sort of thing that is needed. I wont be spending much time trying to "Optimize" texture size. The payoff just isn't there. Especially when you consider how much "eye-candy" is lost for such meager performance gain.

It shows some things I didn't figure on. Smaller textures didn't help nearly as much as I thought in a Browser, or PC.

As far as the "Bad" variable and event handler names, they are fine. Except maybe the var "polyLimitHysteresis". Hahaha
Xdust
Gold Boarder
Gold Boarder
 
Posts: 194
Location: Columbia, PA, USA

Re: Shiva Benchmark

Postby dpharaoh » 19 Feb 2011, 15:43

By vertices, did you mean polys?
dpharaoh
Platinum Boarder
Platinum Boarder
 
Posts: 1520
Location: New Bedford, MA

Re: Shiva Benchmark

Postby error454 » 19 Feb 2011, 17:04

Xdust wrote:It shows some things I didn't figure on. Smaller textures didn't help nearly as much as I thought in a Browser, or PC.

This is something I am still a little skeptical about. The export process in Shiva is a black box to me, I have the feeling that when I select Palm as a target profile that it shrinks all of my textures to 64x64. Just a feeling.

Also, I need to experiment with an OpenGL 1.1 and 2.0 build. Right now I think my Shiva Tools build is built for 1.1 and I'm not sure what difference this will make.

Xdust wrote:As far as the "Bad" variable and event handler names, they are fine. Except maybe the var "polyLimitHysteresis". Hahaha

Haha, I was trying to figure out how to name a variable whose purpose was to average out the spikes in poly count. Thinking of how thermostats work, hysteresis was the 1st word that came to mind :)

dpharaoh wrote:By vertices, did you mean polys?

Some people think in vertices, others in polys. I display both in the benchmark.
error454
Platinum Boarder
Platinum Boarder
 
Posts: 666
Location: Portland, OR USA

Re: Shiva Benchmark

Postby dreamora » 19 Feb 2011, 17:12

Interesting breakdown :)


As for triangles vs vertices:
Triangles offer no meaningfull informations normally, because they don't impact the performance directly.

There is no triangle -> vertices relationship due to smooth and non smooth borders.
3 Triangles can be anything from 5 to 9 vertices technically.
Also triangles don't cause costs in bone skinning and vertex lighting, vertices do.

The thing where triangles are indirectly of importance are amount of rendered and overdrawn pixels and shaders and textures used to render them but those can not easily be calculated and measured in a quantitative way.

So when you do benchmarking and performance evalulation for mobile, its normally "useless" to look at triangles as they don't yield you any information that you can use to decide where to optimize or not optimize something.
I'm no stonetrip representative, just a happy board member like you :)

If you find my portfolio interesting, feel free to contact me or follow my shares on my twitter stream or my game development blog
User avatar
dreamora
Platinum Boarder
Platinum Boarder
 
Posts: 844

Re: Shiva Benchmark

Postby error454 » 19 Feb 2011, 17:35

dreamora wrote:So when you do benchmarking and performance evalulation for mobile, its normally "useless" to look at triangles as they don't yield you any information that you can use to decide where to optimize or not optimize something.


Thanks for weighing in. You say that polys are only indirectly related to overdrawn pixels and the methods used to render them. If mobile devices could use shaders, would polys be more relevant for a benchmark?

I'm curious why the hardware industry seems to use polygons / sec as a standard metric - is this just marketing or is it because of the 2nd gen effects? i.e. Microsoft bragging about 500E6 polys/sec vs the PS3 275E6
error454
Platinum Boarder
Platinum Boarder
 
Posts: 666
Location: Portland, OR USA

Re: Shiva Benchmark

Postby Xdust » 19 Feb 2011, 20:16

error454 wrote:
Xdust wrote:It shows some things I didn't figure on. Smaller textures didn't help nearly as much as I thought in a Browser, or PC.

This is something I am still a little skeptical about. The export process in Shiva is a black box to me, I have the feeling that when I select Palm as a target profile that it shrinks all of my textures to 64x64. Just a feeling.

Also, I need to experiment with an OpenGL 1.1 and 2.0 build. Right now I think my Shiva Tools build is built for 1.1 and I'm not sure what difference this will make.

Xdust wrote:As far as the "Bad" variable and event handler names, they are fine. Except maybe the var "polyLimitHysteresis". Hahaha

Haha, I was trying to figure out how to name a variable whose purpose was to average out the spikes in poly count. Thinking of how thermostats work, hysteresis was the 1st word that came to mind :)

dpharaoh wrote:By vertices, did you mean polys?

Some people think in vertices, others in polys. I display both in the benchmark.


I have to tell you, I was playing with the Editor, PC, and WebPlayer. And I saw almost no difference in performance of 1024 textures vs 256 Textures. There was some difference, but nothing like I thought I would see.

I reached my Target with approx 26000 polys with 1024 Textures.
And reached with approx 26,800 with 256 textures.

Makes me wonder if you are not correct, and that Shiva makes all the textures the same size when run.

OR, the the machines I tested on are very good with One texture. Which is another possibility. Wonder if you start using multiple textures if things would change.

Maybe video cards are very good at optimizing one big texture.
Xdust
Gold Boarder
Gold Boarder
 
Posts: 194
Location: Columbia, PA, USA

Re: Shiva Benchmark

Postby error454 » 19 Feb 2011, 21:57

I was considering this more and think the benchmark is flawed.  There aren't games with only a single texture and the engine probably only has to store a single copy in memory for the entire scene.

I need to create an option for multiple unique textures.
error454
Platinum Boarder
Platinum Boarder
 
Posts: 666
Location: Portland, OR USA

Shiva Benchmark on iDevices

Postby TomHa.Z » 23 Feb 2011, 08:21

Hi there! First post in here ;) so first thanks a lot for sharing your benchmark project.

I ve build the bench app on a ipod 2nd gen device i got some "strange" results like
-no texture , 30 fps , no lighting : 2000 polys 4000 verts
-128 , 30 , no lighting : 700 polys 1400 verts
-128 , 30 , RT lighting : 500 polys 1000 verts

So i guess there are some "perf killer" in the code, i ve removed one call to log.message() (wich i ve discover to be one "perf killer" on my project), but only gave like 10% FPS more.

What i ve check:
-texture are compressed
-use bilinear mipmap
-use vertex lighting
-use single texture map

I am currently in the process of trying to get those 30 FPS for my project (targeting the IPod touch 2 and up, so basicaly from the lowest possible iDevice) and an app like yours sounds really helpfull to get figures about the 3d hardware cap of the device when using shiva.

I ll try to mod your project so i can try different texture settings. I am a bit lost about the right way to do opacity, and get sharp enought textures (like mipmap bias as no effect on iDevice??).
If you agree i may share it later? (if it comes out clean enought for puplic reading ;P)

Maybe sharing the numbers out of your app here in this thread would be helpfull to others too?
TomHa.Z
Junior Boarder
Junior Boarder
 
Posts: 30

Re: Shiva Benchmark

Postby Pedro » 23 Feb 2011, 09:50

Very interesting Benchmark ^^

Like TomHa.Z said, remove the log.message(), that will increase the fps.

You could add for testing :

number of lights
dynamical lightning per vertex/per pixel lighting
Active Effect Map 1
Active Normal Map
Active Specular Map
User avatar
Pedro
Junior Boarder
Junior Boarder
 
Posts: 37

Re: Shiva Benchmark

Postby shivachristy » 23 Feb 2011, 10:33

Hi, This is exactly what Im looking for.

I have released a game on the iphone 4 with less than 4000 polys

I had no idea what the maximum amount would be obviously a very complex calculation.

Here are my thoughts

We should have some guidelines on volumes etc. I am developing a game on my PC for the iPhone and intend to change the original 3D elements with much smaller ones. I have read all I can on optimization. Then found that reducing ambience parameters completely altered the frame rate.

What is the difference between an App on an iPhone and one on an iPad

is it a factor of 3?

I would love to have some solid figures but I dont think it will happen.
http://cur.lv/1x0wx via an ad! Thanks
User avatar
shivachristy
Platinum Boarder
Platinum Boarder
 
Posts: 1035
Location: UK

Re: Shiva Benchmark

Postby dreamora » 23 Feb 2011, 12:28

iphone 4 is basically as powerfull as ipad if you run on retina resolution as the pixel amount on iphone 4 / itouch 4 is only 20% smaller, yet the cpu is also 20% slower.

Though even then, 20k-40k in view should be fine if you didn't by error go with opengl es 2.0 and/or use the "post fx" settings like bloom etc which due to their nature are very intense and troublesome due to the redraw of already drawn pixels (fillrate is likely the most limited resource on ipad and 4th gen devices as they still have the 3GS gpu yet 4 / 5.2 times the pixel to handle)
I'm no stonetrip representative, just a happy board member like you :)

If you find my portfolio interesting, feel free to contact me or follow my shares on my twitter stream or my game development blog
User avatar
dreamora
Platinum Boarder
Platinum Boarder
 
Posts: 844

Re: Shiva Benchmark

Postby error454 » 23 Feb 2011, 15:52

Tomha.Z wrote:I ll try to mod your project so i can try different texture settings.

If you agree i may share it later? (if it comes out clean enought for puplic reading ;P)

That's why I shared the project, feel free to mod away to fulfill your needs. Chances are that if you find it helpful, others will too.

Tomha.Z wrote:Maybe sharing the numbers out of your app here in this thread would be helpfull to others too?

This becomes tricky once different versions of the benchmark get pushed out. The numbers you get are only relevant to the version of the benchmark you are running. Also, a forum is a poor place to post this type of data - it's just really really hard to dig through posts and organize/sort data.

Someone should throw up a website that accepts http posts and we could upload results to it from the benchmark (device type, benchmark version, settings, fps).

Regarding the log calls, I had thought those were stripped out when exporting the .stk. Thanks for noting that they are not.

I have been hacking around some more on the benchmark. Currently I am putting in multiple textures and if I get around to it, I'll throw in multiple lights as well as light mode selection.
error454
Platinum Boarder
Platinum Boarder
 
Posts: 666
Location: Portland, OR USA

Re: Shiva Benchmark

Postby redmotion » 24 Feb 2011, 10:34

I like your approach to benchmarking. It occurred to me that this approach might be really useful when combined with the "device info database" app that I started with giggsy over on this thread:-
http://www.stonetrip.com/developer/forum/viewtopic.php?f=18&t=22133
(note that the latest version hasn't been posted on this thread)

Example database we created for this can be found here:
http://www.scrambled-egg.net/misc/shiva/stats/getStats.php

We could then document the polycount power of many mobile devices too - enabling developers here (who would rather not buy 30+ different phones/tablets) an easy reference point on which devices they would like to support and how they should taylor their apps to suit.

So I guess my question is, would you mind if we implemented a version of this benchmark into the device info app? (For me, the ideal situation would be the device-database-app auto-generates graphs similar to what you posted on your blog.)

(And ultimately, I guess the Shiva editor could even use this info to build a "simulate device mode" for the scene viewer - but maybe I'm just dreaming!)
User avatar
redmotion
Platinum Boarder
Platinum Boarder
 
Posts: 574

Re: Shiva Benchmark

Postby error454 » 24 Feb 2011, 18:49

redmotion wrote:I like your approach to benchmarking. It occurred to me that this approach might be really useful when combined with the "device info database" app that I started with giggsy over on this thread:-
http://www.stonetrip.com/developer/forum/viewtopic.php?f=18&t=22133
(note that the latest version hasn't been posted on this thread)

Example database we created for this can be found here:
http://www.scrambled-egg.net/misc/shiva/stats/getStats.php

Cool stuff, reminds me of newgl.benchmark.com but specific to Shiva engine.

redmotion wrote:We could then document the polycount power of many mobile devices too - enabling developers here (who would rather not buy 30+ different phones/tablets) an easy reference point on which devices they would like to support and how they should taylor their apps to suit.


This brings up an interesting discussion about polycount and pixels and frames per second. I was browsing the newgl site a few days ago, sorting devices by performance and noticed that FPS is an incredibly skewed statistic for comparing device performance. As an example, if you look at the iPhone 3GS vs iPhone 4 and look just at FPS, it will appear that the 3GS is a better performing device.

This is because of the native resolution of each device, something that FPS and polycount just don't show you. It seems that a metric like Pixels/Second would be a much better gauge of overall GPU power. So we'd have:
iPhone 3GS - 320 x 480 @ 28 FPS = 4.3E6 pix/sec
iPhone 4 - 640 x 960 @ 17 FPS = 10.4E6 pix/sec

Sorry, tangent.

redmotion wrote:So I guess my question is, would you mind if we implemented a version of this benchmark into the device info app? (For me, the ideal situation would be the device-database-app auto-generates graphs similar to what you posted on your blog.)

Go for it, I'd even be willing to help if you need help. I've been hacking at the benchmark a bit in my spare time and have a more accurate way to perform the hysteresis now. I think I'd like to officially open source this, maybe post it on GIT. I haven't used any of these open source hosting sites, if anyone has a recommendation let me know and I'll throw it up over the weekend.
error454
Platinum Boarder
Platinum Boarder
 
Posts: 666
Location: Portland, OR USA

Next

Return to W.I.P

cron