Although I have completed this dissertation, the actual program was not a success.  Therefore, I will be keeping this blog alive with the intention of picking up the program once my computer has made its way over here to sunny Sweden!

Dave

The stages of the program are as follows:

1. Initialise sub-systems
2. Create a custom render target
3. Load shaders required
4. Render the scene to the render target
5. Shader stores colour and depth information in same texture
6. Render target is switched to backbuffer
7. Full screen quad rendered with scene texture applied

Work has also been started on creating the actual sky effect itself.  However the screen has been drawing blank since this development started.  It is unclear if the problem is with the code I have added to the project or if it is the driver update I recently installed.  It is more likely to be the first issue as the the program does not fail at run-time.  There is still a context created implying that the OpenGL SDK is working well.

I have been advised by my colleague – Scott Dunbar – to use a piece of software called glslDevil to debug my program and make sure that everything is OK.  The reason for using glslDevil as opposed to Visual Studio for debugging is that Visual Studio does not debug shaders whereas glslDevil does.

By the next progress report I hope to have this problem fixed and continued development on the effect.  Work has been started on the introduction for the dissertation as well and this will be finished by the next report and a good start to the literature review will be underway

Introduction

Post-processing is a technique fundamental to graphics programming today. It allows for effects to be realised in great amounts of detail. This honours project will make use of this technique to demonstrate the potential it has to create visually stunning and accurate effects. This demonstration will be achieved by applying atmospheric scattering to a scene as a post-process to create very realistic skies.

Atmospheric scattering has been chosen as it is very realistic. It describes how incoming light from the sun is affected by the atmosphere to produce the stunning aura of colours observed daily. Another visual effect created with atmospheric scattering is that of haziness. This haze adds to the realism of the game environment and also gives the player much better depth cues.

Research Question

Atmospheric scattering has a reputation for being an expensive effect to implement. Post-processing could speed this up. To that effect, the question being presented is:

What are the benefits of using post-processing techniques to simulate atmospheric scattering in a computer game and what are the issues associated with its implementation?

Literature Review

“Modelling surfaces with polygons is often the most straightforward way to approach the problem of portraying objects in a scene. Polygons are good only up to a point, however. Image-based rendering (IBR) has become a paradigm of its own.” (Möller, T., et al, 2008)

As Möller et al discuss in their book, Real-time Rendering, there are many advantages to using post-processing in a scene. Complex objects like fur and clouds can more easily be added in this 2D image space after the entire scene has been rendered first. Complex lighting techniques, such as atmospheric scattering, are also ideally suited for post-processing. The reason this is true is because in post-processing you are only operating on the final pixel colour. There is no pixel overdraw associated with this technique.

To include post-processing in a game, one must first create an off-screen buffer. The entire scene is then rendered to this buffer instead of the back buffer. Once the drawing is complete, this off-screen buffer must be displayed on the screen. To do this, it is simply added as a texture to a quad that fills the screen. These are the principles of post-processing. The entire process is analogous to editing a photograph, where the photographer has a 2D image and adds Gaussian blurs, increases contrast, and decreases saturation for example, resulting in an altered image. By passing filters over the off-screen buffer, the programmer is able to achieve the same effect as in the photography example.

One important consideration with applying 3D effects, such as atmospheric scattering, as a post-process is that the depth buffer must also be resolved to another texture. The information in this texture must then be reconstructed in world-space to obtain the depth of each pixel. This last step is important as the depths are stored in a non-linear format.

The principles of atmospheric scattering are simple. Two coefficients must be solved individually and then used in the complete equation. These two coefficients are Rayleigh and Mie. Rayleigh describes what happens to light as it passes through the gases of the atmosphere itself, which determines sky colour. During the day the short wavelengths are scattered the most resulting in the blue sky. At sunrise and sunset, when the light travels through more of the atmosphere, the longer wavelengths are scattered more and this gives the red and orange skies.(Tomoyuki et al. 1993)

Mie scattering on the other hand describes how light from the sun interacts with more dense molecules in the air, such as dust and water vapour. This results in haze which limits how far a person can see. According to psychologists “aerial perspective is a fundamental depth cue that humans use to estimate distances, and the only absolute depth cue available for distant unfamiliar objects.” (Preetham, A. J., et al 1999). It is therefore vital that aerial perspective be implemented properly.

These coefficients are then used in the scattering equation to determine the final colour of the pixel. The equation is given by L(s, ?) = L_oF_ex(s)+ L_in(s, ?); where L_o is the initial colour of the pixel, F_ex is the extinction factor which has the effect of darkening objects that are further away and finally L_in is the inscattering component which has the effect of adding more light to distant objects. The function takes two arguments. The first argument, ‘s’, is the distance of from the viewer to the object and the second, ?, is the angle viewer-object vector and the sun direction.

There are two main approaches to implementing atmospheric scattering equations. One is to use the method proposed by A. J. Preetham in 1999 which is an analytic model. His model uses many simplifications and precomputations that give a reasonable effect, but limits the scalability. Another model for consideration is the model proposed in 1993 by Tomoyuki, N., et al. in 1993. This model is far more general and the ability to scale coefficients dynamically is possible. The downside to this model however is its inherent computational expense. Unlike Preetham’s model, Tomoyuki et al designed their model to allow the atmosphere to be viewed from space.

For adding realism to outdoor scenes in CryEngine2, Carsten Wenzel used the model proposed Tomoyuki et al 1993. However, only the skylight model was used and the aerial perspective was neglected. Instead they went with another implementation, which does exactly what the aerial perspective model proposed by Tomoyuki et al would do. This work around is then rather redundant as there are now two separate systems that handle the same thing.

After e-mail discussions with Damianno Iannetta of Rare Ltd on 7^th October 2009 it was recommended that this dissertation also use the model proposed by Tomoyuki et al and use the correct model for aerial perspective to give a more accurate effect.

Methodology

For the implementation of this project OpenGL will be the graphics API of choice. It has been chosen for its ease of use and functionality as well as its performance. The target platform will be Windows, however the framework will make use of SDL as it handles all the events very nicely and requires very little to set up and manage.

A simple scene will be required so that the user has a point of reference for up and down, left and right. This will take the shape of a simple heightmapped terrain. This terrain will be lit and textured in the conventional way with GLSL shaders. The user will be able to walk on, or fly around the terrain to view the effect from different angles.

As for the effect itself, floating point framebuffers will be used to store the fullscreen effect. The reason for this is to add high dynamic range rendering to the scene to make it look sharper. This buffer will then have the entire scene rendered to it before being applied as a texture to a full screen quad. Another texture will need to be used to hold all the depth information for the scene. This will be created in another pass and sampled from at the same time as the fullscreen effect. Together these two textures will contain the information needed for the effect.

Finally, a small scale GUI will have to be created to allow for dynamic changing of variables. This sort of feature is desirable as it allows for different settings to be created in no time at all and the effect seen instantly. This will be done using OpenGL to create sliders and display text information.

Evaluation

When evaluating this particular project there are a few important questions to answer:

1. Does the effect look good, does it do what it should?

2. How has using post-processing techniques benefitted this effect?

The first question is rather subjective, but it is an important and relevant question as the aim of this dissertation is to create something that is visually stunning and accurate. If it does not look good then the variables might need scaling. The second question is one that is slightly harder to answer, given that this dissertation will be focused on post-processing only, however data available from past research in the field of atmospheric scattering should provide some good comparisons.

Issues

As with any project, there are some issues that need addressing. For example, with post-processed atmospheric scattering, the depth buffer must be stored in a texture for sampling in the shader. Traditionally this requires another pass, however it might be possible to store this depth information in the same pass as the initial rendering by storing it in the alpha channel of the scene texture.

While post-processing reduces the number of times that the expensive scattering equations are performed, the equations can still have a detrimental effect on performance. Use of lookup tables and pre-computing coefficients may help alleviate this.

Resources Required

The resources required for this project are fairly minimal. Access to a computer with a suitable graphics card, such as an NVIDIA 8800GT, will suffice. Some sort of debugger for GLSL would be ideal, but not essential as any problems arising will most likely result in visual artefact, making them easier to track down.

Conclusion

At the end of the project it should be clear what the benefits are of post-processing over physical modelling. Some issues may arise that can provide a basis for future work however, most issues that arise should be manageable. Most importantly, by the end of this project there will be a strong application demonstrating atmospheric scattering as a stunning effect that adds accurate skies and usable depth cues for gamers.

References and Bibliography

Möller, T., Haines, E. and Hoffman, N. 2008. Real-time rendering. 3rd ed. Wellesley, Mass.: A.K. Peters

N Hoffman, A. J. P. 2002. Photorealistic real-time outdoor light scattering. Game Developer. pp.32-38.

O’Neil, S. 2005. GPU gems 2 : programming techniques for high-performance graphics and general-purpose computation. London: Addison-Wesley

Preetham, A. J., Peter, S. and Brian, S. 1999 In Proceedings of the 26th annual conference on Computer graphics and interactive techniquesACM Press/Addison-Wesley Publishing Co.

Tomoyuki, N., et al. 1993 In Proceedings of the 20th annual conference on Computer graphics and interactive techniquesACM, Anaheim, CA.

Wenzel, C. 2006 In ACM SIGGRAPH 2006 CoursesACM, Boston, Massachusetts.

Yoshinori, D., Tsuyoshi, Y. and Tomoyuki, N. 2002 In Proceedings of the ACM SIGGRAPH/EUROGRAPHICS conference on Graphics hardwareEurographics Association, Saarbrucken, Germany.

Introduction:

The focus of this dissertation is on atmospheric scattering as a post-processed effect.  Atmospheric scattering is the reason that our sky has colour.  It describes the way that the light from the sun disperses when it comes through our atmosphere.  In terms of game development, it can add REAL depth cues to vast landscapes and increase the immersion that the player experiences.  Plus, it looks good!

CryEngine2 uses this effect very well and creates astoundingly beautiful environments :

The image on the left is of Kualoa Ranch in Hawaii and the image on the right is the CryEngine2 replica.

In this implementation, though, the aerial perspective calculations used were not those defined by the atmospheric model they used.  Instead, they forced their own implementation to work.  Though it looks good, it is still not as good as it should be.  The aim of this honours project, then, is to use the same atmospheric model as CryEngine2 and to add the correct calculations for aerial perspective.

Issues:

The main issue surrounding this effect is that it is very expensive to calculate.  In the original implementations – see Nishita93 and Preetham99 – highly tessellated sky hemispheres were used.  This meant that for better representations of the sky, more vertices had to be used.  In a game, this would have a serious effect on performance.

Introduction

Atmospheric effects in games are becoming increasingly more popular; however some implementations are less efficient than others, giving arguably better results – (Tomoyuki et al. 1993) vs. (Preetham et al. 1999) for example. These models can be performed on a per-pixel basis to provide very realistic looking skies in games. It was suggested in an e-mail discussion with Damianno Iannetta of Rare on 7^th October 2009 that this project should steer itself toward the model proposed in (Tomoyuki et al. 1993) and as such, this dissertation will be focusing on an efficient post-processed implementation of this technique for PC and/or PS3, making the most use of hardware – i.e. GPGPU and Cell for PC and PS3 respectively.

Motivation

“Modelling surfaces with polygons is often the most straightforward way to approach the problem of portraying objects in a scene. Polygons are good only up to a point, however. Image-based rendering (IBR) has become a paradigm of its own.” As written by (Möller et al. 2008) in their book Real-Time Rendering begins to explain the importance of post-processing a 3D scene in a computer game. Things like clouds and fur are too complex to be represented as models. By creating images of the scene, these objects can be added as a post-process at a potentially lower cost – dependent on screen resolution and technique used – to look more realistic.

In a similar manner, this technique can be used to create realistic renderings of the sky between sunrise and sunset as demonstrated by (Wenzel 2006) using CryEngine2 and (Tomoyuki et al. 1993). The atmospheric effect put forward for use in CryEngine2, however, does not use the aerial perspective model put forward by (Tomoyuki et al. 1993), only the scattering model is used. It is therefore desirable to complete this technique by adding the correct method for aerial perspective.

There are numerous benefits to using accurate models for atmospheric conditions. Firstly, when a user begins to play a game, the visuals presented are one of the first things that will be noticed. Certainly, introducing magnificent effects will only aid in increasing the immersion experienced by the player – if supported by sufficient gameplay. According to a recent survey conducted by Steam last month 65.38% of gamers using Steam own an NVIDIA graphics cards and a further 27.26% own an ATI GPU (Steam 2009) . This implies that the majority of PC gamers at least are expecting to buy games with high graphics capabilities.

Secondly, great looking images can help with marketing. Again, when screenshots are released to the public it is important that the game looks appealing and carefully selected graphical effects can enhance a game’s reputation. Another example of how accurate models for atmospheric conditions can enhance immersion is in the simulation genre which includes games such as the Gran Turismo series by Polyphony Digital where the aim is to create the most visually stunning products possible as well as with the most realistic physics. This particular series includes some races that can last several hours and having a sky change over time can help solidify the feeling of time passing in game.

The problem with creating the most accurate model, however, is the effect it can have on gameplay. If it is not coded carefully enough then it could ruin the illusion of being in another world. Other things that can affect performance are the hardware of the player, especially for a PC based product where the variations in player hardware is almost infinite. Hardware such as the NVIDIA G80, G92 and GT2xx series graphics cards are ideal for these graphically intensive games with plenty of statistics supporting especially the latter series in iBXT Labs “NVIDIA GeForce GT2XX Reference” which goes on to explain technologies such as CUDA and PhysX and the acceleration they provide (Berillo 2009).

This brings us to the age old compromise between accuracy and efficiency. This dissertation will strive to get the very most accuracy with the least impact on efficiency. It is hoped that a full implementation of (Tomoyuki et al. 1993) can be achieved without compromise.

Research Question

The research question proposed, then, is:

Can atmospheric scattering be implemented efficiently in computer games, with minimal compromise in accuracy?

Addressing the question

The main issue being addressed is achieving maximum accuracy without affecting the performance of the video game. It has been seen in various white papers that performance is an issue with regard to atmospheric scattering. According to O’Neil in GPU Gems 2, using the methods described in (Tomoyuki et al. 1993) would require upto 300 computations per vertex using 5 samples for each scattering equation. (Pharr and Fernando 2005). It is obvious that using more samples for greater accuracy can greatly increase the number of computations. This project will be calculating the atmospheric model on a per-pixel basis as a post-process to eliminate the need for highly tessellated sky domes, which can help reduce computations further if care is taken. So in the case of the previous example, there would be 300 computations per-pixel.

The basic principal of post-processing is to render the entire scene to an off screen buffer and then apply this as a texture to a quad the size of the screen. An issue with performing atmospheric scattering as a post-process is that all the per-pixel calculations are performed in 2D space. This means there is no depth associated with each pixel and depth is required for attenuation of distant objects for correct aerial perspective. The work around for this is to create another texture that stores just the depth – the same technique used for shadow mapping. This texture can then be sampled at the same time as the scene texture to give distance. (i.e. pixel(ij) = depth(ij))

The final problem associated with this technique is of course efficiency. While it is highly accurate, this comes at the cost of performance, which in games is important. Some options exist for increasing performance without sacrificing accuracy. Firstly, a number of constants can be pre-computed as they are never likely to change during run-time. Another option is multi-threading – a very important concept in computing – meaning that several parts can be computed at any one time in parallel.

One such example of this is GPGPU or General Purpose GPU. GPGPU is an interesting development in graphics hardware as it allows for general purpose computation on the GPU directly, meaning that physics solvers, as an example, could be computed off the CPU either in part or whole, leaving the CPU free to handle other tasks, such as AI, gameplay, audio etc. It should be noted that the most appropriate use of GPGPU is with tasks that lend themselves well to parallelisation. Such tasks could be spread over roughly 12,288 threads on an NVIDIA 8800GTX GPU, implying that there could be a potentially huge performance increase. (David Luebke 2008) This means that potentially, the screen can be split into regions to be worked on by different threads on the GPU, potentially leading to an increase in performance.

On the PS3 there exist 7 programmable cores as well as a 256MB NVIDIA GPU, all of which can be optimised to run this post-processed effect. For example, an SPU could be used to calculate Rayleigh and Mie coefficients for scattering and then the GPU would only have to apply these numbers to each pixel in post-processing the scene. This could also lead to an increase in speed and it this technique that will be researched further for the purposes of the dissertation.

What equipment/resources will be needed/used

Access to the SIGGRAPH database for reference material is required as is a PS3 SDK. Some form of debugger for the PS3 would be desirable for speeding up the bug fixing process. There is little need for 3D models or high quality textures, as the effect is only applied to a texture and generates all required imagery for the effect to look realistic.

Reference

Berillo, A. 2009. NVIDIA GeForce GT2XX Reference. [online]. iXBTLabs.com. Available from: http://ixbtlabs.com/articles3/video/spravka-gt2xx-p2.html [Accessed 20 October 2009].

David Luebke, M. G., John Owens, Kevin Skadron. 2008. GPU Computing: The Democratization of Parallel Computing. [online]. Architectural Support for Programming Languages and Operating Systems. Available from: http://gpgpu.org/static/asplos2008/ASPLOS08-1-intro-overview.pdf [Accessed 19 October 2009].

Möller, T., Haines, E. and Hoffman, N. 2008. Real-time rendering. 3rd ed. Wellesley, Mass.: A.K. Peters

Pharr, M. and Fernando, R. 2005. GPU gems 2 : programming techniques for high-performance graphics and general-purpose computation. London: Addison-Wesley

Preetham, A. J., Peter, S. and Brian, S. 1999 In Proceedings of the 26th annual conference on Computer graphics and interactive techniquesACM Press/Addison-Wesley Publishing Co.

Steam. 2009. Steam Hardware Survey. [online]. http://steampowered.com. Available from: http://store.steampowered.com/hwsurvey/ [Accessed 22 October 2009].

Tomoyuki, N., et al. 1993 In Proceedings of the 20th annual conference on Computer graphics and interactive techniquesACM, Anaheim, CA.

Wenzel, C. 2006 In ACM SIGGRAPH 2006 CoursesACM, Boston, Massachusetts.

I have formulated several possible questions thus far and all are related to the topic of atmospheric scattering, as this is my main topic of interest.  I shall list them below:

• Can atmospheric scattering be used accurately in computer games?
• Is it feasible to include atmospheric scattering in computer games?
• Can post-processed effects – such as atmospheric scattering – be used in computer games without hindering performance?
• Can post-processed effects – such as atmospheric scattering – be used in computer games without negatively affecting overall performance?
• Can atmospheric scattering be implemented efficiently in computer games?
• Can atmospheric scattering be used as an effective depth cue in computer games?
• Can atmospheric scattering be used as an effective enhancement to traditional global illumination models?

While the last one holds the most interest for me, I do not think there is enough time to play with this question as it would involve the use of radiosity, ray tracing or photon mapping.  With that in mind, I feel that the question that will allow me to explore atmospheric scattering and post-processing would be the third-last question:

“Can atmospheric scattering be implemented efficiently in computer games?”

With this particular question, the suggested scope should be small enough to investigate GPGPU techniques and post-processing as well as provide me the choice of using either Preetham or Nishitas’ method for the atmospheric scattering model.

It was also brought to my attention earlier today that using Nishitas’ model would allow an appropriate use of GPGPU techniques as there are lots of variables to compute over several frames.

On a final note then, the research question I am putting forward is:

“Can atmospheric scattering be implemented efficiently in computer games?”

And with the Khronos Group releasing OpenCL in a bid to encapsulate both NVIDIA CUDA and AMD Stream in a similar manner to OpenGL and OpenAL, it is reasonable to expect developers to begin using this in the near future also.

Following are some web-sites to read through to gain a better understanding of CUDA

http://llpanorama.wordpress.com/cuda-tutorial/

http://www.nvidia.com/object/cuda_education.html

http://gpgpu.org/asplos2008

http://gpgpu.org/developer

This second video is very nicely polished and a great example of what can be achieved:

First I must state the topic of interest.

I have a keen interest in landscape photography and lighting composure is a key aspect in getting the perfect photograph.  In particular, sunset and sunrise are very pleasant effects.  I am interested in mimicking these atmospheric conditions in an interactive environment using post-processing techniques.

Currently the issues surrounding these post-processed effects is efficiency in real-time.  Various post-processing effects are used in Adobe Photoshop and The GIMP, where time isn’t really of the essence.  With atmospheric effects – such as scattering – game programmers and artists have traditionally used rough approximations due to computational expense at the hands of mathematical complexity, leading to less believable – and thus less immersive – environments.  However, these effects are becoming more popular now due to better hardware.

As stated I intend to mimic atmospheric scattering effects by applying it as a post-processed effect.  This principally means taking a screenshot of the framebuffer and attaching it to the depth buffer and applying this texture to a quad the size of the screen.  From here it is passed straight to a pixel/fragment shader where the effects are applied.  I will primarily stick to the methods outlined by Preetham in his 1999 SIGGRAPH paper “A Practical Analytic Model for Daylight” as the techniques he outlines have remained virtually unchanged.

There are two ways to implement these techniques:

1. The traditional CPU and GPU combination
2. The new GPGPU – General Purpose GPU – methods

In this project I will be looking into method 2 using NVIDIAs’ CUDA technology.  Due to lack of access, I will not be able to create an implementation using AMD Stream as I do not have access to a required ATI card.  I would be interested as well in creating a port of this project to a major console – time permitting – such as the PS3 or the XBox360.  However, I doubt there will be much time left over to try and make this port.

The aim of this honours project is to create a visual effect that can be used in games to create stunning and practical depth-cues for vast landscapes.

So, see you in September folks!

Bis bald,

Big Dave