A second facet of 'Climategate' is the reported shortcomings in the model code base. Part of the document release included source code. In a discussion with Rand Simberg over breakfast in LA earlier this month I heard that some very knowledgeable open source programmers are having a go at it. If half of what he told me turns out to be true, the models used by IPCC are worse than useless.
I have several times in my career translated serious numerical modeling code from Fortran to modern languages and thus had to deal with the issues of validating the results. In the real world mistakes cost money and sometimes lives. Most recently I translated some aerodynamics code for a New Space company. I spent weeks doing nothing but validating and checking to be sure the output was reasonably trustworthy for questions within the realm of interest. When Rand told me the CRU model code did not even handle numeric overflows I was speechless.
Let me explain. Computers represent numbers in binary. Any signed representation (ie one that handles plus and minus) will use some formatting trick to differentiate the two. The problem is, if a positive number gets incremented to be one bit too big... it may suddenly become a negative number. Regardless of what does happen, any calculation using the value after an overflow might as well be a random number generator. The results are totally, utterly worthless. There is not a chance in hell that the output will be meaningful.
There are ways of dealing with this sort of thing but I will not go into that sort of techno-detail here. My goal is simply to point out that if the statements I heard are true, I must cease to believe the validity of any output from CRU and CRU related models.
No doubt when the analysis is done we will see a release of the analysis.
In the mean time all you pros and cons have a heads up.
Engineering is the art of making what you want from what you can get at a profit.
MSimon wrote:
Let me explain. Computers represent numbers in binary. Any signed representation (ie one that handles plus and minus) will use some formatting trick to differentiate the two. The problem is, if a positive number gets incremented to be one bit too big... it may suddenly become a negative number. Regardless of what does happen, any calculation using the value after an overflow might as well be a random number generator. The results are totally, utterly worthless. There is not a chance in hell that the output will be meaningful.
This is misleading. This is only true for integers, floats behave differently on overflow, in a way that is easily detectable at the end of any computation (no numbers are printed, something like "###" instead).
I doubt anybody would use integers to perform actual computations.
MSimon wrote:
Let me explain. Computers represent numbers in binary. Any signed representation (ie one that handles plus and minus) will use some formatting trick to differentiate the two. The problem is, if a positive number gets incremented to be one bit too big... it may suddenly become a negative number. Regardless of what does happen, any calculation using the value after an overflow might as well be a random number generator. The results are totally, utterly worthless. There is not a chance in hell that the output will be meaningful.
This is misleading. This is only true for integers, floats behave differently on overflow, in a way that is easily detectable at the end of any computation (no numbers are printed, something like "###" instead).
I doubt anybody would use integers to perform actual computations.
Yes, very misleading. Specifically because he makes it appear that all code must handle overflows. But, the fact is that most code doesn't handle overflows because the parameters aren't reasonably expected to overflow. Unless he has identified a reason to expect an overflow that is not handled, he is just being nit picky about style.
The degree to which you need to protect your code against these kinds of things depends on the application. Although the results are critical, the actual real-time execution of this code is not. If it fails, it can be fixed and re-run without anyone dying.
MSimon wrote:
Let me explain. Computers represent numbers in binary. Any signed representation (ie one that handles plus and minus) will use some formatting trick to differentiate the two. The problem is, if a positive number gets incremented to be one bit too big... it may suddenly become a negative number. Regardless of what does happen, any calculation using the value after an overflow might as well be a random number generator. The results are totally, utterly worthless. There is not a chance in hell that the output will be meaningful.
This is misleading. This is only true for integers, floats behave differently on overflow, in a way that is easily detectable at the end of any computation (no numbers are printed, something like "###" instead).
I doubt anybody would use integers to perform actual computations.
Yes, very misleading. Specifically because he makes it appear that all code must handle overflows. But, the fact is that most code doesn't handle overflows because the parameters aren't reasonably expected to overflow. Unless he has identified a reason to expect an overflow that is not handled, he is just being nit picky about style.
The degree to which you need to protect your code against these kinds of things depends on the application. Although the results are critical, the actual real-time execution of this code is not. If it fails, it can be fixed and re-run without anyone dying.
regards
Yes, but iirc this is in reference to an actual segment of code which was processing integer overflows without realizing they were overflows. I think I remember reading about this in the first few days of CG.
MSimon wrote:
Let me explain. Computers represent numbers in binary. Any signed representation (ie one that handles plus and minus) will use some formatting trick to differentiate the two. The problem is, if a positive number gets incremented to be one bit too big... it may suddenly become a negative number. Regardless of what does happen, any calculation using the value after an overflow might as well be a random number generator. The results are totally, utterly worthless. There is not a chance in hell that the output will be meaningful.
This is misleading. This is only true for integers, floats behave differently on overflow, in a way that is easily detectable at the end of any computation (no numbers are printed, something like "###" instead).
I doubt anybody would use integers to perform actual computations.
Yes it is stupid. And yes they did.
You should have gone to the source I linked for details.
Engineering is the art of making what you want from what you can get at a profit.
The most interesting part of the article was the reminder that during WWII the Germans used the Fischer-Tropsch process, for which the feedstock is Carbon Monoxide, on an industrial scale.
That will have contributed to the stabilization of CO2 output during WWII...
The most interesting part of the article was the reminder that during WWII the Germans used the Fischer-Tropsch process, for which the feedstock is Carbon Monoxide, on an industrial scale.
That will have contributed to the stabilization of CO2 output during WWII...
Not really. You should study how it was done. They used their abundant coal resources to make liquid fuel.
The information is only a click or two away and yet you were incurious. I don't get it.
Unless he has identified a reason to expect an overflow that is not handled, he is just being nit picky about style.
I'll give you a clue. The code is adding squares of numbers.
BTW the guy who wrote the article I linked is an engineer.
Squares of numbers get big very fast. But you need them to create a curve fit solution. If I remember my old Vax days in college overflow was a problem.
MSimon wrote:
The information is only a click or two away and yet you were incurious. I don't get it.............
......BTW I learned about this in high school chemistry in 1959. I guess the teaching of chemistry is not as rigorous as it once was. Pity.
LOL, Sometimes M- you are just hilarious. High school chemistry, sigh so true, lol. I got a really good chuckle from MSimon, thanks, OMG, that was funny.
Cripes. a click or 2 away .....
I like the p-B11 resonance peak at 50 KV acceleration. In2 years we'll know.
MSimon wrote:
Let me explain. Computers represent numbers in binary. Any signed representation (ie one that handles plus and minus) will use some formatting trick to differentiate the two. The problem is, if a positive number gets incremented to be one bit too big... it may suddenly become a negative number. Regardless of what does happen, any calculation using the value after an overflow might as well be a random number generator. The results are totally, utterly worthless. There is not a chance in hell that the output will be meaningful.
This is misleading. This is only true for integers, floats behave differently on overflow, in a way that is easily detectable at the end of any computation (no numbers are printed, something like "###" instead).
I doubt anybody would use integers to perform actual computations.
Yes it is stupid. And yes they did.
You should have gone to the source I linked for details.
Well, if the "source" is as accurate and as trustfull as in this quote:
By the middle of this century liquid fuels such as gasoline will be generated using the Fischer-Tropsch process in some updated form. It will be carbon neutral because part of the feedstock will be free for the taking: atmospheric CO2. It will be split using either grid power, mechanical nanotechnology or genetically modified algae (some of which is purportedly working already). With the addition of energy, CO2 -> CO + O, and the Carbon Monoxide may be fed into the same FT process that was used to fuel the Nazi war machine. Towards the end of World War II this was nearly the only source of fuel available to Germany. Anyone who believes this technology is unproven on an industrial scale is simply historically ignorant.
then thanks, but no thanks, I decided to ignore this until I see the actual source code and check it for the integer overflow.
Unless he has identified a reason to expect an overflow that is not handled, he is just being nit picky about style.
I'll give you a clue. The code is adding squares of numbers.
BTW the guy who wrote the article I linked is an engineer.
Squares of numbers get big very fast. But you need them to create a curve fit solution. If I remember my old Vax days in college overflow was a problem.
You can safely ignore old Vax days, in modern computer, unless you do something really stupid (like using integer to perform basically float calculations), overflow is not a problem.
There is a lot of things you can do wrong when doing numerics, most of them are dealing with the fact that representation is approximate and has final precission. But overflow really is not something you have to fight with in any reasonable code.
If it fails, it can be fixed and re-run without anyone dying.
Really? Taking tens of trillions out of the world economy based on this phony baloney won't kill anyone?
You are missing the point. For some kinds of computation, it is OK to stop when error (e.g. overflow) occurs.
You can setup some compilers to check for integer overflow and stop the code if it happens. Some CPUs have even dedicated hardware for doing so.
Of course, there are applications where you have to be more rigid. But running climate model is hardly one where failure of single run would cause any problems. Invalid results, OTOH, are problem. But that is a different issue.
Just to be clear: if such code stops, it means your code is bad and you have to fix it. It is similar thing to Windows displaying "Your program has ecountered error..." and stopping the executable.
What about all the medicines that won't get developed? Life saving technology that will not be developed? Not to mention higher energy costs.
Totally irrelevant to the post.
Please note that I am AGW "denier". But I am getting a little bit bored and nervous with all these overshot claims of climategate miners.
The most interesting part of the article was the reminder that during WWII the Germans used the Fischer-Tropsch process, for which the feedstock is Carbon Monoxide, on an industrial scale.
That will have contributed to the stabilization of CO2 output during WWII...
Not really. You should study how it was done. They used their abundant coal resources to make liquid fuel.
The information is only a click or two away and yet you were incurious. I don't get it.
And how do you get Carbon Monoxide on an industrial scale? One way is to burn coal.
BTW I learned about this in high school chemistry in 1959. I guess the teaching of chemistry is not as rigorous as it once was. Pity.
You should be more curious. If you'd clicked just a bit further, you'd have discovered that they reacted coke with steam to produce carbon monoxide and hydrogen in a water gas reaction.
After the Fischer-Tropsch process, about two thirds of the products were petrol and diesel fuel which would release CO2 when burnt and the other third were oils, alcohols, aldehydes and soft waxes which became lubricants or industrial feedstock.