General

... or why a search engine indexing bot makes 160 000 incorrect requests within 48 hours.

China has implemented their own answer to Google. The search engine is called: Qihoo.com. The are some major differences to Google. The first thing I noticed was, that I cannot read chinese websites..... that´s bad....

But there is major problem with Qihoo.com. I noticed in the statistics of our project www.fertighaus.biz and www.mba-studium.info that we had about 160.000 requests all bringing up a 404 error. After some research in the log files, resolving the ip and using the better search engine Google I found out that the requests came from the Qihoo Bot. After some research using again Google I found statements from various web masters that this bot is working like a Zombie. It does not react on 404 errors either it reacts on 403 errors and it also does not take robots.txt file into account. Furthermore it makes several requests per second which can cause serious performance problems on small hostings. The bot is officially declared as "beta". But if this is the chinese definition of beta I will never ever use their beta products.

In our particular situation we had the problem that our error reporting was a little bit to exhaustive. We are using a front controller answering all request to the domain. Because we are using mod_rewrite our front controller inspects every request. For the wrong requests in made log statements to our application log and during two days the log files filled up our whole web space and the application was not able to create new sessions. This caused our application to malfunction.

Now I just blocked the IP addresses of the Qihoo.com. I really have no interest to be listed on chinese search engines. I used .htaccess and mod_access for it:

order allow,deny
deny from 220.181
allow from all

The most interesting references on the web are:
Qihoo - oder wehe wenn China Google ablösen will
Wenn ein chinesischer Crawler selbständig wird

It has been a while since I made my lost blog entry. In the last two month I had a very intensive private and working life. My contract at Siemens Enterprise is very near to run out and so I already take care for my time after Siemens. The first official and very important step to this will be Friday next week. My brother Manfred Schmelzer, Peter Krammer and I will found a new company. We are going into the Web 2.0 market and are about creating a new very exiting project. I do not want to tell very much about it right now, but check out my blog in the future and I will keep you up to date. It will get especially interesting, when we are looking for Alpha testers...

Another update has to be made for my post about User Manual Driven Design. I really like the idea and I am still thinking a lot about it. I recently found a very interesting blog article of Amazons´s CTO Werner Vogels. He is talking about Working Backwards were he proposes a very similar approach to software development as I do in my post about User Manual Driven Design.

In his last blog entry Michael Stal mentioned the importance of defining a new technology stack for Web 2.0 so that development can fulfil the requirements of upcoming (and already established) usability and interactivity trends on the web. Currently we are using a very chaotic technology stack, which consists of HTTP as a transport protocol, SOAP, REST or JSON for service integration, (D)HTML, CSS for mark up and JS to put logic into our static pages. By the way you have to use JS frameworks, CSS hacks, HTML hacks, HTTP hacks to make all the stuff working on various browsers and then you still have left the full bandwidth of server technologies.
What you see based on this listing that a full fledged end to end web developer must be really multi talented. So it is not really wondering that salaries for web developers are steadily increasing. So this is quite fine for guys like me, which are some sort of end to end web developers but it is very dangerous for the business. This in fact means that development of Web 2.0 like web apps is getting even more expensive. Not to mention the starting nightmare with IE 7 so that we will face the problem to be compatible to at least three major browsers within the next years (FF2.0, IE6, IE7).
So I agree to Michael Stal that we need new technology stacks. But the internet is too big for the revolution approach. This means we can only evolve the technologies using an evolutional approach. And evolution on something outdated like HTTP and HTML will get really hard.

Conclusion
Be careful when deciding to build modern web apps. It will get very expensive. Desktop applications are easier to develop and there must be a very well evaluated decision which technology brings more ROI.

I have just registered at Technorati and so the asked me to make a blog containing a link to my Technorati Profile. Of course I am doing this. I want to attract some more users to my blog.

Based on the suggestion from Wallace in his recent blog entry I also tried out Google Analytics. It is really an amazing tool and again for free up to 5.000.000 page impressions a day. I especially like the easy handling of profile to track special user navigation behaviour and the extreme easy way to integrate it into your own page. It is just 3 lines of js you have to include. I did not tried it out by now but using the given js function it should also be very easy to track AJAX requests or clicks to external links.

I already thought a while about creating my own blog. I have a lot of thoughts about IT, development and project management in my mind. I would like to share this thoughts and my blog should help me to organize this thoughts better and to formulate them more precise.

So this blog will mostly address IT topics. I will keep personal information rather spare because I like exchanging professional thought more than sharing my private information. But the next entry will address a little more about my person. You also find quite much information about me on my web page at www.schmelzer.cc.

I like reading blogs from other developers like Matt Raible or Michael Stal. But I also regularly read the blogs from my former colleagues Hilli and Wallace. Hopefully I can motivate them, to place a link to my blog. This would be very fine, because sharing my thought with at least some visitors is more exciting than just writing them for my own (-;.

First of all I have to sing a hymn on Serendipity. I downloaded it, put it on my server, started the installation script and everything was done. I like software, which can be installed easily very much. It is one of the most important things especially about Open Source software. The users need fast success or otherwise the will look further for the next project. And there are a lot of projects for nearly every problem you might imagine. I will try to take a look into the source code, too. In this project also Sebastian Bergmann was involved, which is a very good sign for the code quality of the project. But I will comment on that, when I have further insight into it.